System and method for machine learning based position estimation for use in microassembly control using a digital computer

Abstract

The present invention is entitled "System and method for machine learning-based position estimation for use in micro-assembly control with a digital computer". Control loop delays can be taken into account by using a hybrid model that includes at least one physics-based model and a machine learning model to predict the position of a moved micro-object. The model is combined using gradient boosting, where the model is created during at least one of the stages based on residual fitting calculated in a previous stage based on comparison with training data. A loss function for each stage is selected based on the created model. The hybrid model is evaluated with data extrapolated and interpolated from the training data to prevent overfitting and ensure that the hybrid model has sufficient predictive power. By including both a physics-based model and a machine learning model, the hybrid model can take into account both deterministic and random components involved in the movement of the micro-object, thereby increasing the accuracy and throughput of micro-assembly.

Description

Technical Field

[0001] The present patent application relates generally to microassembly control, and in particular to a system and method for machine learning based position estimation for use in microassembly control with a digital computer. Background Art

[0002] Micron- and nanometer-scale particle manipulation has received much research interest. The degree to which the assembly of micro-objects (i.e., objects whose sizes are measured in microns) and nano-objects (i.e., objects whose sizes are measured in nanometers) can be controlled can produce significant differences in many technologies (including in micro-manufacturing, biology, and medicine). For example, the manufacture of reconfigurable circuits can be improved by being able to accurately control the position of micro-objects (such as capacitors and resistors) to manufacture circuits with desired behaviors. Similarly, the production of photovoltaic solar cell arrays can benefit from being able to put photovoltaic cells with certain qualities into specific locations on the array. Such batteries may be too small to allow the desired placement of batteries via human or robot manipulation, and another type of transport mechanism is required. The micro- and nano-assembly of particles can also be used for the microstructure of engineered materials (such as biological cells assembled into tissues). There are many other technical fields in which increasing the control of micro-nano object assembly can provide significant benefits.

[0003] Many techniques for microparticle and nanoparticle manipulation use electrodes to generate electrode potentials to move particles of interest in the desired direction. The initial position of the particle is determined using a sensor, and the electrode potential is generated based on the desired position and the initial position. However, the throughput and accuracy of such systems are significantly limited by control loop delays, that is, the delay between when the particle is in a specific position and when the electrode potential is generated based on the particle being in the specific position. The exact length of the delay depends on the software and hardware of the system used to manipulate the particles. The delay has multiple components. One component is the actuation delay, that is, the delay between when a Sprite (control signal for an electrode) is generated and when an electrode potential specified by a Sprite is generated by an electrode. Another component of the delay is the sensing delay, which is the time between when the sensor observes that the particle occupies a specific position and that time. The analysis of the sensor signal is completed to obtain the particle position. Such delays include the time to transfer the captured image stream to a memory for storage and to send image data for image analysis to determine the position, as well as the time to run the image analysis algorithm. These delays can be as high as 80ms-100ms, and as high as 300ms or more, depending on various latencies and processor load, which can account for up to five frames at 60hz or slower frame rates.

[0004] Prediction delay, and therefore prediction needs to move the actual position of micron-level or nanometer-level particles will allow to improve the accuracy and efficiency of micro-assembly or nano-assembly.This type of modeling has generally utilized a model based on physics, utilizing a model of the known physical properties of the manipulated setting, environment (such as assembly is in air, in liquid, on the earth, in space or in another environment) and the particles moved. However, the model based on physics for closed-loop systems is not easy to consider the dynamic changes of the control delay that occurs during micro-particle or nano-particle manipulation (such as particle rotation, target distribution and the transfer that occurs on the small or large subset of particles). This type of model based on physics is also not easy to consider the dynamic changes between the different settings of the manipulation conditions, such as using different types of particles or different lighting conditions. This type of model based on physics is also not robust in the following cases: when modeling a specific variable in a specific manipulation setting (such as particle static friction or particle swing motion), or when modeling a parameter with unknown value (such as the average number of pixels that the particle translates during the control cycle).

[0005] Therefore, a way is needed to accurately account for control loop delays to predict the actual position of a particle moved by the resulting electrode potential. Summary of the invention

[0006] Control loop delays can be taken into account by using a hybrid model that includes at least one physics-based model and a machine learning model to predict the position of the moved micro-object. A gradient-enhanced combined model is used, wherein the model is created during at least one of the stages based on residual fitting calculated in a previous stage based on comparison with training data. A loss function for each stage is selected based on the created model. The hybrid model is evaluated with data extrapolated and interpolated from the training data to prevent overfitting and ensure that the hybrid model has sufficient predictive power. By including both a physics-based model and a machine learning model, the hybrid model can take into account both deterministic and random components involved in the movement of the micro-object, thereby increasing the accuracy and throughput of micro-assembly.

[0007] In one embodiment, a system and method for micro-assembly control enabled by a digital computer for machine learning is provided. One or more parameters of a closed-loop system for locating one or more micro-objects in a plurality of micro-objects are obtained, the system comprising a plurality of programmable electrodes configured to induce movement of the micro-objects when the micro-objects approach the electrodes after actuating the electrodes, each actuation in the actuation comprising one or more potentials generated by one or more electrodes in the electrodes. Training data is obtained, the training data comprising a state associated with a previous movement of one or more micro-objects in a plurality of micro-objects due to an earlier actuation in the actuation. One or more physics-based models for predicting the position of at least one micro-object in the micro-objects after an actuation in the actuation are defined. A hybrid model for predicting the position of at least one micro-object in the micro-objects after an actuation in the actuation is constructed using the training data via gradient enhancement, the hybrid model comprising a plurality of models, the plurality of models comprising one or more physics-based models and one or more machine learning models in a physics-based model. The position of one or more micro-objects in the micro-objects after an actuation in the actuation is predicted based on at least one sensor measurement result. Another position of one or more micro-objects in the micro-objects is received. Another actuation in the actuation is performed using the predicted position and another position.

[0008] Other embodiments of the present invention will become apparent to those skilled in the art from the following detailed description, wherein embodiments of the present invention are described by illustrating the best mode contemplated for carrying out the present invention. As will be appreciated, the present invention is capable of other and different embodiments, and its several details can be modified in various obvious aspects, all without departing from the spirit and scope of the present invention. Therefore, the drawings and detailed description should be regarded as illustrative rather than restrictive in nature. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a block diagram showing a system for machine learning based position estimation for use in microassembly control with the aid of a digital computer.

[0010] Figure 2 is a flow chart illustrating a method for machine learning based position estimation for use in microassembly control with the aid of a digital computer.

[0011] Figure 3 is a diagram showing a method for generating a Figure 2 Flowchart of the routine for training data used in the method.

[0012] Figure 4 is a diagram showing a method for constructing a Figure 2 Flowchart of the routines for the mixed model used in the method.

[0013] Figure 5A and Figure 5B is a graph showing the positions acquired by the micro-objects in the recorded training data (reference Figure 5A ) and the positions of the micro-objects extrapolated from the recorded training data (ref. Figure 5B An illustration of an example of ).

[0014] Figure 6 is a graph showing test results of Estimator 1.0 compared with the use of a hybrid model including an ARIMA model as a machine learning model and the use of a hybrid model including an RNN model as a machine learning model.

[0015] Figure 7 is a plot showing the results of using estimator 1.0 with the ARIMA model and the RNN model on the extrapolated test dataset.

[0016] Figure 8 is a graph showing a simulation of the entire extrapolated dataset (~50 minutes) using a trained System ID model without supervision.

[0017] Fig. 9 is a diagram showing a simulation of an extrapolated dataset using a trained System ID model without supervision.

[0018] Fig. 10A and Fig. 10B is a graph showing the test estimator 2.0 (mixed model) versus linear, dense, and ARIMA models implemented in the Microassembler software using parsimony depth (cpp).

[0019] Fig.11 is a diagram illustrating gradient enhancement according to one embodiment.

[0020] Fig.12 is a graphical representation of the relationship between recorded data, extrapolated data, and interpolated data as it relates to complex shell definitions. DETAILED DESCRIPTION

[0021] The system and method described below allow control loop delays to be taken into account when generating control signals, thereby allowing micro-assembly or nano-assembly to improve the accuracy of positioning. As described below, the system and method utilize a hybrid model to predict the position of a micro-object after actuation, wherein the hybrid model includes one or more physics-based models and one or more machine learning models. Among them, the physics-based model has the following advantages: no data needs to be constructed, thereby allowing extrapolation beyond the data used to fit the parameters and providing high interpretability of its predictions, and these models only capture a small part of the behavior of the system used to perform micro-assembly.

[0022] The complete behavior of the system 10 includes unmodeled degrees of freedom, and the use of machine learning models allows these unmodeled degrees of freedom to be captured. The overall system behavior given by Y is the physical model f(X; θ), where X is the previous history and measurements, θ is the parameter, and the noise N(ф), where ф is the noise parameter of the random noise model. The neural network function NN(X; W) as a function of the input and the trainable weights W captures the unmodeled degrees of freedom according to:

[0023]

[0024] The use of hybrid models allows combining the advantages of physics-based models (explanatory power, stability, and extrapolation capabilities) with the flexibility of machine learning models and their ability to capture unmodeled degrees of freedom, leading to more accurate predictions of the positions of micro-objects and thus improved throughput of micro-assembly. While machine learning models consider the stochastic components of the system, physics-based models consider the deterministic components of the system.

[0025] Although in the following description, the object being moved is described as a micro-object, the same technique can also be applied to control the position of a nano-object. In addition, although the hybrid model described below is described as being applied to a specific actuation hardware, in another embodiment, the hybrid model can be used to predict the position of a micro (or nano) object that moves using different hardware settings. In yet another embodiment, the hybrid model can be used to control other types of systems. For example, the hybrid model can be applied to control a mechanical linkage driven by a motor with recoil. The physical properties of the motor and the rotation of the mechanical linkage can be easily modeled using a physical model containing the physical parameters of the motor and the connecting rod (such as the inertia moment of the winding and the series resistance). The recoil of the linkage is much more complex, nonlinear and depends on previous history. The motor system is subjected to a series of random motion controls, changing the frequency and amplitude, and comparing the measured response with the expected response from the physical model, the error of the physical model as a function of the motor input and state can be obtained. The data is used for supervised training of a neural network, which is trained to predict the recoil error. Force the nearby time errors to be unrelated, thereby providing further constraints on overfitting. The remaining errors are fitted to a stochastic model which in this case is independent, identically distributed Gaussian noise. Noise variations can be determined based on the residuals from the physics and neural network models. The hybrid model is used for system identification, i.e., predicting the response of the system as a function of various control actions and the past and current state of the system. The prediction can be incorporated into various control algorithms (such as model predictive control) to achieve improved performance over simpler previous control algorithms (such as typical proportional integral derivative (PID) control algorithms). Still other uses of the hybrid model are possible.

[0026] Figure 1 is a block diagram illustrating a system 10 for machine learning based position estimation for use in microassembly control with the aid of a digital computer. The system 10 allows for coordinated assembly involving multiple objects 11, such as micro-objects. The size of the objects 11 varies between the nanoscale (<1 μm) and the microscale (1 um to several hundred μm), but other sizes are possible. The objects 11 can be spherical, but other shapes are possible, such as a rectangular shape, but still other shapes are possible. In one embodiment, the diameter of the spherical object 11 can be 5 μm to 50 μm, but other diameters are possible. In one embodiment, the size of the rectangular object 11 can be 200 μm×300 μm×100 μm, but other sizes are possible. The objects 11 are immersed in a dielectric fluid (not shown) contained in a housing (not shown), wherein a membrane layer (not shown) is lower than the dielectric fluid contained in the housing. In one embodiment, the dielectric fluid is The dielectric fluid 17 is made of a dielectric material such as di-2-ethylhexyl sulfosuccinate (AOT) charge director molecules, but other additives are also possible. Object 11 can be made of aluminum oxide (AlOx), but other materials are possible. Each of the objects 11 can be a semiconductor chip, an integrated circuit, a particle, a nano device or a structure, or any object that an electrode array can physically manipulate. In the description below, object 11 is referred to as micro-object 11, although other ways of naming object 11 are also possible.

[0027] Below the suspended micro-object 11 are a plurality of programmable electrodes 12 forming an array 35, the electrodes 12 being configured to generate a dynamic potential energy landscape for manipulating the object using both dielectrophoretic ("DEP") forces and electrophoretic ("EP") forces. A membrane is laminated onto the electrodes 12. In one embodiment, the electrodes may be square in shape and made of copper, but other shapes and materials are possible. In one embodiment, the dimensions of the square electrodes 12 may be 16 μm in width and 100 nm in thickness, but in another embodiment, other dimensions are possible. The array 35 may include multiple rows of electrodes 12, wherein each row includes multiple electrodes 12.

[0028] The potential generated by the electrode 12 is controlled by an array 13 of phototransistors, which includes an active matrix phototransistor backplane disposed on glass. A plurality of phototransistors on the backplane form an array 13, wherein each of the phototransistors in the array 13 controls the potential generated by a single electrode 12. Specifically, each of the phototransistors in the array 13 is attached to one electrode 12. The array 13 of phototransistors may have additional features, such as those described in "Chiplet micro-assembly printer" by Rupp et al., pp. 1312-1315, 2019 IEEE 69th Electronic Components and Technology Conference (ECTC), May 2019, the disclosure of which is incorporated by reference.

[0029] The array 13 is optically addressed by a video projector 14 to enable dynamic control of the electrostatic energy potential and manipulation of the positions of multiple micro-objects 11 simultaneously. Specifically, the video projector 14 is used to address the electrodes 12 controlled by each phototransistor, thereby allowing easy scaling factor changes and stitching into larger arrays. The video projector 14 actuates the electrodes 12 by projecting a predefined pattern: an image 37 generated based on a control input: the potential generated by the electrode to achieve the desired movement of the micro-object 11, as further described below. The pixelated light constituting the image projected by the projector 14 charges each individual phototransistor 13 in the array to the necessary extent so that the electrode 12 charged by the phototransistor produces the desired potential. In one embodiment, voltages up to + / -200V are controlled by these phototransistors to display dynamically changing complex potential landscapes. A single voltage signal is synchronized with the optical addressing to charge the storage capacitor on each pixel. A dynamic electric field pattern is generated by refreshing the projected image pattern (control input). Arrays 13 with 50 μm, 10 μm and even 3 μm pitch have been constructed, where the pitch is the distance between the centers of the electrodes to which the phototransistors are attached. Dielectric Fluid 17 A dielectric (isopar M) and various additives are used to control the charge.

[0030] The system 10 also includes a high-speed camera 15 that tracks the position of the micro-object 11 being moved. Both the video projector 14 and the camera 15 are interfaced to one or more computing devices 16, which can control the electrodes 12 through the projector 14 to cause the formation of the desired micro-object 11 pattern. Other micro-object 11 patterns are also possible. In another embodiment, another sensor (such as using capacitive sensing) can be used to measure and track the position of the micro-object. In another embodiment, the electrode array 35 may have an integrated active matrix addressing electronic device, and the potential on each electrode of the array is updated electronically when the electrode potential necessary for the desired micro-object movement has been generated. The additional characteristics of the array 13, the electrode 12, the video projector 14 and the camera 15 are described in the U.S. Patent Publication Application No. US20200207617 published by Plochowietz et al. on July 2, 2020, and the disclosure of the patent is incorporated by reference. Moreover, additional components of the system 10 that interact with the array 14, the electrode 12, the video projector 14 and the camera 15 are also possible.

[0031] The connection between one or more computing devices 16 can be direct, such as via a wire or wireless connection to one or more computing devices 16 that are physically close, or the connection can reach one or more computing devices 16 that are physically remote via an interconnected network, such as via the Internet or a cellular network. One or more computing devices 16 include one or more circuits that perform data processing, such as a computer processor, but other types of circuits for processing data are also possible. In one embodiment, the computer processor can be a central processing unit (CPU), a graphics processing unit (GPU), or a mixture of CPUs and GPUs, although other types of processors or mixtures of processors are also possible. In one embodiment, two or more of the processors can perform processing in parallel, as further described in the following patents: U.S. patent application serial number 17 / 098,816 entitled "System and Method For Multi-Object Micro-Assembly Control With The Aid of A Digital Computer" filed by Matei et al. on November 16, 2020, the disclosure of which is incorporated by reference. Additional efficiency and improved scalability can be obtained by parallel computing power for tracking multiple particles and for voltage pattern generation. In yet another embodiment, a circuit other than the CPU and GPU may perform the above-described data processing.

[0032] Although one or more computing devices 16 are shown as servers, other types of computer devices are possible. Computing device 16 may include one or more modules for executing the embodiments disclosed herein. Modules may be implemented as computer programs or procedures written as source code in conventional programming languages, and are presented for processors to execute as objects or byte codes. Alternatively, modules may also be implemented in hardware, as integrated circuits or burned into read-only memory components, and each computing device in computing device 16 may act as a dedicated computer. For example, when a module is implemented as hardware, the specific hardware is specifically used to perform the above-mentioned calculations and communications, and other computers cannot be used. In addition, when a module is burned into a read-only memory component, the computer storing the read-only memory becomes specifically used to perform operations that other computers described above cannot perform. Source code and various embodiments of objects and byte codes may be maintained on computer-readable storage media, such as floppy disks, hard drives, digital video disks (DVDs), random access memories (RAMs), read-only memories (ROMs), and similar storage media. Other types of modules and module functions are possible, as well as other physical hardware components. For example, computing device 16 may include other components found in programmable computing devices, such as input / output ports, network interfaces, and non-volatile storage, but other components are possible. In embodiments where computing device 16 is a server, the server may also be a cloud-based server or a dedicated server.

[0033] One or more computing devices 16 interface with the storage device 17 and execute a hybrid model creator 18, which obtains parameters 19 of the system 10 and stores the parameters 19 in the storage device 17. The parameters 19 may include the diameter of the micro-object 11, the size of the electrode 12, the dielectric fluid constant (e.g., ε=2), the fixed position of the electrode 12, the materials of the micro-object 11 and the electrode 12, and the vertical distance between the micro-object 11 and the electrode 12 (the "height" of the micro-object 11). Other parameters 19 are still possible. The hybrid model creator 18 also creates a hybrid model 24 for predicting the position of a specific micro-object 11, which can also be stored in the storage device 17. The hybrid model 24 includes a physics-based model 26 that predicts the position of the micro-object 11 after electrode actuation based on the following items: the generated control signal (and therefore the potential generated by the electrode), the speed of the micro-object 11 at a specific time, and the starting position 28 (the position before actuation) determined based on the micro-object image 29 captured by the camera 15. J. Willard, X. Jia, S. Xu, M. Steinbach, and V. Kumar, “Integrating physics-based modeling with machine learning: A survey” (arXiv:2003.04919 [physics.stat] (2020)), describes an example of a physics-based model, the disclosure of which is incorporated by reference, but other physics-based models 26 are also possible. The hybrid model 24 also includes one or more machine learning models 27. The machine learning model may include a deep model (a machine learning model utilizing a neural network made of multiple layers), a linear model, a dense model, an auto-regressive integrated moving average (ARIMA) model, a recursive neural network (RNN) model, but other kinds of machine learning models are also possible. In one embodiment, the simple RNN layer of the ARIMA model can be changed to an LSTM (long short-term memory, RNN architecture), but other kinds of ARIMA models are also possible. As described above, the physics-based model 26 takes into account the deterministic behavior of the system 10, while the machine learning model 27 takes into account the deterministic aspects of the system 10. Therefore, the behavior of the system 10 as a whole can be given by f(X; θ) as a function of the physical model, where X is recorded data associated with previous actuations of the system 10 (referred to as training data 22 below), θ is a parameter, noise N(ф), where ф is the noise parameter of the random noise model, and the system behavior is further represented by a neural network function NN(X; W) (associated with the machine learning model 27), which is a function of the input and the trainable weights.

[0034] The hybrid model 24 is constructed using the training data 21. The training data describes states associated with the movement of the micro-object 11, including: (1) input states of the micro-object, which are minimally the position and velocity of the micro-object 11 before the control input actuates the electrode (but may also include other states, such as orientation and angle); (2) control actions, such as an illumination pattern applied by a video projector 14 to the phototransistor control electrode 12 (which indicates the potential generated by the electrode) or another force applied to the micro-object 11; (3) output states generated by the input data (the position of the micro-object 11 has been achieved after actuation at a specific time and velocity of the micro-object). The training data 22 is used for supervised training of sequences of parameter models that constitute the hybrid model 24, such as Fig.11 shown. Fig.11 The "traditional" model shown may be a physically based model 24, followed by, for example, a model 2 on the residuals (e.g. Fig.11 as shown) and a neural network model that fits another model on the residuals.

[0035] The hybrid model is tested using interpolated data 33 (also referred to as interpolated data 33) and extrapolated data 23 (also referred to as extrapolated data 23) used as test data 101, as further described below. The interpolated data 33 may initially be part of the same data set as the training data 22 and include input states, control actions, and output states whose values ​​are within the range of values ​​of the training data 21. The extrapolated training data 23 may also be part of the same data set as the training data 21 and also include input states, control actions, and output states, but include values ​​of at least one of the input states, control actions, and output states that are outside the range of values ​​of the input states, control actions, and output states of the training data 21. For example, at least one of the position, velocity, or actuation applied to the micro-object 11 of the micro-object 11 is for values ​​that extend beyond those used in the training data 21.

[0036] Figure 5A and Figure 5B is an annotated bright field image of the experimental (record 21) data, in which the micro-object 11 is highlighted by a white circle and the set of target locations between which the micro-object was translated are highlighted by magenta squares. Figure 5A and Figure 5B Two different experiments with two different sets of target positions are shown. The experimental data 21 can be used as training data 22. Fig.12 is a diagrammatic representation of the relationship between training data 22, extrapolated data 23 and interpolated data 33 when relating to a complex shell definition. Still other kinds of training data are possible.

[0037] As reference below Figure 2To begin further description, the hybrid model 24 is created using gradient boosting implemented in multiple stages, during which one or more physics-based models are trained by selecting a loss function, where the next stage is implemented by selecting another model (machine learning model 27) using the residual 31 and the previous input as new input and selecting a loss for boosting (training) the parameters of the next stage, where the creation of additional models is iterated multiple times in this manner.

[0038] Thus, during gradient boosting, at least one physics-based model 26 is initially defined, and then the difference between the output training data 22 (the position of the micro-object 11 at a specific time after actuation) and one or more predictions made by using the physics-based model 26 about the position of the micro-object 11 achieved simultaneously, as described by the output training data after the same actuation based on the input training data, is analyzed using a loss function 30 (also called a whitening objective function 30). The result of this analysis is one or more residuals 31 (also called pseudo residuals 31), wherein each residual 31 describes the difference between a position predicted by the physics-based model 26 based on a piece of input training data 22 and an output training data 22 produced by the piece of input training data 22 on which the prediction is based. Subsequently, during the next stage, the residuals 31 derived from the initial physics-based model 26 are used to construct a first machine learning model 27, which is fitted based on the generalized residuals 31 of the physics-based model 26. Subsequently, during the next stage, residuals 31 for the first machine learning model are generated based on the training data 22 in the same manner as described above - by using a loss function 30 to process the differences between the micro-object positions predicted based on the input training data 22 and the positions in the output training data 21. The residuals 31 from the first machine learning model 30 are used to define another deep model, where the residuals 31 for the second deep model are calculated in the same manner as described above. Fitting the generalized residuals of one machine learning model 27 to the next machine learning model 27 continues in multiple stages as a cascade until a desired number of models 26, 27 have been generated (in one embodiment, the desired number of models is based on past data about the accuracy of a hybrid model including a certain number of models 26, 27; in another embodiment, the desired number of models 26, 27 is set based on empirical testing of whether a predetermined number of models produces the desired accuracy). Although in the above description, only a single physics-based model 26 is used as part of the hybrid model 24, in another embodiment, multiple physics-based models 26 can be used as part of the hybrid model. One or more physics-based models 26 in the hybrid model 24 act as weak learners, i.e., models whose predictions are only slightly better than random guessing, while the trained machine learning models 27 act as strong learners, which have higher prediction accuracy than weak learners. The predictions of all models 26, 27 are aggregated into a single prediction of the hybrid model 24, such as by assigning predictions from different model weights and averaging the predictions, although other ways of aggregating predictions are possible. For example, according to equation (2) below, if residuals are represented as factors rather than differences, the predictions of the various model stages 26, 27 can be aggregated into a combined prediction of the hybrid model 24 as a sum or as a result refinement. Various weights can also be assigned to the various stages.Mathematically, the gradient boosting performed by the hybrid modeler 18 can be defined as follows: .

[0039] A sequence of M learners is used to refine the predicted output. The weak (or base) model includes a physical / deterministic model 26, followed by a machine learning model to capture the predictable but complex parts.

[0040]

[0041] Where M is the number of iterations, (f0) is the initial guess and (f i ) is an enhanced or additional family of approximations

[0042]

[0043] in

[0044]

[0045] Among them Ψ i(.,.) Loss function for stage I. In summary, sequential methods involve multiple stages of fitting a model to the residuals of the previous stages using a specific loss function. In the current case of a physical model, the domain knowledge of the assembly or physical system is contained in the first stage.

[0046] Fig.11 is a diagram illustrating gradient enhancement according to one embodiment.

[0047] Return to Figure 1, while the training data 22 is used to calculate the residual 31 after creating the hybrid model 24, the hybrid modeler 18 uses the extrapolated test data 23 and the interpolated test data 33 to test the predictions made by the hybrid model 24. Initially, the hybrid model 24 is tested against the interpolated test data 33. If the position predictions made by the hybrid model 24 based on the input interpolated data 33 are within a certain error threshold from the output interpolated data 33 (indicating that the model 24 has been adequately trained), the hybrid model 24 remains unchanged. If the position predictions made by the hybrid model 24 are outside the error threshold, the hybrid model 24 is modified. Such testing prevents overfitting of the hybrid model 24. After testing with the interpolated data 33, the hybrid model is tested with the extrapolated test data 23 in the same manner as the interpolated data 33 described above: by comparing the position predictions made by the hybrid model 24 based on the input extrapolated data 23 with the output extrapolated data 23. If the position predictions made by the hybrid model 24 based on the input extrapolated data 23 are within a certain error threshold from the output interpolated data 23, the hybrid model 24 remains unchanged. If the position prediction made by the hybrid model 24 is outside the error threshold, the hybrid model 24 is modified. Such testing using the extrapolated data 23 set helps ensure that their hybrid model 24 has sufficient prediction and extrapolation capabilities, rather than being tied to just a single training data 22 set.

[0048] As described above, the Ψ(.,.) loss function 30, represented as follows, is used during the calculation of the residual 31 during each of the stages of gradient enhancement, and the loss function 30 must be selected by the hybrid model creator 18 for each of the stages. The loss of the function 30 is selected based on the stochastic model 102 (a model describing those aspects of the modeled system that are not easy to predict and can vary from run to run) associated with the data that the hybrid model 24 attempts to predict (and therefore, associated with each stage of building the hybrid model 24). For example, when the hybrid model 24 is to be used to predict continuous data (such as in the case of the position of the micro-object 11), the stochastic model 102 is a Gaussian model. If the predicted data is counts, the stochastic model 102 is a Poisson model. If the prediction model is a category, the stochastic model 102 is a polynomial model. Other kinds of stochastic models 102 are also possible.

[0049] Different loss functions 30 may be used in different stages of gradient boosting. When the physically based model 26 is the mean square error (MSE), one loss function 30 that may be used during the calculation of the residual is:

[0050]

[0051] is sufficient, where is the autocorrelation function of the residual at the ith lag

[0052] ri =y i -f(x i ) (6)

[0053] However, MSE does not generally ensure that the residuals calculated as described above are compatible with the stochastic model 102, which generally assumes independent, identically distributed (IID) random variables. Therefore, the mixture model creator 18 compares the residuals 31 calculated during each stage of gradient boosting with the stochastic model 102 using an autocorrelation function (such as a zero-mean Gaussian with variance σ 2 To select the loss function 30, the hybrid model creator 18 may use the Ljung-Box (LJB) loss function:

[0054]

[0055] where ρ is the autocorrelation function at lag k and n is the sample size.

[0056] The use of the LJB function ensures the compatibility of the uncorrelated residuals 31 and the residuals generated using the loss function 30 with the stochastic model 102. In equation (7), L is a hyperparameter greater than a possibly relevant one. For each stage of gradient enhancement, a loss function is selected to drive the output residual 31 toward the desired target (optimizing the loss function). Planned subsequent actuation allows compensation of control loop delays, and thus improves the performance of the system 10. In one embodiment, where the frame rate is every 60 Hz (although other values ​​of the frame rate are also possible), there are about 2-3 frames of control loops from when the particle 11 is in a physical position to when the particle is assessed to be in that position (due to the computational cost of image transfer and image processing analysis), and there is a delay of about 1 frame for actuation (from when it is determined that the particle is in a position to when an electrode potential is generated to move the particle 11). The control strategy 34 allows compensation of the 3-4 frame delay and estimates the particle position for the first 3-4 frames.

[0057] When moving the micro-object 11 to the desired position 35 most often requires multiple actuations, having an accurate starting position 28 of the micro-object 11 after each actuation is critical to generating the proper electrical potentials during subsequent actuations to move the micro-object 11 toward the desired position 35 in the fastest manner. Figure 2 is a flow chart illustrating a method 40 for machine learning based position estimation for use in microassembly control using a digital computer according to one embodiment. The method 40 may be used Figure 1 Initially, the parameters 19 of the system 10 are obtained, and the control model 39 is generated using the parameters 19 (step 41). The training data 22 is obtained, as shown below with reference to Figure 3A hybrid model 24 is constructed (step 43) for predicting the position of one or more micro-objects 11, as described further below. Figure 4 As further described above, Figure 1 The interpolated data 33 described above is used to test the accuracy of the hybrid model 24 (step 44), and if the test result is satisfactory and the test is passed (step 45), the method moves to step 46. If the test result is not satisfactory, the routine returns to step 43. Using the above reference Figure 1 The described extrapolated data further tests the accuracy of the hybrid model 24 (step 46), and if the test results are satisfactory and the test is passed (step 47), the method moves to step 48. If the test results are not satisfactory, the routine returns to step 43.

[0058] The desired position of one or more of the micro-objects 11 is received, such as from a user (step 48). The starting position of the micro-object 11 is obtained by using only the image 29 captured by the camera 15 if the micro-object 11 is stationary, or by using the hybrid model 24 (which takes into account the data derived from the image 29) if a previous actuation has been performed and the micro-object 11 is now in motion (and therefore the control loop delay is now a factor in determining its position) (step 49). The control strategy 34 for moving the micro-object 11 toward the desired position 35 is generated using the starting position 28, the control model 39, and the desired position 35 (step 50), and the electrode 12 is actuated via the video projector 14 using the control strategy 35 (step 51). If the desired position 35 has been achieved after actuation (such as determined using the hybrid model 24) (step 52), the method 40 ends. If the desired position has not yet been achieved, the method 40 returns to step 50.

[0059] The recorded data 21 is necessary for both training and testing of the hybrid model 24 . Figure 3 is a diagram showing a method for generating a Figure 2 Flow chart of routine 60 for training data 22 and test data 101 used in method 40 of . A set of recorded data 21 is obtained (step 61), and is divided into training data 22 and test data 101, wherein the test data includes interpolated data 33 (data falling within the boundaries of the data set as training data 22) and extrapolated data 23 (data falling outside the interface of the data set as training data 22) (step 62), and then the routine is terminated. In one embodiment, the data used as training data is 80% of the data set, and the data used as test data 101 is 20% of the data set, but other percentages are also possible.

[0060] A hybrid model 24 constructed using gradient boosting combines predictions from at least one physics-based model 26 and one or more machine learning models 27 to achieve a more accurate prediction of the position of the micro-object 11 at a specific time (the time of electrode actuation using the predicted position) by taking into account both the stochastic and deterministic dynamics of the system 10. Figure 4 is a diagram showing a method for constructing a Figure 2 Flowchart of a routine 70 for configuring a hybrid model 24 used in method 40 of . Initially, a number of models 26, 27 to be included in the hybrid model 24 are set, such as based on user input, such as based on past performance of a hybrid model 24 having a certain number of models 26, 27, or based on continued testing of the hybrid model 24 created during method 60 (step 71). A physics-based model 26 is defined, such as using parameters 19 (step 72). Based on one or more input record data 21, one or more predictions are made using the physics-based model 26 regarding the location of one or more micro-objects 11 at a particular time (step 73). As described above with reference to Figure 1 As described above, the loss model 30 is selected and the Figure 1 The stochastic model 102 associated with the predicted data is selected (step 74). One or more residuals 31 are calculated using the selected loss function 30 based on the difference between the prediction made using the physics-based model and the output data associated with the input data used to make the prediction (the recorded micro-object position achieved due to the actuation associated with the input data) (step 75). An iterative processing loop is initiated for all remaining models (steps 76-81). A model is constructed (fitted) based on the residuals of the previous model (step 77). Based on one or more input recorded data 21, a prediction is made regarding the position of one or more micro-objects 11 at a particular time using the model constructed in step 77 (step 78). The stochastic model 102 is selected for use as described above with reference to Figure 1 The predicted data, and based on the random model 102, the loss function 30 is selected for the model constructed in step 77, as described above with reference to Figure 1The process of steps 76-81 is repeated until the number of models 26, 27 to be included in the hybrid model 24 set in step 71 (including the physically based model 26 defined in step 62) has been achieved, thereby terminating routine 70. In the above description, in one embodiment, all models created during loop 76-81 can be machine learning models 27. In another embodiment, at least some of the models created during loop 76-81 can be physically based models 26. In yet another embodiment, a machine learning model 27 can be created during step 72 instead of a physically based model 26, and at least one physically based model can be created during loop 75-81.

[0061] The advantages of using the hybrid model 24 described above were tested experimentally, and the experimental data is given below for purposes of illustration and not limitation. The model 24 was implemented in the system 10 using a minimalistic deep cpp library (although other implementations are possible), and the performance of the model 24 was compared to the performance of what is referred to below as the estimator 1.0 (which was implemented using a system similar to the system 10 except that the hybrid model 24 was not employed), which estimated the particle position at time 1 by using its measured position at time 1 within feedback control - #n*1 / 60s control loop delay and an estimated constant #n for the control loop delay. The performance of the hybrid model 24 was also compared to the performance of a system similar to the system 10, except that only the machine learning model 26 was used instead of the hybrid model 24 to determine the particle position. Figure 6 is a graph showing test results of Estimator 1.0 compared to: using a hybrid model 24 including an RNN model as a machine learning model 27 ("Estimator 1.0 enhanced with RNN"); using only an ARIMA model to predict particle locations; using an ARIMA model enhanced with an RNN model (two machine learning models 26) to predict particle locations; and using an RNN model as the machine learning model 27. The hybrid model 24 including the ARIMA model and the hybrid model including the RNN model show more than twice better predictions about the location (x, y, surrounded by boxes) of micro-objects 11 (called "particles") than Estimator 1.0. Using reference Figure 5A The recorded data 21 shown achieve these results (wherein the positions are also referred to as "waypoints").

[0062] Even when the extrapolated data 23 is used for testing, the hybrid model 24 performs better than the estimator 1.0. Figure 7 is a graph showing the results of using Estimator 1.0 with the ARIMA model and the RNN model in the extrapolated test dataset. The hybrid model 24 including the ARIMA model and the hybrid model 24 including the ARIMA model enhanced with the RNN model show about 1.5 times better prediction of the particle (micro-object 11) position represented by x and y (surrounded by the box), where the RNN model performs the worst on the extrapolated test dataset (surrounded by the box).

[0063] Figure 8 is a diagram showing the simulation of the entire extrapolated data set (~50 minutes) using the unsupervised trained system ID model. For the final position of a 10um spherical particle (micro-object 11), the hybrid model 24 including the RNN-enhanced Arima model performed best, with a cumulative RMSE error of ~10 camera pixels and an inaccuracy of 5um. Estimator 1.0 resulted in a cumulative RMSE error of about 200-300 camera pixels (inaccuracy of the final particle position of ~100-150um).

[0064] Fig. 9 2 is a diagram showing a simulation of an extrapolated data set using an unsupervised training system ID model. For the final position of a 10um spherical particle (micro-object 11), a hybrid model 24 including an RNN-enhanced Arima model performs best, with the true (gray) and predicted (black) particle positions slightly deviating and having a cumulative RMSE error of ~10 camera pixels and an inaccuracy of 5um. Using estimator 1.0 results in a cumulative RMSE error of about 200-300 camera pixels (inaccuracy of the final particle position of ~100-150um). The RNN model inaccurately predicts the particle position.

[0065]

[0066] Table 1 .

[0067] Hybrid model 24 also performs better than using a pure machine-based learning model. Table 1 shows the comparative accuracy of hybrid model 24 (referred to as estimator 2.0 in Table 1) compared with linear, dense and Arima models implemented using minimalistic depth (cpp). In the simulator mode of the microassembler software, the model was trained on a set of randomly generated waypoints (micro-object positions) for 2h and tested for 1 minute in a new waypoint set. Compared with the improved estimator model, i.e., estimator 2.0 (darker colored boxes), the ARIMA model again shows a 2-fold improvement in the predicted particle position. The implementation runs faster than 60Hz (control cycle clock) for predicting the position of hundreds of particles (micro-objects 11), but was not amplified to 1000 particles (lighter colored boxes) in this particular experiment.

[0068] Fig. 10A and Fig. 10B is a graph showing the test estimator 2.0 (mixture model 24) versus linear, dense, and ARIMA models implemented in the Microassembler software using parsimony depth (cpp). Fig. 10A It is shown that the average prediction error rate of the RNN (LSTM) enhanced Arima model is reduced by > 1 camera pixel compared to the Improved Estimator 2.0 model. Fig. 10B It is shown that an embodiment using deep and RNN models runs faster than 60 Hz (internal control loop clock) for prediction of <300 particles (micro-objects 11), however it was not scaled up to 1000 particles in this particular experiment.

[0069] While the invention has been particularly shown and described with reference to embodiments thereof, it will be understood by those skilled in the art that the foregoing and other changes in form and details may be made therein without departing from the spirit and scope of the invention.

Claims

1. A method for micro-assembly control enabled by machine learning with a digital computer, comprising the following steps: obtaining one or more parameters of a closed-loop system for locating one or more micro-objects of a plurality of micro-objects, the system comprising a plurality of programmable electrodes configured to induce movement of the micro-objects when the micro-objects are in proximity to the electrodes upon actuation of the electrodes, each of the actuations comprising generating one or more electrical potentials by one or more of the electrodes; obtaining training data including states associated with previous movements of one or more micro-objects of the plurality of micro-objects due to an earlier one of the actuations; defining one or more physics-based models for predicting a position of at least one of the micro-objects after one of the actuations; using the training data via gradient boosting to construct a hybrid model for predicting a position of at least one of the micro-objects after one of the actuations, the hybrid model comprising a plurality of models including one or more of the physics-based models and one or more machine learning models; Get the dataset; Dividing the data set into the training data and interpolated test data; extrapolating the state of the previous move to generate extrapolated data; testing the hybrid model using the interpolated test data and the extrapolated data; After the hybrid model passes the test, controlling the closed-loop system includes: predicting a position of one or more of the micro-objects after one of the actuations based on at least one sensor measurement; receiving another position of one or more of the micro-objects; and performing another of the actuations using the predicted position and the another position, The steps described therein are performed by a suitably programmed computer.

2. The method of claim 1, wherein the gradient boosting comprises a plurality of stages, wherein one of the models is constructed during each of the stages, further comprising: for each of the models, determining, using a loss function, a residual between a prediction of the one model based on an input portion of the training data and output data associated with the training data; For each of the models except the model constructed during a first of the stages, the model is defined using the residuals associated with the model constructed during the stage prior to the stage in which the model was constructed.

3. The method according to claim 2, further comprising: The loss function is selected for determining the residual for each of the models for a random model for each of the models based on the stage associated with the model. The method according to claim 3 , wherein the loss function is a decorrelation function.

5. The method of claim 4, wherein a Ljung-Box loss function is used to select the loss function for at least one of the stages.

6. A method according to claim 4, wherein the model constructed during the first stage is a physically based model among the physically based models, and the model during the last stage in the stage is a machine learning model among the machine learning models.

7. The method of claim 1, wherein the one or more processors define a recurrent neural network.

8. The method of claim 7, wherein the machine learning model comprises one or more of a linear model, a dense model, a deep model, and an ARIMA model.

9. The method of claim 1, wherein the computer comprises at least one of a central processing unit and a graphics processing unit.

10. A system for micro-assembly control enabled by machine learning with a digital computer, comprising: at least one processor, the at least one processor being configured to: obtaining one or more parameters of a closed-loop system for locating one or more micro-objects of a plurality of micro-objects, the system comprising a plurality of programmable electrodes configured to induce movement of the micro-objects when the micro-objects are in proximity to the electrodes upon actuation of the electrodes, each of the actuations comprising generating one or more electrical potentials by one or more of the electrodes; obtaining training data including states associated with previous movements of one or more micro-objects of the plurality of micro-objects due to an earlier one of the actuations; defining one or more physics-based models for predicting a position of at least one of the micro-objects after one of the actuations; using the training data via gradient boosting to construct a hybrid model for predicting a position of at least one of the micro-objects after one of the actuations, the hybrid model comprising a plurality of models including one or more of the physics-based models and one or more machine learning models; Get the dataset; Dividing the data set into the training data and interpolated test data; extrapolating the state of the previous move to generate extrapolated data; testing the hybrid model using the interpolated test data and the extrapolated data; After the hybrid model passes the test, controlling the closed-loop system includes: predicting a position of one or more of the micro-objects after one of the actuations based on at least one sensor measurement; receiving another position of one or more of the micro-objects; and Another of the actuations is performed using the predicted position and the other position.

11. The system of claim 10, wherein the gradient boosting comprises a plurality of stages, wherein one of the models is constructed during each of the stages, the at least one processor being further configured to: for each of the models, determining, using a loss function, a residual between a prediction of the one model based on an input portion of the training data and output data associated with the training data; For each of the models except the model constructed during a first of the stages, the model is defined using the residuals associated with the model constructed during the stage prior to the stage in which the model was constructed.

12. The system of claim 11, wherein the at least one processor is further configured to: The loss function is selected for determining the residual for each of the models for a random model for each of the models based on the stage associated with the model.

13. The system of claim 12, wherein the loss function is a decorrelation function.

14. The system of claim 13, wherein a Ljung-Box loss function is used to select the loss function for at least one of the stages.

15. A system according to claim 13, wherein the model built during the first stage is a physically based model among the physically based models, and the model during the last stage in the stage is a machine learning model among the machine learning models.

16. The system of claim 10, wherein the one or more processors define a recurrent neural network.

17. The system of claim 16, wherein the machine learning model comprises one or more of a linear model, a dense model, a deep model, and an ARIMA model.

18. The system of claim 10, wherein the at least one processor is at least one of a central processing unit and a graphics processing unit.

19. A system for hybrid model-based microassembly control enabled by machine learning with a digital computer, comprising: at least one processor, the at least one processor being configured to: obtaining one or more parameters of a closed-loop system for locating one or more micro-objects of a plurality of micro-objects, the system comprising a plurality of programmable electrodes configured to induce movement of the micro-objects when the micro-objects are in proximity to the electrodes upon actuation of the electrodes, each of the actuations comprising generating one or more electrical potentials by one or more of the electrodes; obtaining training data including states associated with previous movements of one or more micro-objects of the plurality of micro-objects due to an earlier one of the actuations; defining one or more physics-based models for predicting a position of at least one of the micro-objects after one of the actuations; using the training data to construct a hybrid model for predicting a position of at least one of the micro-objects after one of the actuations via gradient boosting, the hybrid model comprising a plurality of models, the plurality of models comprising one or more of the physics-based models and one or more machine learning models, the gradient boosting comprising a plurality of stages, wherein one of the models is constructed during each of the stages, the at least one processor being further configured to: selecting a loss function, the loss function comprising a decorrelation function, for determining a residual for each of the models for a stochastic model for each of the models based on the stage associated with the model; for each of the models, determining, using a loss function, a residual between a prediction of the one model based on an input portion of the training data and output data associated with the training data; for each of the models except the model constructed during a first one of the stages, defining the model using the residuals associated with the model constructed during the stage prior to the stage in which the model was constructed; Controlling the closed-loop system, comprising: predicting a position of one or more of the micro-objects after one of the actuations based on at least one sensor measurement; receiving another position of one or more of the micro-objects; and Another of the actuations is performed using the predicted position and the other position.

Images