Diffusion optimal control of inverse problems

Abstract

A method for image generation with a diffusion process with optimal control is disclosed. Optimal control is applied in the inverse diffusion problem to mitigate processing costs associated with image generation. The method includes performing an outer loop of a predetermined number of iterations, with two inner loops (a backward propagation loop and a forward propagation loop) that compute a desired image over a plurality of time steps from a starting state (e.g., static noise or measurement) after completing the predetermined number of iterations.

Description

Technical Field

[0001] This disclosure relates to solving inverse problems for applications such as control signal generation and image generation, and more specifically, to incorporating optimal control into diffusion algorithms for generating control signals and / or images. Background Technology

[0002] Diffusion models are a class of generative models used in various applications, such as image generation. Diffusion models can implement a two-stage process involving forward diffusion and either backward or reverse diffusion. Forward diffusion gradually adds noise to the data, while backward diffusion removes the noise to reconstruct the original data (e.g., an image).

[0003] During forward diffusion, data points can be progressively transformed into noise by gradually or iteratively adding small amounts of Gaussian noise. The forward process can disrupt the data into a noisy distribution, which can be mathematically modeled using, for example, a Gaussian distribution.

[0004] The reverse diffusion process begins with a sample of pure noise, and the data is denoised step by step. As the data is denoised, it is reconstructed. This can be implemented by predicting the noise added at each step and subtracting the predicted noise. Summary of the Invention

[0005] A method for image generation utilizing a diffusion process with optimal control is disclosed. In various embodiments, the disclosed method uses optimal control in the inverse diffusion problem to mitigate the processing costs associated with image generation. The method includes an outer loop executed iteratively multiple times, wherein two inner loops (a backpropagation loop and a forward propagation loop) compute the desired image from a starting state (e.g., static noise or measurement) at multiple time steps. Attached Figure Description

[0006] Figure 1 A system 100 for training neural networks is shown.

[0007] Figure 2 A computer-implemented method 200 for training a neural network is shown.

[0008] Figure 3A and 3B The method of this disclosure is illustrated graphically.

[0009] Figures 4A-4C An example of image generation in this disclosure is illustrated relative to other methods.

[0010] Figure 5 A schematic diagram depicting the interaction between the computer-controlled machine 510 and the control system 512 is shown.

[0011] Figure 6 It depicts manufacturing machines (such as stamping and cutting machines, cutting machines, or gun drills) configured to control manufacturing systems (such as part of a production line). Figure 1 A schematic diagram of the control system.

[0012] Figure 7 Depicting a system configured to control an automated personal assistant Figure 1 A schematic diagram of the control system.

[0013] Figure 8 It describes a system configured to control and monitor (such as a control access system or a surveillance system). Figure 1 A schematic diagram of the control system.

[0014] Figure 9 It depicts a system configured to control an imaging system (such as an MM device, an X-ray imaging device, or an ultrasound device). Figure 1 A schematic diagram of the control system. Detailed Implementation

[0015] This document describes embodiments of the present disclosure. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various alternative forms. The drawings are not necessarily to scale; some features may be enlarged or reduced to show details of specific components. Therefore, the specific structural and functional details disclosed herein should not be construed as limiting, but merely as a representative basis for teaching those skilled in the art to utilize the embodiments in various ways. As will be understood by those skilled in the art, various features illustrated and described with reference to any of the drawings may be combined with features illustrated in one or more other drawings to produce embodiments not explicitly illustrated or described. The combinations of illustrated features provide representative embodiments of typical applications. However, for a particular application or implementation, various combinations and modifications of features consistent with the teachings of this disclosure may be desired.

[0016] As used herein, “one,” “an,” and “the” refer to both the singular and the plural of the referred to objects, unless the context clearly indicates otherwise. For example, a “processor” programmed to perform various functions refers to one processor programmed to perform each function, or more than one processor programmed together to perform each of the various functions.

[0017] Diffusion models have proven adept at conditional generation tasks, partly due to their iterative sampling algorithm, which, by introducing an additional guiding term u, allows for the sampling of uncontrolled prior score functions ∇. x log p t The dynamic guide (x) is directed to any posterior distribution. When the guide term is a conditional score ∇ x log p tWhen (y|x), the obtained sample can be proven to be obtained from the expected conditional distribution p(x|y). A major obstacle to this framework is obtaining the conditional score function ∇. x log p t The general difficulty of (y|x) is due to its dependence on the noisy, diffuse variable x. t This involves more than just the final sample x0. In large-scale conditional generation tasks, such as class or text conditional sampling, the computational cost of training time-dependent conditional score functions from scratch is considered acceptable. However, this solution is unacceptable in inverse problems, where the goal is to design a universal solver that will work for any forward model with zero-shot capacity.

[0018] This bottleneck has prompted a series of recent studies dedicated to improving conditional scoring. x log p t (y|x t The approximation is a simple logarithm of the noiseless likelihood function log p(y|x0). However, these approximations impose a significant cost on the performance of the resulting algorithm.

[0019] This disclosure utilizes the understanding that some of the costs discussed above can be mitigated by incorporating optimal control theory. Therefore, various approaches to solving the diffusion problem for image generation incorporate a framework built from optimal control theory, where such approximations are no longer necessary. By expressing the backdiffusion process as an optimal control problem, the inverse problem solver can be freed from the strict requirements of the given conditional sampling equations while still utilizing the prior knowledge of the unconditional diffusion process.

[0020] Optimal control is a class of algorithms derived from robotics, where the objective is to minimize a user-defined cost (or energy) function describing the task to be performed, with respect to the control trajectory under constraints of system dynamics (e.g., the system could be a robot or any autonomous agent). In this disclosure, optimal control is applied to an inverse problem, a class of problems where the objective is to recover a hidden or unknown variable x0 using only (potentially) noisy measurements y. More specifically, optimal control is applied to diffusion models. Diffusion models are a class of deep generative models that leverage the functional approximation capabilities of deep neural networks to learn a score function (mathematically derived from...). The scoring function (represented by the gradient of the data distribution) provides the direction in which the data is most likely to occur. Once trained, the scoring function can be used to sample from the learned data distribution (i.e., "generate" samples, or even new samples). Details and examples of this approach will be discussed further below.

[0021] Figure 1A system 100 for training a neural network (e.g., a deep neural network) is shown. The neural network or deep neural network shown and described is merely an example of the types of machine learning networks or neural networks that can be used. System 100 may include an input interface for accessing training data 102 for the neural network. For example, as Figure 1 As shown, the input interface can be composed of a data storage interface 104, which can access the training data 102 from the data storage device 106. For example, the data storage interface 104 can be a memory interface or a persistent storage device interface, such as a hard disk or SSD interface, but it can also be a LAN, local area network, or wide area network interface, such as a Bluetooth, Zigbee, or Wi-Fi interface, or an Ethernet or fiber optic interface. The data storage device 106 can be an internal data storage device of the system 100, such as a hard disk drive or SSD, but it can also be an external data storage device, such as a network-accessible data storage device.

[0022] In some embodiments, data storage device 106 may further include a data representation 108 of an untrained version of the neural network, which can be accessed by system 100 from data storage device 106. However, it should be understood that training data 102 and the data representation 108 of the untrained neural network may also be accessed from different data storage devices, for example, via different subsystems of data storage interface 104. Each subsystem may be of one type of data storage interface 104 as described above. In other embodiments, the data representation 108 of the untrained neural network may be generated internally by system 100 based on the design parameters of the neural network, and therefore may not be explicitly stored on data storage device 106.

[0023] System 100 may further include a processor subsystem 110, which may be configured to provide an iterative function as a replacement for a stack of layers of a neural network to be trained during operation of system 100. Here, the individual layers in the replaced stack may have mutually shared weights and may receive inputs such as the output of the previous layer, or, for the first layer in the stack, the initial activation and a portion of the stack's inputs. Processor subsystem 110 may be further configured to iteratively train the neural network using training data 102. Here, the training iterations of processor subsystem 110 may include a forward propagation portion and a backward propagation portion. Processor subsystem 110 may be configured to perform the forward propagation portion by determining an equilibrium point of the iterative function and other operations that may be performed to define the forward propagation portion, at which the iterative function converges to a fixed point.

[0024] System 100 may also include an output interface for outputting a data representation 112 of the trained neural network, which may also be referred to as trained model data 112. For example, as Figure 1 As shown, the output interface can be comprised of a data storage interface 104, which in these embodiments is an input / output (“IO”) interface through which trained model data 112 can be stored in data storage device 106. For example, the data representation 108 defining an “untrained” neural network can be at least partially replaced by the data representation 112 of the trained neural network during or after training, because the parameters of the neural network (such as the network weights, hyperparameters, and other types of parameters) can be adapted to reflect the training on the training data 102. This also... Figure 1 The reference numerals 108 and 112 indicate the same data records on the data storage device 106. In other embodiments, the data representation 112 may be stored separately from the data representation 108 defining the "untrained" neural network. In some embodiments, the output interface may be separate from the data storage interface 104, but it can typically be of the type described above for the data storage interface 104.

[0025] In various embodiments, the system for training the neural network can be implemented in a system for image generation, for example, using a diffusion model that incorporates optimal control within the image generation algorithm. The image generation system can be used in a wide variety of applications where it receives sensor signals and generates images based on the sensed data. Sensor types can include cameras (video or still images), radar, lidar, ultrasound, motion sensors, etc., and the images captured by the sensors can be used to calculate control signals for controlling a physical system. Such a physical system can include computer-controlled machines, such as robots, vehicles, home appliances, manufacturing machines, personal assistants, or access control systems. Systems for transmitting information can also be utilized. Figure 1 The systems shown are for training neural networks, including surveillance systems or medical (imaging) systems.

[0026] Figure 2 Systems 200 implementing the machine learning models described herein are depicted, such as the diffusion models discussed above and further below. Other types of machine learning models can be used, and the DNNs described herein are not the only type of machine learning model that can be used in the systems disclosed herein. For example, if the input image contains an ordered sequence of pixels after the CSI values ​​have been converted into pixels in the image, a CNN can be utilized.

[0027] System 200 can be implemented to perform one or more stages of the image recognition process described herein. System 200 may include at least one computing system 202. Computing system 202 may include at least one processor 204 operatively connected to storage unit 208. Processor 204 may include one or more integrated circuits implementing the functions of a central processing unit (CPU) 206. CPU 206 may be a commercially available processing unit implementing an instruction set such as ×86, ARM, Power, or MIPS instruction set families. During operation, CPU 206 may execute stored program instructions retrieved from storage unit 208. The stored program instructions may include software controlling the operation of CPU 206 to perform the operations described herein. In some examples, processor 204 may be a system-on-a-chip (SoC) that integrates the functions of CPU 206, storage unit 208, network interface, and input / output interface into a single integrated device. Computing system 202 may implement an operating system for managing various aspects of operation. Figure 2 The diagram shows a processor 204, a CPU 206, and a memory 208, but of course, more than one of each of these can be used in the entire system.

[0028] Storage unit 208 may include volatile and non-volatile memory for storing instructions and data. Non-volatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computing system 202 is disabled or loses power. Volatile memory may include static and dynamic random access memory (RAM) for storing program instructions and data. For example, storage unit 208 may store a machine learning model 210 or algorithm, a training dataset 212 for the machine learning model 210, and the original source dataset 216.

[0029] The computing system 202 may include a network interface device 222 configured to provide communication with external systems and devices. For example, the network interface device 222 may include wired and / or wireless Ethernet interfaces defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards. The network interface device 222 may include a cellular communication interface for communicating with cellular networks (e.g., 3G, 4G, 5G). The network interface device 222 may be further configured to provide a communication interface to an external network 224 or the cloud.

[0030] External network 224 may be referred to as the World Wide Web or the Internet. External network 224 can establish standard communication protocols between computing devices. External network 224 can allow information and data to be easily exchanged between computing devices and the network. One or more servers 230 can communicate with external network 224.

[0031] The computing system 202 may include an input / output (I / O) interface 220, which may be configured to provide digital and / or analog inputs and outputs. The I / O interface 220 is used to transfer information between internal storage devices and external input and / or output devices (e.g., HMI devices). The I / O interface 220 may include associated circuitry or a bus (BUS) network for transferring information to or between one or more processors and storage devices. For example, the I / O interface 220 may include digital I / O logic lines that can be read or set by one or more processors, handshake lines for monitoring data transmission via the I / O lines, timing and counting facilities, and other known structures that provide these functions. Examples of input devices include keyboards, mice, sensors, etc. Examples of output devices include monitors, printers, speakers, etc. The I / O interface 220 may include an additional serial interface (e.g., a Universal Serial Bus (USB) interface) for communicating with external devices. The I / O interface 220 may be referred to as an input interface (because it transmits data from external inputs, such as sensors) or an output interface (because it transmits data to external outputs, such as displays).

[0032] The computing system 202 may include a human-machine interface (HMI) device 218, which may include any device that enables the system 200 to receive control input. Examples of input devices may include human interface inputs such as a keyboard, mouse, touchscreen, voice input device, and other similar devices. The computing system 202 may include a display device 232. The computing system 202 may include hardware and software for outputting graphical and textual information to the display device 232. The display device 232 may include an electronic display screen, projector, printer, or other suitable device for displaying information to a user or operator. The computing system 202 may also be configured to allow interaction with remote HMIs and remote display devices via a network interface device 222.

[0033] System 200 can be implemented using one or more computing systems. While this example depicts a single computing system 202 implementing all the described features, the intention is that various features and functions can be implemented separately and by multiple computing units that communicate with each other. The specific system architecture chosen can depend on a variety of factors.

[0034] System 200 may implement machine learning algorithm 210, which is configured to analyze raw source dataset 216. Raw source dataset 216 may include raw or unprocessed sensor data, which may represent the input dataset for the machine learning system. Raw source dataset 216 may include video, video clips, images, text-based information, audio or human speech, time-series data (e.g., pressure sensor signals varying over time), raw or partially processed sensor data (e.g., radar images of objects), and wireless signals according to CSI, RSSI, or CIR. Furthermore, raw source dataset 216 may be input data derived from associated sensors, such as cameras, lidar, radar, ultrasonic sensors, motion sensors, thermal imaging cameras, wireless receivers, or any other type of sensor that produces associated data with spatial dimensions, where some concept of "foreground" and "background" exists within those spatial dimensions. The input or input "image" mentioned herein does not necessarily originate from a camera but may originate from any of the sensors listed above. Several different input examples are shown and described with reference to the accompanying figures discussed below. In some examples, the machine learning algorithm 210 may be a neural network algorithm (e.g., a deep neural network) designed to perform a predetermined function. For example, the neural network algorithm may be configured to identify defects (e.g., cracks, stress, bulges, etc.) in a part after it has been manufactured but before it leaves the factory.

[0035] Computer system 200 may store a training dataset 212 for machine learning algorithm 210. Training dataset 212 may represent a set of previously constructed data used to train machine learning algorithm 210. Machine learning algorithm 210 can use training dataset 212 to learn weighting factors associated with a neural network algorithm. Training dataset 212 may include a set of source data that has a corresponding output or result that machine learning algorithm 210 attempts to replicate through the learning process.

[0036] Machine learning algorithm 210 can be operated in learning mode using training dataset 212 as input. Machine learning algorithm 210 can be executed iteratively multiple times using data from training dataset 212. With each iteration, machine learning algorithm 210 can update its internal weighting factors based on the obtained results. For example, machine learning algorithm 210 can compare the output (e.g., a reconstructed or supplemented image given image data as input) with those results included in training dataset 212. Since training dataset 212 includes expected results, machine learning algorithm 210 can determine when performance is acceptable. After machine learning algorithm 210 reaches a predetermined performance level (e.g., 100% consistency with outputs associated with training dataset 212) or converges, machine learning algorithm 210 can be executed using data not in training dataset 212. It should be understood that in this disclosure, "convergence" can mean that a set (e.g., predetermined) number of iterations has occurred, or the residual is sufficiently small (e.g., the change in approximate probability during iterations changes in a manner less than a threshold), or other convergence conditions. The trained machine learning algorithm 210 can be applied to new datasets to generate annotated data.

[0037] Machine learning algorithm 210 can be configured to identify specific features in raw source data 216. Raw source data 216 may include multiple instances or input datasets expected to supplement the results. For example, machine learning algorithm 210 can be configured to identify the presence of pedestrians in video images and annotate these occurrences. In another example, machine learning algorithm 210 can be configured to identify the presence of defects in a manufactured part by capturing an image of that part. Machine learning algorithm 210 can be programmed to process raw source data 216 to identify the presence of specific features. Machine learning algorithm 210 can be configured to identify features in raw source data 216 as predetermined features (e.g., obstacles, pedestrians, road signs, etc.). Raw source data 216 can be derived from various sources. For example, raw source data 216 can be actual input data collected by a machine learning system. Raw source data 216 can be machine-generated for testing systems. For example, raw source data 216 may include raw video images from a camera.

[0038] Figure 3A and 3B The method of this disclosure is illustrated graphically. Figure 3A An algorithm is illustrated graphically, where, in the initial state x TTrajectories are calculated between the final states x', x”, ..., x0, where x0 represents a desired image based on, for example, sensing data (using an optical sensor). Each calculated trajectory represents one iteration of the algorithm, which can be executed a predetermined number of iterations. Each predetermined trajectory includes the execution of first and second inner loops, which include a backpropagation loop and a forward propagation loop. The algorithm shown in this paper can use data obtained from sensors for various applications, where the generated or recovered images are used to generate control signal outputs, such as for process control. The algorithm is shown below and will be explained in further detail below.

[0039] The algorithm shown above starts with a "nominal" trajectory, an unguided process that produces a random image derived from the modeled distribution. The goal is to guide the value of x at each step t so that x0 looks like a solution to our inverse problem. That is, given a measurement y = A(x*) + noise (where x* is the actual data), we want x0 to be as close to x* as possible. Three-loop framework: Algorithm 1 has three loops. The first loop is the main loop of the algorithm and can be executed for a predetermined number of iterations (num_iters). However, it is possible and considered to execute this loop until a convergence value is reached. The second and third loops are inner loops relative to the main loop.

[0040] Given a trajectory, the execution of the second loop in our algorithm generates the gradient and all the main mathematical objects (V). x V xx k t K t (See below). Note that, by convention, time runs backward during the diffusion process, i.e., t = T, T-1, ..., t, t-1, ... 0, as reflected in the second inner loop. Value k t It is the feedforward gain, and K t It is feedback gain. The subscript t is used to represent the time exponent t, meaning that the mathematical object is time-varying, and the subscript refers to the value taken by the object at a specific time t. V x and V xx These are the first and second derivatives of the valued function, respectively, and are also time-varying. The calculation of these terms will be discussed in further detail below. The second loop, executed from t = 1 to T, moves forward in time, where T is the total time.

[0041] In the third loop, use k t and K T The calculated value, the third loop generates a new set of actions. u tand determine from x T The initial corresponding state trajectory, such as Figure 3A As shown. Value λ It is the step size value. u t It is the control vector value of the time index, which is used to perturb at the corresponding time step. x t The result after all iterations of the outer loop is... x T The value and x 0 The trajectory between the values, such as Figure 3A As shown, this represents the final image obtained from executing the algorithm. The development of this algorithm and the computations involved will now be discussed in more detail.

[0042] Diffusion model: Diffusion models are a type of deep generative model that utilizes the function approximation ability of deep neural networks to learn a score function, mathematically represented as: , It provides the direction of highest probability for the data (i.e., the gradient of the data distribution). Once trained, the score function can be used to sample from the learned data distribution (i.e., "generate" samples, or even new samples). This is achieved using the inverse-time Itö stochastic differential equation (Itö-SDE) given by the following equation: ， (1) in, It is a state vector (in the case of image data, this is the flattened vector of the 3D RGB image tensor). t It is a continuous time variable used to represent the passage of time, because diffusion is a continuously evolving process, and therefore... x t Represents the state vector in time t The value at that location. Vector-valued function. It is called the drift vector function, and g(t) This is called the diffusion variable function. The drift vector controls signal attenuation, while the diffusion variable controls noise injection during the process. Finally, the actual noise variable is... w tIt is mathematically a vector of mutually independent Brownian motions. The reason this is an inverse-time SDE rather than a forward-time SDE is because the diffusion model is trained to reverse a certain forward process. Specifically, the forward process is a noise-adding (or destructive) process that iteratively destroys the signal (i.e., iteratively increases the signal-to-noise ratio) over many time steps using additive Gaussian noise until we are left with noisy samples that can essentially be assumed to have a Gaussian distribution. The diffusion model is then trained to reverse this destructive process and, starting with noisy samples taken from a Gaussian distribution, provides “clean” (i.e., noise-free) samples from the underlying data distribution. This is why the generation or sampling process of the trained diffusion model is an inverse-time SDE, because one starts by sampling from a Gaussian distribution and then proceeds backward in time as given by equation (1) above, iteratively removing noise from the samples using the learned score function.

[0043] Although continuous-time processes exist in nature, they need to be discretized for implementation on a computer. For this purpose, we begin with the so-called probabilistic flow ODE (PF-ODE). This ODE is given by the following equation: ， (2) It has the same edge distribution as SDE p t ( x t Practical implementations of the diffusion sampler may require time discretization of the PF-ODE. One such discretization is the well-known Euler discretization, which gives: ， (3) Here, ∆t is the length of the discretization interval, and we reverse the time evolution by changing the sign of the drift. We are not limited to using only Euler discretization, but can also employ any higher-order discretization technique. More precisely, this leads to: x t−1 = h(x t ), where h: (4) It describes the general nonlinear dynamics of the corresponding discrete-time diffusion sampler.

[0044] Inverse problem: The inverse problem is a class of problems whose goal is to use only (potentially) noisy measurements. y To recover hidden unknown variables x 0 The measured values ​​are correlated with the unknown variables via a forward measurement process, and are given by the following equation: (5) in, It is the so-called forward operator, and It has variance Gaussian noise variables.

[0045] Assuming access to a pre-trained, off-the-shelf diffusion model is available, this model has already been used with the aforementioned unknown variables. x 0 The model was trained on large-scale, publicly available data of the same type (such as models available on the internet, e.g., stable diffusion models). Therefore, people can access the distribution in the form of the diffusion model. p(x) (We remove the subscript 0 to imply a general distribution, relative to a specific sample from the distribution that will be represented by the subscript), which can be used to sample instances using the above equation (1), i.e. x 0 ~ p(x) Mathematically, given a value corresponding to something (potentially) unseen during the training of the diffusion model, x 0 Measured values y , restore the unknown x 0 The goal mathematically corresponds to from p(x|y) The posterior distribution is sampled.

[0046] Given forward model equation 5 and measurements y From the posterior distribution p(x|y) Sampling can then be performed by solving the conditional Iton-SDE: ， (6) This involves invoking Bayesian rules: , (7).

[0047] Similar to unconditional dynamics, equation 6 has a corresponding ODE: ， (8) It has an approximate solution obtained through Euler discretization: 。 (9) Optimal control: Optimal control is a class of algorithms derived from robotics, where the objective is to minimize a user-defined cost (or energy) function that mathematically describes the task to be performed, given the control trajectory under the constraints of the system's dynamics (e.g., the system could be a robot or any autonomous agent). Specifically, starting from the current state, the cost function is computed along the trajectory of the system's future states, obtained by performing the current iteration of the control trajectory on a dynamic model of the system.

[0048] These algorithms assume the availability of a dynamic model, which can be a physics-based model, a data-driven model, or a combination of both. Therefore, these algorithms are also known as trajectory optimization (or simply TrajOpt) algorithms, and their output (after convergence, i.e., after the trajectory cost has been minimized) is the optimal control trajectory, which, when executed on a real system, will lead to optimal behavior with respect to the cost function.

[0049] The TrajOpt algorithm commonly used in the literature is called the Iterative Linear Quadratic Regulator (iLQR). This algorithm has been applied to control various systems in different fields, such as robotics, biology, aerospace systems, and quantitative finance. The iLQR algorithm uses a first-order approximation of the system's dynamic model and a second-order approximation of the value function.

[0050] When applying iLQR to problems in machine learning (ML), an ML algorithm or one of its sub-components can be viewed as a discrete-time dynamic system to be controlled. TrajOpt aims to improve the performance of algorithms by using control, thereby transferring the successful use of iLQR and tools developed from other scientific research fields, such as robotics, to the ML domain.

[0051] The above example of a dynamic system encountered in diffusion modeling is given by equation (9). This is an uncontrolled dynamic system because no control is added (i.e., no modification is made to equation (9)). This disclosure allows the original algorithm to be restored. The TrajOpt algorithm can be used to change the evolution of the inverse-time dynamics of the diffusion model, thereby changing the final clean image obtained from the generation (or sampling) process. x 0 This is achieved by adding a control term to the right-hand side of Equation 9, designing the cost function to be minimized, and using the TrajOpt algorithm to compute the optimal control trajectory, which minimizes the cost function under the constraints of the control modification of Equation 9. Now, let's explain iLQR in more detail.

[0052] In a broad sense, a cost function is a mathematical formula used to quantify the "cost" or "error" of a particular solution or model, typically used in optimization problems and machine learning. For the purposes of this disclosure, a user-defined global cost function can be written as follows: (10) The scalar operating cost function and the terminal cost function ℓ t The sum of ℓ0 and its components. Optimal control theory defines the value function. The following recursive relation, known as Bellman's Principle of Optimality, is satisfied: (11) The iLQR algorithm is centered on an approximate state-action value function. (12) Therefore, the value function can be restored to... .

[0053] Then, given the state transition function x t = h (x t+1 , u t+1 ), where we note that when time is defined as flowing backward from t = T, ..., 0, the iLQR algorithm has respectively k t and K t The feedforward and feedback gains are defined as follows: 。 (13) Therefore, the update of the value function can be written as: 。 (14) Given feedback and feedforward gain and For each time step t, a new optimal control trajectory can be obtained recursively, which serves as the state. x t and control u t The current version of the function is as follows: (15) . (16) optimal control of diffusion: As mentioned above, in the context of applying the TrajOpt algorithm to improve the performance of deep generative models, equation (9) can be viewed as an uncontrolled discrete-time dynamic system. Since it represents the generative process, it starts from the initial time... t = T Start to final time t = 0It evolves backwards in time. The TrajOpt algorithm is used to compute the algorithm for... t = { T , T-1 , T- 2 The optimal control vector of {,…,1} The trajectory makes it possible for each time step to... t = { T , T-1 , T-2 When the position ,…,1} is injected into the dynamics of equation (9), it will t The final clean image is obtained at point = 0, which makes... x 0 ~ p(x|y) That is, the unknown variables can be recovered with high precision. There are two possible ways to inject control into equation (9): 1. In input disturbance control, we apply [the following] before the diffusion step. u t : (17) 2. In output disturbance control, it is applied after the diffusion step. u t : (18) It is observed that iLQR can be formulated for general discrete-time dynamic processes. When specifically applied to the back-diffusion dynamics of a diffusion model, several simplifications can be performed. First, it can be assumed that there is no access to any guide except at time t = 0—that is, ℓ t ( x t , u t (Not dependent on) x t .

[0054] Under input disturbance control, it can be observed from Equation 17 that... h x = h u The output disturbance control situation implies h u = I This leads to the equations on the left and right sides, respectively: Note that in equations 19-23 above, the bold terms represent partial derivatives. For example, yes Regarding vectors partial derivatives or about gradient, yes about Hessian, and, yes about Subsequently The second-order partial derivatives. Note that, to avoid cluttered comments and for the sake of brevity, the time index... t The items have been removed from all the items above, but each of the above items is time-varying and depends on the time index. t .

[0055] Then, the derivative of V can be backpropagated using the following equation: In high-dimensional systems such as Equation 3, matrices can be singular. Therefore, a Tikhonov regularization variant of iLQR can be employed, where the matrix inverse is regularized by a diagonal matrix.

[0056] Figure 3B This is another graphical illustration of the method disclosed herein. Specifically, Figure 3B Compare this method (on the right) with the calculation using Tweedie's Formula. Compared with previous methods, It is true Approximation for computation The method disclosed herein alternatively uses precise... This leads to A more accurate calculation. In general, such as... Figure 3B As shown, the final clean image generated on the right. (Using the disclosed method) it is closer to the true unknown variable than the one on the left. .

[0057] Figure 3B It was also shown on the left and right sides. x T and The corresponding trajectories between them. As a result of this method, the trajectory changes less on the right side than on the left, leading to a more accurate final image.

[0058] Figure 4A The illustration shows a first example of image generation in this disclosure relative to another method. Figure 3A and 3B The abstract things in Figure 4A The text is qualitatively shown, where the use of the Tweedy formula leads to the predicted... A less precise approximation, while the prediction Considered to be around the true value "Jumping". This is on top. Figure 4A The different ones are visualized in the middle. Images representing different people. On the other hand, the disclosed method can lead to accurate data from any diffusion time step. This is because the published method relies on unfolding the actual diffusion model dynamics to obtain the true... Instead of relying on the Tweedy formula.

[0059] As derived in the accompanying documentation, this method establishes the theoretical results given by the statement of the following theorem: Assume that Equation 3 is a discretized sampling equation (Equation 18) of a diffusion model with output disturbance mode control. Furthermore, let the terminal cost... , (27) It is differentiable twice and has an operating cost of , (28) Then, an iterative linear quadratic regulator with a Tikhonov regularization term α generates control: 。 (29) As shown above, in the disclosed method, the dynamics used for generation via the diffusion model are deterministic (unlike the stochastic dynamics used in previous work), leading to The above theorem shows that, with an appropriate choice of the Tikhonov regularization constant α, the published method can recover the true posterior sampler. Therefore, for the conditional score function... The approximation is unnecessary.

[0060] Figure 4B A second example of image generation in this disclosure is illustrated, relative to another method. Figure 4B The illustration further illustrates that the method of this disclosure is unaffected by the discretization of the generating equation.

[0061] The equations used for the generation process in the diffusion model are discrete (in time) versions of the continuous-time equations. As explained above, Equation 3 is a discrete-time version of the continuous-time equation 2. This makes it possible to implement the diffusion model on a digital computer. However, the accuracy of the generation process depends on the number of discretization steps. The more steps, the more accurate the generation process. This problem is amplified when using the methods listed in the prior art section to solve the inverse problem using the diffusion model. This is because the prior art relies on approximating the Tweedy formula. x 0In order to calculate the conditional score function When the number of discretization steps is large, there are more opportunities to correct previous steps that use poor approximations of x0. However, when the number of discretization steps is small, poor approximations lead to large errors, thus yielding results such as... Figure 4B The top row shows the unknown variables. x 0 Inaccurate predictions.

[0062] On the other hand, the method disclosed herein is not affected by the number of discretization steps because it uses the discretized generative equations and control trajectories to “unfold” or perform dynamics and obtain the true form. x 0 Therefore, even if the discretization steps are very few (e.g., in...), Figure 4B In the first column (T=4), the disclosed method can also calculate the necessary control disturbances in a small number of steps to obtain the unknown variables. x 0 An accurate estimate.

[0063] Figure 4C A third example of image generation in this disclosure, relative to another method, is illustrated. Furthermore, Figure 4C The illustration shows that the method of this disclosure is not affected by the quality of the diffusion model (i.e., the trained score function).

[0064] Many other methods rely on the Tweedy formula, which in turn relies on a scoring function to approximate the result. Therefore, a poor-quality diffusion model (i.e., a poor-quality score function) will lead to... The poor approximation affects the solution of the inverse problem. Because the disclosed method relies on trajectory optimization, which computes the optimal control trajectory for any generalized dynamic system, it can even work with fully stochastic diffusion models. This is because stochastic diffusion models also define a unique dynamic system, and the trajectory optimization algorithm then finds the optimal control trajectory to solve the inverse problem under those unique dynamic constraints. This is in Figure 4C It is described in the text.

[0065] Figure 4C The illustration shows a measured image (leftmost) and a corresponding real image (rightmost) with the results of the disclosed method (“Disclosed”) and another method. However, the disclosed method results in a fairly accurate depiction of the real image (and close to the measured image), while the other method (“Other”) produces essentially static noise.

[0066] The method disclosed herein has wide applications. It can be implemented in various types of systems that receive signals from various types of sensors, such as video, radar, lidar, ultrasound, motion, etc. Using the generated images, one or more control signals can be calculated and generated to control physical systems, such as various types of computer-controlled machines. Such machines can include robots, vehicles, home appliances, power tools, manufacturing machines, personal assistants, or access control systems.

[0067] This disclosure also contemplates systems in which the disclosed methods are used to calculate and generate control signals within systems used for transmitting information. This could include surveillance systems, medical imaging systems, surveying systems (e.g., for surveying underwater environments using acoustic and / or magnetic sensors), etc.

[0068] The disclosed methods can operate in a variety of applications by solving image enhancement tasks, such as image super-resolution, image inpainting / filling, and image deblurring (Gaussian blur or motion blur). This could be useful in various medical imaging applications such as CT scans. The ability to solve more complex inverse problems could also be useful in applications such as X-ray crystallography and transmission electron microscopy.

[0069] In a manufacturing environment, the disclosed methods can be used to potentially reduce the cost of sensors and ultimately the cost of manufactured goods. We might need as few as a single high-fidelity sensor to record the dataset and train a diffusion model to serve as our prior model. Cheaper sensors can then be used, and their signals can be combined with the pre-trained diffusion prior model and the methods discussed above to produce high-fidelity signals that would be obtained using expensive high-fidelity sensors. When new assembly lines are established in new factories, and for future maintenance, where sensors may need to be replaced and potentially with cheaper sensors, this can lead to improved cost-effectiveness.

[0070] Figure 5 A schematic diagram depicting the interaction between a computer-controlled machine 500 and a control system 502 is provided. The computer-controlled machine 500 includes an actuator 504 and a sensor 506. The actuator 504 may include one or more actuators, and the sensor 506 may include one or more sensors. The sensor 506 is configured to sense the condition of the computer-controlled machine 500. The sensor 506 may be configured to encode the sensed condition into a sensor signal 508 and transmit the sensor signal 508 to the control system 502. Non-limiting examples of the sensor 506 include wireless receivers, video, radar, lidar, ultrasonic, and motion sensors, as referenced above. Figure 1-2 As described above, embodiments combining different sensors are also possible and have been considered.

[0071] In various embodiments, sensor 506 may be a sensor type capable of capturing raw image data. Using methods disclosed elsewhere herein, control system 502 and / or computer-controlled machine 500 may generate a final image from the measured raw image data, which reflects the sensed information at a desired level of accuracy. In various embodiments, this image may be used by control system 502 to generate one or more control signals to perform its intended control functions. Images generated for further analysis (e.g., in medical imaging) are another possible and considered application.

[0072] The control system 502 is configured to receive sensor signals 508 from the computer-controlled machine 500. As described below, the control system 502 can also be configured to calculate actuator control commands 510 based on the sensor signals and transmit the actuator control commands 510 to the actuator 504 of the computer-controlled machine 500.

[0073] like Figure 5 As shown, the control system 502 includes a receiving unit 512. The receiving unit 512 can be configured to receive sensor signals 508 from sensor 506 and convert the sensor signals 508 into input signals x. In an alternative embodiment, the sensor signals 508 are received directly as input signals x, without a receiving unit 512. Each input signal x can be a portion of each sensor signal 508. The receiving unit 512 can be configured to process each sensor signal 508 to generate each input signal x. The input signal x can include data corresponding to the image recorded by sensor 506.

[0074] The control system 502 includes a classifier 514. The classifier 514 can be configured to classify an input signal x into one or more labels using a machine learning (ML) algorithm, such as the neural network described above. The classifier 514 is configured to be parameterized by parameters, such as those described above (e.g., parameter θ). The parameter θ can be stored in and provided by a non-volatile storage device 516. The classifier 514 is configured to determine an output signal y from the input signal x. Each output signal y includes information assigning one or more labels to each input signal x. The classifier 514 can transmit the output signal y to a conversion unit 518. The conversion unit 518 is configured to convert the output signal y into an actuator control command 510. The control system 502 is configured to transmit the actuator control command 510 to an actuator 504, which is configured to drive a computer-controlled machine 500 in response to the actuator control command 510. In another embodiment, actuator 504 is configured to drive computer-controlled machine 500 directly based on output signal y.

[0075] When actuator 504 receives actuator control command 510, actuator 504 is configured to perform an action corresponding to the relevant actuator control command 510. Actuator 504 may include control logic configured to translate actuator control command 510 into a second actuator control command for controlling actuator 504. In one or more embodiments, instead of or attached to the actuator, actuator control command 510 may also be used to control a display.

[0076] In another embodiment, the computer-controlled machine 500 may include a sensor 506, and the control system 502 may also include a sensor 506. The computer-controlled machine 500 may also include an actuator 504, and the control system 502 may also include an actuator 504.

[0077] like Figure 5 As shown, the control system 502 also includes a processor 520 and a memory 522. The processor 520 may include one or more processors. The memory 522 may include one or more storage devices. A classifier 514 of one or more embodiments (e.g., machine learning algorithms, such as those described above with respect to pre-trained classifier 306) may be implemented by the control system 502, which includes a non-volatile storage device 516, a processor 520, and a memory 522.

[0078] Non-volatile storage device 516 may include one or more persistent data storage devices, such as hard disk drives, optical disk drives, magnetic tape drives, non-volatile solid-state devices, cloud storage, or any other device capable of persistently storing information. Processor 520 may include one or more devices selected from a high-performance computing (HPC) system, including high-performance cores, microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field-programmable gate arrays, programmable logic devices, state machines, logic circuits, analog circuits, digital circuits, or any other device that manipulates signals (analog or digital) based on computer-executable instructions residing in memory 522. Memory 522 may include a single memory device or multiple memory devices, including but not limited to random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information.

[0079] Processor 520 can be configured to read computer-executable instructions into memory 522 and execute them, the computer-executable instructions residing in non-volatile storage device 516 and embodying one or more ML algorithms and / or methods of one or more embodiments. Non-volatile storage device 516 may include one or more operating systems and applications. Non-volatile storage device 516 may store programs compiled and / or interpreted from computer programs created using various programming languages ​​and / or technologies, including but not limited to Java, C, C++, C#, Objective C, Fortran, Pascal, JavaScript, Python, Perl, and PL / SQL, individually or in combination.

[0080] When executed by processor 520, the computer-executable instructions of non-volatile storage device 516 may cause control system 502 to implement one or more of the ML algorithms and / or methods disclosed herein. Non-volatile storage device 516 may also include ML data (including data parameters) supporting the functionality, features, and processes of one or more embodiments described herein.

[0081] Program code embodying the algorithms and / or methods described herein can be distributed individually or collectively as a program product in a variety of different forms. The program code can be distributed using a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to perform aspects of one or more embodiments. Inherently non-transitory computer-readable storage media can include tangible media implemented in any method or technology that are volatile and non-volatile, and removable and non-removable, for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media can also include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technologies, portable optical disc read-only memory (CD-ROM) or other optical storage devices, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be read by a computer. Computer-readable program instructions can be downloaded from the computer-readable storage medium to a computer, another type of programmable data processing device, or another device, or downloaded via a network to an external computer or external storage device.

[0082] Computer-readable program instructions stored in a computer-readable medium can be used to direct a computer, other type of programmable data processing apparatus, or other device to operate in a particular manner, causing the instructions stored in the computer-readable medium to produce an article of art that includes instructions to implement the functions, actions, and / or operations specified in a flowchart or diagram. In some alternative embodiments, according to one or more embodiments, the functions, actions, and / or operations specified in the flowchart and diagram can be reordered, processed sequentially, and / or processed concurrently. Furthermore, any flowchart and / or diagram may include more or fewer nodes or blocks than illustrated according to one or more embodiments.

[0083] These processes, methods, or algorithms can be implemented in whole or in part using appropriate hardware components, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices, or a combination of hardware, software, and firmware components.

[0084] Figure 6 A schematic diagram of a control system 502 is depicted, which is configured to control a system 600 (e.g., a manufacturing machine), such as a stamping and cutting machine, a cutting machine, or a gun drill, of a manufacturing system 602 (such as a part of a production line). The control system 502 may be configured to control an actuator 504, which is configured to control the control system 600 (e.g., the manufacturing machine).

[0085] Sensor 506 of system 600 (e.g., a manufacturing machine) may be an optical sensor (such as those described above) configured to capture one or more attributes of the manufactured product 604. Classifier 514 may be configured to determine the state of the manufactured product 604 from one or more of the captured attributes. Actuator 504 may be configured to control system 600 (e.g., a manufacturing machine) based on the determined state of the manufactured product 604 for subsequent manufacturing steps of the manufactured product 604, or to discard the manufactured product 604 if it has detected defects (e.g., discard, sort, mark, trim, or repair). Actuator 504 may be configured to control system 600 (e.g., a manufacturing machine) on subsequent manufactured products 606 of system 600 (e.g., a manufacturing machine) based on the determined state of the manufactured product 604.

[0086] In some embodiments, sensor 506 may be coupled to receive and process raw image data, and control system 502 may use the information obtained therefrom to generate control signals to control the process performed by system 600. More specifically, system 600 may use the raw image data collected by the sensor to generate images according to methods described elsewhere herein to control the process.

[0087] Figure 7 A schematic diagram is depicted of a control system 502 configured to control an automated personal assistant 700. The control system 502 can be configured to control an actuator 504, which is also configured to control the automated personal assistant 700. The automated personal assistant 700 is configured to process image data and act accordingly. For example, the automated personal assistant may be able to move autonomously and perform control over that movement based on image data collected from sensors and processed according to the methods of this disclosure. Sensors used to collect image data may include lidar, video, acoustic, wireless, and any other type of sensor from which images can be generated.

[0088] Sensor 506 may be an optical sensor and / or an audio sensor. The optical sensor may be configured to receive video images of the user 702's posture 704. The audio sensor may be configured to receive voice commands from the user 702. Sensor 506 may also be able to record other types of image data, such as thermal imaging.

[0089] The control system 502 of the automated personal assistant 700 can be configured to determine actuator control commands 510 for controlling the movement of the automated personal assistant 700. The control system 502 can be configured to determine the actuator control commands 510 based on sensor signals 508 from sensor 506. The automated personal assistant 700 is configured to transmit sensor signals 508 to the control system 502. A classifier 514 of the control system 502 can be configured to execute a gesture recognition algorithm to identify a gesture 704 made by the user 702 to determine the actuator control commands 510, and transmit the actuator control commands 510 to the actuator 504. The classifier 514 can be configured to retrieve information from non-volatile storage in response to gesture 704 and output the retrieved information in a form suitable for the user 702 to receive. The control system 502 can also be configured to execute various diffusion modules described herein to generate an image for generating control signals, which can be translated into various commands in the actuator control commands 510.

[0090] Figure 8 A schematic diagram of a control system 502 configured to control a monitoring system 800 is depicted. The monitoring system 800 can be configured to physically control access through a door 802. A sensor 506 can be configured to detect scenarios related to determining whether access is permitted. The sensor 506 can be an optical sensor configured to generate and transmit image and / or video data. The control system 502 can use this data to detect people and can use the image data acquired and processed according to the methods discussed above for personal identification.

[0091] The classifier 514 of the control system 502 of the monitoring system 800 can be configured to determine a person's identity by interpreting image and / or video data by matching the identity of a known person stored in non-volatile storage 516. The classifier 514 can be configured to generate an actuator control command 510 in response to the interpretation of the image and / or video data. The control system 502 is configured to transmit the actuator control command 510 to an actuator 504. In this embodiment, the actuator 504 can be configured to lock or unlock a door 802 in response to the actuator control command 510. In other embodiments, non-physical logical access control is also possible. In some embodiments, the classifier 514 can be configured to generate actuator control commands based on the identification of a specific person, which is based on wireless signals received by a wireless receiver in the sensor 506. For example, the classifier 514 can generate a command to cause the actuator 504 to adjust the temperature setting based on the specific person identified by the wireless signals.

[0092] The monitoring system 800 can also be a surveillance system. In such an embodiment, sensor 506 can be an optical sensor or a wireless receiver and / or wireless transmitter, configured to detect the scene under surveillance, and control system 502 is configured to control display 804. Classifier 514 is configured to determine the classification of the scene, such as whether the scene detected by sensor 506 is suspicious. Control system 502 is configured to transmit actuator control command 510 to display 804 in response to classification. Display 804 can be configured to adjust the displayed content in response to actuator control command 510. For example, display 804 can highlight objects deemed suspicious by classifier 514. Using embodiments of the disclosed system, the surveillance system can predict the appearance of objects at specific times in the future.

[0093] Figure 9 A schematic diagram is depicted of a control system 502 configured to control an imaging system 900, such as a milling device, an X-ray imaging device, or an ultrasound device. A sensor 506 may be, for example, an imaging sensor. A classifier 514 may be configured to determine a classification of all or a portion of the sensed image. The classifier 514 may be configured to determine or select an actuator control command 510 in response to a classification obtained by a trained neural network. For example, the classifier 514 may interpret a region of the sensed image as a potential anomaly. In this case, an actuator control command 510 may be determined or selected to cause the display 902 to display the image and highlight the potentially anomalous region.

[0094] In various embodiments, sensor 506 may also include circuitry for receiving and processing wireless signals. Classifier 514 may use the wireless signals to identify a specific person. Based on this identification, classifier 514 may cause the generation of one or more commands to control imaging system 900.

[0095] While exemplary embodiments have been described above, this does not mean that these embodiments describe all possible forms covered by the claims. The language used in this specification is descriptive and not restrictive, and it should be understood that various changes may be made without departing from the spirit and scope of this disclosure. As described above, features of various embodiments may be combined to form further embodiments of the invention that may not be explicitly described or illustrated. While various embodiments may have been described as providing an advantage or superiority over other embodiments or prior art implementations in one or more desired characteristics, those skilled in the art will recognize that one or more features or characteristics may be compromised to achieve desired overall system properties depending on the particular application and implementation. These properties may include, but are not limited to, cost, strength, durability, lifecycle cost, merchantability, appearance, packaging, size, suitability, weight, manufacturability, ease of assembly, etc. Therefore, any embodiment described to the extent that it is less desirable than other embodiments or prior art in one or more aspects is not outside the scope of this disclosure and may be ideal for a particular application.

Claims

1. A method for generating an image, the method comprising: Receive the step size value, state vector, and measurement value corresponding to the initial image; The first gradient value and the second gradient value are determined based on the measured values; From the first time value to the last time value, a first loop is executed in a first plurality of steps, each step having a specific increment size, wherein executing the first loop includes: Update the first gradient value and the second gradient value using the value of the state vector; and The feedforward gain and feedback gain are calculated at each of the first plurality of steps using the first gradient value and the second gradient value; and From the last time value to the first time value, a second loop is executed in a second plurality of steps, each having the step size value, wherein executing the second loop includes: The value of the state vector is recalculated using the feedforward gain value and the feedback gain value calculated during the execution of the first loop; and After recalculation, an updated value for the control vector is generated based on the value of the state vector; Repeat the first loop and the second loop a predetermined number of iterations; and After the first and second loops are repeated, a final image is generated based on the values ​​of the state vector and the control vector.

2. The method according to claim 1, further comprising: The control system generates one or more control signals based on the final image.

3. The method according to claim 2, wherein, The control system includes one or more sensors configured to generate the measured values.

4. The method according to claim 3, wherein, The one or more sensors include a video camera.

5. The method according to claim 3, wherein, The one or more sensors include one or more X-ray detectors.

6. The method of claim 3, wherein, The one or more sensors include lidar.

7. The method according to claim 2, further comprising: The control system controls the manufacturing process.

8. The method according to claim 2, further comprising: The control system controls the autonomous personal assistant.

9. The method according to claim 2, further comprising: The control system generates signal control signals for the actuator based on the final image.

10. The method according to claim 1, wherein, Executing the first loop includes executing the diffusion model.

11. A system for generating images based on measurement data, the system comprising: One or more sensors are configured to generate measurements corresponding to an initial image; The processor is configured as follows: The first gradient value and the second gradient value are determined based on the measured values; From a first time value to a last time value, a first loop is executed in a first plurality of steps, each step having a specific increment size, wherein, in order to execute the first loop, the processor is further configured to: Update the first gradient value and the second gradient value using the values ​​of the state vector; and The feedforward gain and feedback gain are calculated at each of the first plurality of steps using the first gradient value and the second gradient value; and From the last time value to the first time value, a second loop is executed in a second plurality of steps, each having a step size value, wherein, in order to execute the second loop, the processor is further configured to: The value of the state vector is recalculated using the feedforward gain value and the feedback gain value calculated during the execution of the first loop; and After recalculation, an updated value for the control vector is generated based on the value of the state vector; Repeat the first loop and the second loop a predetermined number of iterations; and After the first and second loops are repeated, a final image is generated based on the values ​​of the state vector and the control vector.

12. The system according to claim 11, further comprising: A control system configured to generate one or more control signals based on the final image.

13. The system according to claim 12, wherein, The control system is configured to generate one or more actuator control commands based on the one or more control signals.

14. The system of claim 11, wherein the one or more sensors include a video camera.

15. The system of claim 11, wherein the one or more sensors include an X-ray detector.

16. The system of claim 11, wherein the one or more sensors include lidar.

17. The system according to claim 11, wherein, During the execution of the first loop, the processor is configured to execute the diffusion model.

18. A non-transitory computer-readable medium storing instructions, which, when executed by a computer system, cause the computer system to perform operations, the operations including: Step size values, state vectors, and measurements corresponding to the initial image are received from one or more sensors coupled to the computer system; The first gradient value and the second gradient value are determined based on the measured values; From the first time value to the last time value, a first loop is executed in a first plurality of steps, each step having a specific increment size, wherein executing the first loop includes: Update the first gradient value and the second gradient value using the value of the state vector; and The feedforward gain and feedback gain are calculated at each of the first plurality of steps using the first gradient value and the second gradient value; and From the last time value to the first time value, a second loop is executed in a second plurality of steps, each having the step size value, wherein executing the second loop includes: The value of the state vector is recalculated using the feedforward gain value and the feedback gain value calculated during the execution of the first loop; and After recalculation, an updated value for the control vector is generated based on the value of the state vector; Repeat the first loop and the second loop a predetermined number of iterations; and After the first and second loops are repeated, a final image is generated based on the values ​​of the state vector and the control vector.

19. The computer-readable medium of claim 18, further comprising instructions that, when executed by the computer system, cause the computer system to: One or more control signals are generated based on the final image; and One or more actuator control commands are generated based on the control signals.

20. The computer-readable medium of claim 18, further comprising instructions for converting raw sensor data from the one or more sensors into an initial image, the initial image being from one or more sensor types: Video camera; X-ray sensor; Thermal sensor; LiDAR sensor.

Images