Systems and methods with robust deep generative models

Abstract

A machine learning system includes an encoder and a decoder network. The machine learning system is configured to obtain input data including sensor data and an allowable perturbation ℓ p a radius of a norm ball. Input boundary data is generated based on the input data. First boundary data and second boundary data are generated by propagating the input boundary data on first and second outputs of the encoder network, respectively. Third boundary data is generated associated with a latent variable based on the first boundary data and the second boundary data. Fourth boundary data is generated by propagating the third boundary data on an output of the decoder network. A lower boundary of a lower boundary of evidence is generated based on the first, second, third, and fourth boundary data to establish a robustness credential with respect to the input data. The encoder and decoder networks are updated based on the robustness credential to provide robustness to the machine learning system in defending against the allowable perturbation.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to machine learning systems, and more specifically, to deep generative models that are robust to adversarial attacks. BACKGROUND

[0002] Generally, machine learning systems, such as deep neural networks, are vulnerable to adversarial attacks. For example, a machine learning system can be attacked via its input. Such adversarial attacks include perturbations to the input that cause changes in the output of the machine learning system. For example, when the adversarial attack involves sensor data, perturbations to that sensor data can cause the machine learning system to behave in an undesirable manner, e.g., by providing incorrect output data, resulting in negative consequences and impacts. While there has been some work related to adversarial attacks in classification settings, and to a lesser extent in other supervised settings such as object detection or image segmentation, there appears to be little work in providing defense solutions to adversarial attacks to generative models. SUMMARY

[0003] The following is a summary of certain embodiments detailed below. The described aspects are presented merely to provide the reader with a brief summary of these certain embodiments and the description of these aspects is not intended to limit the scope of the disclosure. Indeed, the disclosure can encompass a variety of aspects that can not be explicitly set forth below.

[0004] According to at least one aspect, a computer-implemented method includes obtaining input data. The input data includes sensor data and an allowable perturbation p a norm ball. The method includes generating input bounding data based on the input data. The method includes generating first bounding data and second bounding data by propagating the input bounding data on first and second outputs of an encoder network. The method includes generating third bounding data associated with a latent variable and based on an output of the encoder network. The method includes generating fourth bounding data by propagating the third bounding data on an output of a decoder network. The method includes establishing a robustness credential with respect to the input data by generating a lower bound on an evidence lower bound (ELBO) based on the first bounding data, the second bounding data, the third bounding data, and the fourth bounding data. The method includes updating the encoder network and the decoder network based on the robustness credential such that a machine learning system including the encoder network and the decoder network is robust in defending against the allowable perturbation.

[0005] According to at least one aspect, a system includes an actuator, a sensor system, a non-transitory computer-readable medium, and a control system. The sensor system includes at least one sensor. The non-transitory computer-readable medium stores a machine learning system having an encoder network and a decoder network trained based on a robustness certificate forming a lower bound of a loss function of the machine learning system. The control system is operable to control the actuator based on communication with the sensor system and the machine learning system. The control system includes at least one electronic processor operable to obtain input data including sensor data from the sensor system and perturbation data from a disturbance, wherein the sensor data is perturbed by the perturbation data. The input data is processed via the machine learning system. Output data is generated via the machine learning system. The output data is a reconstruction of the sensor data. The output data is associated with a likelihood of not being perturbed by the perturbation data. The likelihood corresponds to an ELBO. The sensor data and the output data are in-distribution data corresponding to a model distribution associated with the machine learning system. Even though the perturbation data is structured to cause the machine learning system to identify and process the input data as out-of-distribution data outside of the model distribution, the machine learning system identifies and processes the input data as in-distribution data within a range of the model distribution.

[0006] According to at least one aspect, a non-transitory computer-readable medium includes at least computer-readable data operable, when executed by an electronic processor, to implement a method for training a machine learning system to be robust to perturbations. The method includes obtaining input data including sensor data and an allowable perturbation. p a norm ball. The method includes generating input boundary data based on the input data. The method includes generating first boundary data and second boundary data by propagating the input boundary data on first and second outputs of the encoder network. The method includes generating third boundary data associated with the latent variable and based on the output of the encoder network. The method includes generating fourth boundary data by propagating the third boundary data on an output of the decoder network. The method includes establishing a robustness certificate with respect to the input data by generating a lower bound of an evidence lower bound (ELBO) based on the first boundary data, the second boundary data, the third boundary data, and the fourth boundary data. The method includes updating the encoder network and the decoder network based on the robustness certificate such that the machine learning system including the encoder network and the decoder network is robust in defending against the allowable perturbation.

[0007] These and other features, aspects, and advantages of the present disclosure are discussed in the following detailed description, taken in connection with the accompanying drawings, throughout which like reference numbers represent like or similar elements. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1A is a diagram of a system including a machine learning system according to example embodiments of the present disclosure.

[0009] Figure 1B is a diagram of an example of a system related to mobile machine technology according to example embodiments of the present disclosure. Figure 1A

[0010] Figure 1C is a diagram of an example of a system related to manufacturing technology according to example embodiments of the present disclosure. Figure 1A

[0011] Figure 2A is a flowchart associated with training provably robust deep generative models according to example embodiments of the present disclosure.

[0012] Figure 2B is a flowchart associated with operating a trained provably robust deep generative model according to example embodiments of the present disclosure. Figure 2A

[0013] Figure 3A is a block diagram associated with training provably robust deep generative models according to example embodiments of the present disclosure. Figure 3B is a conceptual diagram of training provably robust deep generative models including variational autoencoders according to example embodiments of the present disclosure.

[0014] Figure 4 is a flowchart of a method of training provably robust deep generative models according to example embodiments of the present disclosure.

[0015] Figure 5A , 5B , 5C, and 5D illustrate non-limiting examples of input data according to example embodiments of the present disclosure.

[0016] Figure 6A illustrates a diagram with representations of effects of adversarial attacks on a first data set associated with different generative models according to example embodiments.

[0017] Figure 6B illustrates a diagram with representations of effects of adversarial attacks on a second data set associated with different generative models according to example embodiments. DETAILED DESCRIPTION

[0018] ​​​The embodiments described herein, and many of the attendant advantages of this application, will be understood by the foregoing description, and it will be apparent that various changes can be made in the form, construction and arrangement of the components without departing from the disclosed subject matter or without sacrificing all of its advantages. Indeed, it will be understood that the described form of these embodiments is merely an explanatory one and it is intended to encompass and guide the scope of the following claims and their equivalents. It is therefore intended that the disclosure not be limited to the particular embodiment described but that the disclosure will include all modifications and alternatives coming within the spirit and scope of the disclosure.

[0019] As described herein, embodiments relate to the application of provably robust training in the context of generative models. More specifically, these embodiments construct provable bounds related to loss functions in the context of unsupervised generative models, as opposed to supervised classification tasks. For example, in example embodiments, provably robust training involves at least one generative model, such as a variational autoencoder (VAE). In this regard, a robust lower bound is defined on a variational bound on likelihood, and then optimized during training to generate a provably robust VAE (“proVAE”). Moreover, these provably robust generative models are evaluated as significantly more robust to adversarial attacks (e.g., an adversary attempts to perturb inputs so as to substantially reduce their likelihood under the generative model) as compared to a control group of generative models.

[0020] Figure 1A A block diagram of a system 100 is illustrated in accordance with example embodiments. The system 100 includes a sensor system 110 having one or more sensors. For example, the sensor system 110 includes an image sensor, a camera, a radar sensor, a light detection and ranging (LIDAR) sensor, a thermal sensor, an ultrasonic sensor, an infrared sensor, a motion sensor, a satellite-based navigation sensor (e.g., a global positioning system (GPS) sensor), any suitable sensor, or any combination thereof. The sensor system 110 is operable to communicate with a control system 120.

[0021] The control system 120 is configured to obtain sensor data directly or indirectly from one or more sensors of the sensor system 110. Upon receiving input data (e.g., sensor data and / or image data based on sensor data), the control system 120 is configured to process the input data via the processing system 140 in conjunction with the machine learning system 200. In this regard, the processing system 140 includes at least one processor. For example, the processing system 140 includes an electronic processor, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), any suitable processing technology, or any combination thereof. Upon processing the input data, the processing system 140 is operable to generate output data via the machine learning system 200. Additionally or alternatively, the processing system 140 is operable to generate classification data that classifies the output data of the machine learning system 200. Further, the processing system 140 is operable to provide control data to the actuator system 170 based on the classification data and / or the output data of the machine learning system 200.

[0022] In example embodiments, the machine learning system 200 is stored in the memory system 160. In example embodiments, the memory system 160 is a computer or electronic storage system configured to store and provide access to various data to at least enable operations and functions as disclosed herein. In example embodiments, the memory system 160 includes a single device or multiple devices. In example embodiments, the memory system 160 can include electrical, electronic, magnetic, optical, semiconductor, electromagnetic, or any suitable technology. For example, in example embodiments, the memory system 160 can include random access memory (RAM), read-only memory (ROM), flash memory, disk drives, memory cards, optical storage devices, magnetic storage devices, memory modules, any suitable type of memory device, or any combination thereof. In example embodiments, the memory system 160 is local, remote, or a combination thereof (e.g., partially local and partially remote) with respect to the control system 120 and / or the processing system 140. For example, the memory system 160 can include at least a cloud-based storage system (e.g., a cloud-based database system) that is remote from the processing system 140 and / or other components of the control system 120.

[0023] In example embodiments, the machine learning system 200 includes at least one deep neural network. More specifically, the deep neural network includes at least one trained provably robust generative model (“PROVAE”). In response to input data, the processing system 140 (in combination with the machine learning system 200) is operable to generate output data that is a reconstruction of the input data. For example, when the input data is sensor data (and / or image data based on the sensor data), the processing system 140 is operable to generate output data via the machine learning system 200, where the output data is a reconstruction of the sensor data. As another example, when the input data includes sensor data (and / or image data based on the sensor data) that is perturbed by perturbation data, the processing system 140 is operable to generate output data via the machine learning system 200, where the output data is a reconstruction of the sensor data, whereby the likelihood effect is not disrupted by the perturbation data. This feature of the machine learning system 200 is advantageous in providing a defense solution against adversarial attacks, as such perturbation data does not cause a likelihood effect and / or drastic changes in the output data of the machine learning system 200.

[0024] Further, the system 100 includes other components that facilitate the operation of the control system 120 in relation to the sensor system 110 and the actuator system 170. For example, as shown in FIG. 1, the memory system 160 is also configured to store other relevant data related to the operation of the system 100 in relation to one or more of its components (e.g., the sensor system 110, the machine learning system 200, the actuator system 170, etc.). Moreover, the control system 120 includes an input / output (I / O) system 130 that includes one or more interfaces to one or more I / O devices related to the system 100. Also, the control system 120 is configured to provide other functional modules 150 that assist or facilitate the functioning of the system 100, such as any appropriate hardware, software, or combination thereof. For example, the other functional modules 150 include an operating system and communication technology that enable the components of the system 100 to communicate with each other as described herein. In at least Figure 1A With the configuration discussed in the example, the system 100 is suitable for use in various technologies.

[0025] Figure 1B An example of a first application of the system 100 in relation to mobile machine technology (e.g., robots, etc.) is illustrated. In this example, the system 100 is used in a warehouse environment to facilitate the movement of a mobile machine (e.g., a robot) in relation to a warehouse. Figure 1BIn particular embodiments, system 100 is employed by vehicle 10, where control system 120 controls at least one actuator system 170 of vehicle 10 in accordance with sensor data from sensor system 110. More particularly, as non-limiting examples, control system 120 is configured to identify an entity (e.g., an object) using sensor data from sensor system 110. For example, system 100 is configured to identify a sign (e.g., a stop sign) upon obtaining image data from an image sensor. As another example, system 100 is configured to detect a pedestrian from image data that is generated based on sensor data obtained from an image sensor and a LIDAR sensor. In this regard, sensor data is generated and / or obtained from one or more sensors of sensor system 110. In this example, upon receiving at least sensor data and / or image data, processing system 140, via machine learning system 200, is configured to identify an entity from the sensor data (and / or image data) upon estimating with a high likelihood value that the sensor data (and / or image data) is generated based on and / or is within a distribution associated with the entity. Advantageously, PROVAE 200A is configured to estimate with a high likelihood value even when input data (e.g., sensor data and / or image data) is perturbed (e.g., by noise), thereby being robust with respect to out-of-distribution attacks. For example, as a non-limiting example, if processing system 140, via machine learning system 200, identifies another vehicle as an entity (within its model distribution) upon receiving, from sensor system 110, input data including sensor data that is unperturbed / perturbed, control system 120 is configured to control actuator system 170 (e.g., a steering system, a braking system, or any suitable actuator system) to avoid a collision with the identified vehicle. In this example, actuator system 170 is configured to control or assist a driving function of the vehicle that is autonomous, highly autonomous, partially autonomous, conditionally autonomous, or driver-assisted.

[0026] Figure 1C An example of a second application of system 100 with respect to manufacturing technology is illustrated. In this example, system 100 is employed by a manufacturing system 10, where control system 120 controls at least one actuator system 170 of manufacturing system 10 in accordance with sensor data from sensor system 110. More particularly, as non-limiting examples, control system 120 is configured to identify an entity (e.g., an object) using sensor data from sensor system 110. For example, system 100 is configured to identify a defect (e.g., a crack) upon obtaining image data from an image sensor. As another example, system 100 is configured to detect a tool (e.g., a drill) from image data that is generated based on sensor data obtained from an image sensor and a LIDAR sensor. In this regard, sensor data is generated and / or obtained from one or more sensors of sensor system 110. In this example, upon receiving at least sensor data and / or image data, processing system 140, via machine learning system 200, is configured to identify an entity from the sensor data (and / or image data) upon estimating with a high likelihood value that the sensor data (and / or image data) is generated based on and / or is within a distribution associated with the entity. Advantageously, PROVAE 200A is configured to estimate with a high likelihood value even when input data (e.g., sensor data and / or image data) is perturbed (e.g., by noise), thereby being robust with respect to out-of-distribution attacks. For example, as a non-limiting example, if processing system 140, via machine learning system 200, identifies another vehicle as an entity (within its model distribution) upon receiving, from sensor system 110, input data including sensor data that is unperturbed / perturbed, control system 120 is configured to control actuator system 170 (e.g., a steering system, a braking system, or any suitable actuator system) to avoid a collision with the identified vehicle. In this example, actuator system 170 is configured to control or assist a driving function of the vehicle that is autonomous, highly autonomous, partially autonomous, conditionally autonomous, or driver-assisted. Figure 1CIn particular embodiments, system 100 is employed by a manufacturing system 20, where control system 120 controls at least one actuator system 170 of manufacturing machine 22 in accordance with sensor data from sensor system 110. More particularly, as a non-limiting example, control system 120 is configured to identify a fault in product 24 by modeling a distribution of images of defective components of product 24. In this case, PROVAE 200A is configured to identify a defect or fault in a component of product 24 even in cases where sensor data is perturbed by relatively small (and / or imperceptible) perturbations. For example, if perturbed input data includes an image of a defective component of product 24, processing system 140, via machine learning system 200, is configured to identify product 24 as defective and not identify product 24 as non-defective as the perturbed data would suggest. In this case, in response to processing of this perturbed input data, control system 120 is configured to classify the defective product as belonging to a distribution of defective products and is configured to control actuator system 170 based on the output of machine learning system 200 and / or classifying product 24 as within the distribution of defective products 24. In this non-limiting example, for example, actuator system 170 is configured to control an action of a robot or manufacturing machine based on product 24 being identified as defective.

[0027] In addition to or instead of first application ( Figure 1B ) and second application ( Figure 1C ), system 100 (and / or control system 120) can also operate in other applications. For example, control system 120 is configured to generate training data for training these types of machine learning systems 200. Moreover, system 100 (and / or control system 120) is also applicable in various fields, such as computer-controlled machines, robots, home appliances, power tools, electronic personal assistants, medical / medical systems, mobile machines, security systems, simulation systems, etc. Moreover, system 100 (and / or control system 120 and / or machine learning system 200) is not limited to the applications mentioned above, but can be applied to any suitable application that benefits from a machine learning system 200 having at least one trained generative model that is provably robust to perturbations.

[0028] Figure 2A A flowchart is illustrated in accordance with an example embodiment relating to a training process 204 for machine learning system 200 to generate at least one provably robust generative model (e.g., PROVAE 200A). As shown in Figure 2A In particular embodiments, system 100 is employed by a manufacturing system 20, where control system 120 controls at least one actuator system 170 of manufacturing machine 22 in accordance with sensor data from sensor system 110. More particularly, as a non-limiting example, control system 120 is configured to identify a fault in product 24 by modeling a distribution of images of defective components of product 24. In this case, PROVAE 200A is configured to identify a defect or fault in a component of product 24 even in cases where sensor data is perturbed by relatively small (and / or imperceptible) perturbations. For example, if perturbed input data includes an image of a defective component of product 24, processing system 140, via machine learning system 200, is configured to identify product 24 as defective and not identify product 24 as non-defective as the perturbed data would suggest. In this case, in response to processing of this perturbed input data, control system 120 is configured to classify the defective product as belonging to a distribution of defective products and is configured to control actuator system 170 based on the output of machine learning system 200 and / or classifying product 24 as within the distribution of defective products 24. In this non-limiting example, for example, actuator system 170 is configured to control an action of a robot or manufacturing machine based on product 24 being identified as defective. Figure 4) utilizing in-distribution data 202 to train the generative model (e.g., VAE). As an example, in-distribution data 202 refers to input data selected or drawn from a distribution of x e D associated with the generative model. This feature is advantageous because, during training process 204, a provably robust generative model can be proven without requiring other types of training data (e.g., out-of-distribution data, perturbed data, adversarial examples, etc.). After training process 204 is completed utilizing in-distribution data 202, a trained provably robust generative model (e.g., PROVAE 200A) is generated and ready for operation.

[0029] Figure 2B illustrates a flow diagram associated with operation of a machine learning system 200 including at least one trained provably robust generative model (e.g., PROVAE 200A) in accordance with example embodiments. This flow diagram highlights advantages of the provably robust generative model. For example, as an advantage, upon receiving input data including in-distribution data 202 perturbed by perturbed data 208, then PROVAE 200A is operable to identify this input data as in-distribution data and further generate output data as in-distribution data 202. Alternatively, although not shown, upon receiving input data including in-distribution data 202 not perturbed by perturbed data, then PROVAE 200A is configured to generate output data as in-distribution data 202.

[0030] Figure 3A is a block diagram associated with training a provably robust deep generative model in accordance with example embodiments. As shown in Figure 3A In example embodiments, memory system 300 is a computer or electronic storage system configured to store and provide access to various data to at least enable operations and functionalities as disclosed herein. In example embodiments, memory system 300 includes a single device or multiple devices. In example embodiments, memory system 300 includes electrical, electronic, magnetic, optical, semiconductor, electromagnetic, or any suitable technology. For example, memory system 300 includes RAM, ROM, flash memory, disk drives, memory cards, optical storage devices, magnetic storage devices, memory modules, any suitable type of memory device, or any combination thereof. In example embodiments, with respect to processing system 310, memory system 300 is local, remote, or a combination thereof (e.g., partially local and partially remote). For example, memory system 300 is configured to include at least a cloud-based storage system (e.g., a cloud-based database system) that is remote from processing system 310.

[0031] In example embodiments, as Figure 3AAs shown, the memory system 300 includes at least training data 302, machine learning data 304, and a provably robust generative model (“PROVAE”) 200A. The training data 302 includes at least sensor data (and / or image data based on the sensor data). Advantageously, the training process 204 is operable to train against permissible perturbations without requiring the perturbations as input data. Furthermore, the machine learning data 304 includes methods 400 for training and generating the PROVAE 200A. Figure 4 Related machine learning algorithms and other relevant data 304A Figure 3B ), such as various boundary data (e.g., vector data), robust credentials, etc. PROVAE 200A includes neural network data (e.g., layers, parameter data, etc.) associated with its training and / or operation. train (Data, etc.). After completing training process 204, PROVAE 200A can be... Figure 1A The system 100 or any suitable application to deploy and / or adopt.

[0032] Upon receiving training data 302 (e.g., sensor data and / or image data based on sensor data), processing system 310 is configured to combine it with machine learning data 304 to train a generative model. In this regard, processing system 310 includes at least one processor. For example, processing system 310 includes an electronic processor, CPU, GPU, microprocessor, FPGA, ASIC, any suitable processing technology, or any combination thereof. In an example embodiment, processing system 310 communicates with memory system 300 to generate a trained, provably robust generative model (“PROVAE”) 200A based on training data 302 and machine learning data 304.

[0033] Figure 3B The illustration depicts a conceptual diagram of training a generative model (e.g., a VAE) according to an example embodiment to generate a provably robust generative model (e.g., PROVAE 200A). Generally, a VAE includes at least an encoder with an encoder network and a decoder with a decoder network. In this respect, the encoder comprises a recognition model, while the decoder comprises a generative model. In the example embodiment, at least as... Figure 3B As shown, the encoder and decoder use Interval Boundary Propagation (IBP) to propagate the effects of perturbations on the input. The effects of the perturbation propagate throughout the VAE. Furthermore, for convenience, Figure 3B The diagram illustrates the mapping of some components of the VAE (e.g., μ(x), σ(x), g(z), etc.) and some representations of other related data 304A associated with the VAE, to aid in understanding... Figure 4 Together, we will demonstrate a training process that proves robustness. Furthermore, Figure 3BAn example of sensor data x (e.g., an input image 304B with a handwritten digit ‘4’) and output data is a reconstruction of this sensor data (e.g., an output image 304C with a reconstructed handwritten digit ‘4’). Both the input image 304B and the output image 304C are identified as in-distribution data (or within the distribution of “handwritten digits”).

[0034] In general, VAEs are trained based on a bound on the log-likelihood, the processing system 310 is configured to further form this bound on the log-likelihood in an adversarial setting. VAEs are trained based on the so-called evidence lower bound (ELBO) L(x) which expresses the probability p(x) in terms of the latent variable z e R k and then forms the bound on this likelihood as follows:

[0035]

[0036] where q(z\ x) is the so-called variational distribution which attempts to approximate the posterior p(z\ x) (for which the bound is tight for this case), but which does so via a more tractable class of distributions. In the VAE setting, the processing system 310 chooses:

[0037]

[0038] where m θ (x) and are encoder networks which predict the mean and variance of a normal distribution q from the input x, and g θ (z) is a decoder network which generates a sample in the input space given a latent vector z.

[0039] Under these assumptions, the ELBO has the explicit form:

[0040]

[0041] where c is a constant. In general, the encoder and decoder networks are jointly trained to maximize this lower bound using, for example, stochastic gradient descent, as represented by the following equation:

[0042]

[0043] where the processing system 310 replaces the sampling process with an equivalent process to draw samples, and ensures that the mean and variance terms can be backpropagated through via so-called reparameterization techniques.

[0044] Figure 4 ​A flowchart of a method 400 for generating a training process 204 for a provably robust generative model (e.g., PROVAE 200A) is illustrated in accordance with example embodiments. The method 400 provides a solution to the technical problem of formulating a robust defense against out-of-distribution attacks for deep generative models. The method 400 achieves such a defense by obtaining a provably robust deep generative model (e.g., PROVAE 200A) that forms a lower bound on the ELBO against admissible perturbations. More specifically, the method 400 identifies a set of admissible perturbations, and identifies l co as a ball of radius ∈ train centered at x. In the present disclosure, these terms and Δ(x) are used interchangeably for the sake of brevity of notation.

[0045] The processing system 310 is configured to obtain a lower bound on the ELBO for all possible perturbations δ ∈ Δ(x) as L(x) ≤ L(x + δ) ≤ log(p(x + δ)). This lower bound provides a certificate of robustness of the ELBO. Any possible perturbation in Δ(x) will have an effect on the ELBO that is bounded by L the lower bound. The optimization of the lower bound L is effectively training the network to be robust against the strongest possible out-of-distribution attack within Δ(x), the ι∞ ball of radius ∈ train around x.

[0046] To form the lower bound on the ELBO, the processing system 310 performs IBP throughout the layers of μ θ , σ θ , and g θ such that the processing system 310 obtains a bound on the propagation of admissible perturbations of the ELBO to the input space. The processing system 310 is thus configured to form a bound on both the kulback-Leibler (KL) divergence of the perturbed input KL(q(z|x + δ) || p(z)) and the expected value of the perturbed conditional log-likelihood . To do so, the processing system 310 performs IBP on the encoder networks μ θ and σ θ , and IBP on the decoder network g θ .

[0047] As a preliminary to the method 400, the processing system 310 propagates lower and upper bounds on the building blocks of the encoder and decoder networks. Generally, the building blocks include at least linear and convolutional layers, and monotonic element-wise activation functions. These features enable the processing system 310 to sequentially connect different interval bounds from input to output of a deep neural network (e.g., VAE). In this disclosure, for the sake of notational convenience and brevity, the upper bound on μ is denoted as and the lower bound is denoted as μ, which is considered element-wise when addressing multi-dimensional entities.

[0048] With respect to linear operators, the processing system 310 considers Wv to be a linear operator W applied to v, and are the element-wise upper and lower bounds on v. The processing system 310 decomposes the linear operator W into positive and negative operators W = W + + W - such that W + = max(W, 0) and W - = min(W, 0), where max and min correspond to the element-wise maximum and minimum. The processing system 310 then applies the upper and lower bounds on linear operators to v. More specifically, the processing system 310 applies the upper bound (Equation 7) and the lower bound (Equation 8), which hold for convolutional and linear layers due to their linear nature.

[0049]

[0050] With respect to monotonic functions, the processing system 310 is configured to denote v t = h(v t-1 ) as a monotonic (non-decreasing or non-increasing) function applied element-wise on v t-1 . The processing system 310 expresses the upper and lower bounds on v t with respect to h and the upper and lower bounds on v t-1 as follows,

[0051]

[0052] These bounds hold for monotonic activation functions, such as ReLU and sigmoid.

[0053] With respect to the l2 norm square, the processing system 310 is configured to obtain the lower and upper bounds on the l2 norm square of v by recognizing that there is an element-wise dependence on the lower and upper bounds on v. Since where v i denotes the i-th component of v, the processing system 310 obtains the respective upper and lower bounds on v as functions of and v as follows,

[0054]

[0055] After performing the preparatory measures, the processing system 310 implements the method 400 to optimize the robustness credentials obtained from the worst-case perturbation, e.g., in terms of at least one log-likelihood function. The method 400 comprises one or more iterations (or epochs). In this case, each iteration (or epoch) comprises computing robustness credentials by boundary propagation (e.g., steps 402-412) and optimizing the robustness credentials (e.g., step 414), thereby providing robustly trained encoder and decoder networks (e.g., step 416). With reference to Figure 4 , the method 400 comprises at least the following implementations, as described below.

[0056] At step 402, the processing system 310 is configured to obtain an input x and generate at least one boundary on the input x. For example, the input x comprises training data, such as X = {x1...xN}, and where n For x e X, the processing system 310 is configured to generate input boundary data on the input x i . The input boundary data comprises upper boundary data on the input x i and lower boundary data on the input x i .

[0057] Further, given a first encoding component μ θ (x) and a second encoding component σ θ (x), the processing system 310 configures the encoder network as a succession of convolutional layers with ReLU activations, where at least one last layer is at least one fully connected linear layer. Further, in the case where no perturbation is required as input x, the processing system 310 is operable to define the perturbation data as and to consider any admissible perturbed input x i + δ by representing the lower and upper boundaries of the admissible perturbed input as follows:

[0058] x i = x i - ∈ train 1 (13)

[0059]

[0060] At step 404, the processing system 310 is configured to generate boundaries for the output of the encoder network. These boundaries comprise first and second boundary data for the VAE. The first and second boundary data relate to respective outputs of the first and second encoding components of the encoder. More specifically, the processing system 310 is configured to generate first boundary data for the first encoding component μ θ ​the first upper and lower boundary data of the output of (x). Further, the processing system 310 is configured to generate the second encoding component σ θ the second upper and lower boundary data of the output of (x). The processing system 310 is configured to generate the first encoding component μ θ (x) independently of the second boundary data of (x). In this regard, the processing system 310 is configured to generate the first boundary data and the second boundary data at the same time or at different times. θ (x). In this regard, the processing system 310 is configured to generate the first boundary data and the second boundary data at the same time or at different times.

[0061] In the case of the propagation of the interval boundaries for linear layers and convolutional layers in Equations 7-8 and for activation functions in Equations 9-10, the processing system 310 is configured to generate the boundaries of the output of the encoder network based on x i and The IBP of and traversing the components of the encoder network forms the boundaries of the output of the encoder network via

[0062]

[0063] where μ i = μ θ (x i ) and σ i = σ θ (x i ) are the outputs of the encoder, μ θ and and σ θ and correspond to the propagation of the lower and upper boundaries of the encoder network, respectively. This results in the boundaries of the outputs of the encoding components μ i and σ i and log σ i as a function of the magnitude ∈ train of the perturbation Δ(x i ).

[0064] Given the boundaries on the output of the encoder network, the processing system 310 is configured to form the boundaries of the KL divergence between and via

[0065]

[0066] where and denote the i-th component of the squared mean and covariance of the i-th sample output by the encoder. Further, the processing system 310 is configured to generate the boundaries of the KL divergence between i and σi The upper boundary continues so that IBP can be performed via the decoder network.

[0067] At step 406, the processing system 310 is configured to extract at least one sample labeled “ε” and compute the boundary on the latent variable labeled “z”. For example, the processing system 310 is configured to obtain the boundary on the latent variable (or the third boundary data of the VAE) via a reparameterization technique. More specifically, in the case of the reparameterization technique, since reparameterization is a linear operator, the boundary on the latent variable follows the boundary for linear operators in Equations 7-8.

[0068] For example, processing system 310 is configured to process samples, such as those generated by... ∈ + =min(∈, 0) and ∈ - =min(∈, 0) indicates that ∈ = ∈ + +∈ - , where 0 represents the mean and I represents the identifier vector of the covariance. This reparameterization technique expresses the latent variables as z i =μ i +σ i This decouples randomness from the encoder. After using the reparameterization technique, the processing system 310 is configured to form the latent variable z. i The boundary (e.g., generating third boundary data) is expressed as:

[0069]

[0070] At step 408, the processing system 310 is configured to generate a function for the decoder network g. θ The boundary of the output of (x). At this point, the processing system 310 is configured to generate the fourth boundary data of the VAE, which includes the decoder network g. θ The fourth upper boundary data and the fourth lower boundary data of (x). For example, after step 408, the boundaries on the latent variable (i.e., z) are then passed through the decoder network g. θ And the propagation of this decoder network g θ It includes linear and convolutional layers (e.g., linear operators, where the boundaries can be propagated using equations 7-8), which have ReLU and sigmoid activations (e.g., monotonic activation functions, where the boundaries can be propagated using equations 9-10). Therefore, like the encoder network, the processing system 310 provides the latent vector z. i The function on the boundary of the decoder network outputs the boundary.

[0071] Furthermore, the processing system 310 solves the conditional log-likelihood log p(x)i |z i ) of the boundary problem. To this end, the processing system 310 will form the boundary problem of the conditional log-likelihood as the boundary problem of θ I fixed in . The processing system 310 thus simplifies the boundary problem of forming the conditional log-likelihood to the boundary problem of forming ||x i -g θ (z i )| 2 . After solving equations 11-12, the processing system 310 is configured to form the boundary of this function via:

[0072]

[0073] where the processing system 310 is configured to employ the element-wise max and min across the elements of x and the sum in j.

[0074] At step 410, the processing system 310 is configured to generate a robustness credential by generating at least one bound of the ELBO L that serves as a loss function. More specifically, the processing system 310 is configured to generate lower bound data on the ELBO. For example, the processing system 310 combines the upper and lower bounds on the encoder network and the decoder network, and the associated lower bound on the conditional log-likelihood and the upper bound on the KL divergence (since the ELBO accounts for the negative value of the KL divergence), thereby obtaining the lower bound data from the following lower bound:

[0075]

[0076] where the upper and lower bounds on the encoder network are propagated, and the reparameterization technique is:

[0077]

[0078] μ i = min{ μ θ (x i +∈ train 1) , μ θ (x i -∈ train 1)} (28)

[0079]

[0080] σ i = min{ σ θ (x i +∈ train 1) , σ θ (x i -∈ tain 1)} (30)

[0081]

[0082] The resulting lower bound on the ELBO forms a lower bound on the log-likelihood log p(x iThe lower bound of +δ) serves as a robustness credential against disturbances. This means that if the input interval boundary fed into the encoder is x i -∈ train 1 and x i +∈ train 1 (radius is ∈ train With x i Centered l ∞ In the case of L≥α (for all δ), this guarantees that for all δ: ||δ|| ∞ ≤∈ train In this case, log p(x+δ)≥α.

[0083] Method 400 includes training the VAE by optimizing the lower bound. For example, processing system 310 optimizes ELBO. L The lower bound is used to train a provably robust deep generative model, which corresponds to optimizing the robustness credential rather than directly optimizing ELBOL.

[0084] At step 412, the processing system 310 is configured to update the encoder and decoder networks to optimize the lower bound of the ELBO. For example, the processing system 310 is operable to update the parameters of the VAE and directly maximize the lower bound of the ELBO L. In this case, the parameters (i.e., θ) include at least the internal weights associated with the encoder and decoder networks of the VAE.

[0085] At step 414, the processing system 310 is configured to output a robustly trained network, which includes at least a first encoder network μ. θ (x), Second encoder network σ θ (x) and decoder network g θ (x). Once the processing system 310 outputs a robustly trained network, the processing system 310 is configured to deploy or transport the PROVAE 200A for use. For example, once robustly trained, the PROVAE 200A can be... Figure 1A The system 100 or any suitable application can be deployed and / or adopted by them.

[0086] Furthermore, method 400 is not limited to Figure 4The steps shown are not limited to IBP; various modifications may be included, provided that method 400 provides similar or substantially similar functionality, effects, and / or results. In this regard, as an example of modification, instead of IBP, method 400 is configured to include other boundary propagation techniques in steps 404, 406, and 410. These other boundary propagation techniques can provide tighter boundaries, but at the cost of greater complexity than that provided by IBP. As another example of modification, method 400 includes performing steps 408, 410, and 412 more than once (e.g., a set of times) and averaging the results of that set. Additionally or alternatively, method 400 includes implementing Monte Carlo integration to solve for... As another example of the modification, method 400 further includes classifying the output of the generative model via a classifier, such that the new robustness credentials cover a penalty for misclassified samples, which forces not only the formation of a reduced boundary in the log-likelihood, but also ensures that the classification output will not be altered by noise or adversarial attacks.

[0087] Figure 5A , 5B Figures 5C and 5D illustrate non-limiting examples of input data according to example embodiments. Examples of such input data include sample images (e.g., images of handwritten digits) obtained from a modified National Institute of Standards and Technology (“MNIST Database”). More specifically, Figure 5A The illustration shows an unperturbed sample of the input data, comprising 500 unperturbed image data points and excluding perturbed data. In contrast, Figure 5B-5D The illustration shows an adversarial example of input data, which includes the same image data 500 that has already been perturbed by the perturbed data. For example, Figure 5B The illustration shows an adversarial example 502, which includes image data 500 ( Figure 5A This is in conjunction with projective gradient descent (PGD) attacks on PROVAE. Meanwhile, Figure 5C The illustration shows an adversarial example 504, which includes ∈ with respect to PROVAE 200A. train Image data of 500 (=0.01 PGD attack) Figure 5A ). Figure 5D The illustration shows an adversarial example 506, which includes ∈ with respect to PROVAE 200A. train Image data of 500 (=0.1 PGD attack) Figure 5A ).

[0088] In example evaluations, the VAE and the PROVAE 200A were evaluated based on an un-perturbed sample of the image data 500. When this un-perturbed sample of the image data 500 was presented as input data to the VAE, then the loss was represented as L = -28.28. For comparison, for example, when the un-perturbed sample of the image data 500 was presented as input data to the PROVAE 200A with ∈ train = 0.01, then the loss was represented as L = -31.10. As another comparison, for example, when the un-perturbed sample of the image data 500 was presented as input data to the PROVAE 200A with ∈ train = 0.1, then the loss was represented as L = -41.31. As these evaluations demonstrate, there was no significant difference in performance between the VAE and the PROVAE 200A. Moreover, as the loss values demonstrate, the VAE and the PROVAE 200A were operable to correctly identify and process the image data 500 as in-distribution data and within the range of handwritten digits.

[0089] In other example evaluations, the VAE and the PROVAE 200A were evaluated based on a perturbed sample of input data in which the image data 500( Figure 5A ) was perturbed by the perturbation data. In this regard, when comparing the VAE to the PROVAE 200A with respect to these adversarial examples of attack data, then the advantages of the PROVAE 200A over the VAE were highlighted. More specifically, when the adversarial example 502 was presented as input data to the VAE, then the loss was represented as L = -113.97. That is, in this case, because L = -113.97, the VAE incorrectly interpreted this adversarial example 502 as out-of-distribution data (e.g., not within the range of handwritten digits) due to the attack of the perturbation data, even though this adversarial example 502 did include image data 500 of a handwritten digit. In contrast, when the adversarial example 504 was presented as input data to the PROVAE 200A with ∈ train = 0.01, then the loss was represented as L = -59.08, whereby the PROVAE 200A correctly identified and processed this adversarial example 504 as in-distribution data (e.g., within the range of handwritten digits), thereby exhibiting robustness as compared to the VAE. As another comparison, when the adversarial example 506 was presented as input data to the PROVAE with ∈ train = 0.1, then the loss was represented as L = -50.06, whereby the PROVAE 200A correctly identified and processed this adversarial example 504 as in-distribution data (e.g., within the range of handwritten digits), thereby exhibiting robustness as compared to the VAE. Moreover, Figure 6A and Figure 6BThe benefits of PROVAE 200A are also illustrated in FIG. 2, which includes a comparison of a VAE to PROVAE 200A.

[0090] Figure 6A and 6B A graphical illustration of a plot illustrating the effect on loss of an adversarial attack (e.g., a PGD attack for 0 < ∈ attack ≤ 0.2, with a radius of ∈ attack in the l ∞ ball, as represented by the ELBO L, for classifiers trained with different values of ∈ train . Figure 6A Graphical illustration 600 is based on a first data set (e.g., sample images from the MNIST database), while Figure 6B Graphical illustration 610 is based on a second data set (e.g., sample images from the Canadian Institute for Advanced Research (CIFAR10) database). In Figure 6A , VAE representation 602 corresponds to ∈ train = 0 based on the first data set. In Figure 6B , VAE representation 612 corresponds to ∈ train = 0 based on the second data set. Each of VAE representations 602 and 612 clearly illustrates the rapid decay in loss L for a non-robust classifier, and how training a provably robust classifier mitigates this decay at the cost of lower values of L on data that is not disturbed (where ∈ attack = 0). Moreover, for comparison, Figure 6A and 6B illustrate proVAE (∈ train = 0.01) representations 604 and 614 based on the first data set and the second data set, respectively. Also, for comparison, Figure 6A and 6B illustrate proVAE (∈ train = 0.10) representations 606 and 616 based on the first data set and the second data set, respectively. Thus, as shown in Figure 6A and 6B , each of proVAE representations 604, 606, 614, and 616 performs better for increasing ∈ attack compared to VAE representations 602 and 612.

[0091] As described herein, embodiments include a number of advantageous features and benefits. For example, these embodiments relate to training and generating provably robust generative models that are based on robust lower bounds that define variational lower bounds on likelihood (i.e., ELBOs), and optimizing these lower bounds to train provably robust generative models. These embodiments introduce provable defenses against adversarial attacks (i.e., out-of-distribution attacks) in the domain of generative models, where samples within the distribution of the model are perturbed to reduce their likelihood.

[0092] Further, Figure 6A-6B The effectiveness of this provable defense is demonstrated, and other trade-offs associated with model robustness are introduced. For example, Figure 6A-6B It is shown that PROVAE 200a provides better defenses against adversarial attacks compared to VAEs that have been robustly trained via Figure 4 the training process of PROVAE 200a. For example, it is shown that PROVAE 200a provides better defenses against adversarial attacks (e.g., by increasing the log-likelihood exhibited by the model at higher values of ∈ train as compared to VAEs that have been robustly trained via attack the training process of PROVAE 200a. Further, the teachings in this disclosure are extendable to an improved direction that involves creating defenses against into-distribution attacks, where adversarial attacks can perturb samples that are out-of-distribution to make it appear as if it came from the distribution (x ∈ D).

[0093] Moreover, embodiments are advantageous in providing technical solutions to technical problems associated with the susceptibility of machine learning systems (e.g., deep generative models) to adversarial attacks. These adversarial attacks have been known to cause imperceptible changes to input data that can result in drastic changes in the likelihood function, providing incorrect output data. In addressing this technical problem, embodiments as disclosed herein provide provably robust generative models where these small changes (e.g., perturbations) to the input of the machine learning system do not cause drastic changes in the likelihood function of the machine learning system. Thus, as discussed above, embodiments described herein are advantageous in providing defense solutions to adversarial attacks to generative models.

[0094] That is, the above description is intended to be illustrative, and not restrictive, and is provided in the context of a particular application and its requirements. Those skilled in the art will appreciate from the foregoing description that the present application can be implemented in a variety of forms, and that the various embodiments can be implemented alone or in combination. Therefore, although the embodiments of the present application have been described in connection with certain exemplary embodiments, the general principles described herein can be applied to other embodiments and applications without departing from the spirit and scope of the described embodiments. The true scope of the present application and the embodiments and / or methods of the present application are not limited to the expressly disclosed embodiments, and will be determined by the following claims, in which various modifications can be made to the embodiments and / or methods described herein without departing from the spirit and scope of the disclosure. For example, components and functions can be separated or combined, and embodiments can be implemented using different terminology to describe the described embodiments in different ways. These and other variations, modifications, additions, and improvements can fall within the scope of the disclosure as defined by the following claims.

Claims

1. A computer implementation method for training a machine learning system to be robust to perturbations, the method comprising: The input data includes image data based on sensor data and permissible perturbation l. p The radius of the norm sphere; Based on the input data, generate input boundary data; The first boundary data is generated by propagating the input boundary data onto the first output of the encoder network; The second boundary data is generated by propagating the input boundary data onto the second output of the encoder network; Generate third boundary data associated with latent variables based on the first and second boundary data; Fourth boundary data is generated by propagating third boundary data onto the output of the decoder network; Robust credentials are established about the input data by generating the lower boundary of the lower boundary of evidence (ELBO) based on the first boundary data, the second boundary data, the third boundary data, and the fourth boundary data. The encoder network and the decoder network are updated based on the robustness credentials, making the machine learning system including the encoder network and the decoder network robust against permissible perturbations; and Output data is generated via the machine learning system, wherein the output data is a reconstruction of the image data.

2. The computer implementation method according to claim 1, wherein: The ELBO includes a conditional log-likelihood function; and The steps of updating the encoder network and the decoder network include optimizing the lower boundary of the ELBO.

3. The computer implementation method according to claim 1, wherein the step of updating the encoder network and the decoder network includes: The parameters of the encoder network and the decoder network are updated based on the lower boundary of the ELBO to maximize the lower boundary of the ELBO.

4. The computer implementation method according to claim 1, wherein: The machine learning system includes a variational autoencoder, the variational autoencoder comprising an encoder network and a decoder network; and The latent variables are associated with the variational autoencoder.

5. The computer implementation method according to claim 1, wherein: The encoder network includes a first encoding component for generating a first output of the encoder network and a second encoding component for generating a second output of the encoder network; The step of generating the first boundary data includes generating first upper boundary data and first lower boundary data of the first encoding component; and The step of generating the second boundary data includes generating the second upper boundary data and the second lower boundary data of the second encoding component.

6. The computer implementation method according to claim 1, wherein the robust credential generates the lower bound of the ELBO to ensure that for conditions satisfying ||δ|| ∞ ≤∈ train For all δ, log p(x+δ)≥L(x+δ)≥ L(x) ,in x represents image data. δ represents the perturbation data. log p(x+δ) represents the log-likelihood of (x+δ). L(x+δ) represents the ELBO of the image data perturbed by the perturbed data. L(x) Indicates the lower boundary of the ELBO, and ∈ train l indicates the permissible disturbance p The radius of the norm sphere.

7. The computer implementation method according to claim 1, wherein: The input boundary data includes the upper boundary data and the lower boundary data of the input data; The first boundary data includes the first upper boundary data and the first lower boundary data of the first output of the first encoding component of the encoder network; The second boundary data includes the second upper boundary data and the second lower boundary data of the second output of the second encoding component of the encoder network. The third boundary data includes the third upper boundary data and the third lower boundary data associated with the latent variables; and The fourth boundary data includes the fourth upper boundary data and the fourth lower boundary data of the output of the decoder network.

8. A system comprising: Actuator; A sensor system, comprising at least one sensor; A non-transitory computer-readable medium storing a machine learning system having a variational autoencoder trained to be robust to perturbations, the variational autoencoder comprising an encoder network and a decoder network trained based on robustness credentials that form the lower bound of the loss function of the machine learning system. as well as A control system that controls the actuator based on communication with the sensor system and the machine learning system, the control system including at least one electronic processor operable to: Obtain input data, which includes sensor data from the sensor system and disturbance data from interference, wherein the sensor data is disturbed by the disturbance data; The input data is processed via the machine learning system. as well as Output data is generated via the machine learning system, wherein the output data is a reconstruction of the sensor data and is associated with a likelihood that has not been disturbed by the perturbation data; in: The sensor data and the output data are in-distribution data, which correspond to the model distribution associated with the machine learning system; and Even if the perturbation data is constructed to cause the machine learning system to identify and process the input data as out-of-distribution data outside the model distribution, the machine learning system will also identify and process the input data as within the range of in-distribution data; and Prior to communication with the control system, the variational autoencoder is trained through a training process, which includes: Input boundary data is generated based on training data, which includes other sensor data and permissible perturbation l. p The radius of the norm sphere; The first boundary data is generated by propagating the input boundary data onto the first output of the encoder network; The second boundary data is generated by propagating the input boundary data onto the second output of the encoder network; Generate third boundary data associated with latent variables based on the first and second boundary data; Fourth boundary data is generated by propagating third boundary data onto the output of the decoder network; The robust credentials are established based on the training data by generating a lower boundary of the Evidence Lower Boundary (ELBO) based on the first boundary data, the second boundary data, the third boundary data, and the fourth boundary data; and The encoder network and the decoder network are updated based on the robustness credentials, making the machine learning system robust against permissible perturbations.

9. The system according to claim 8, wherein: The control system is operable to generate classified data after classifying the output data; The control system is operable to generate control data for the actuator based on the classification data; as well as The actuator performs an action in response to the control data.

10. The system of claim 8, wherein the step of training the machine learning system to update the encoder network and the decoder network comprises: The parameters of the encoder network and the decoder network are updated based on the lower boundary of the ELBO to maximize the lower boundary of the ELBO.

11. The system of claim 8, wherein the machine learning system is trained such that the robust credentials generate the lower bound of the ELBO to ensure that for ||δ|| ∞ ≤∈ train For all δ, log p(x+δ)≥L(x+δ)≥ L (x) , in x represents sensor data. δ represents the perturbation data. log p(x+δ) represents the log-likelihood of (x+δ). L(x+δ) represents the ELBO based on the sensor data perturbed by the disturbed data. L(x) Indicates the lower boundary of the ELBO, and ∈ train l indicates the permissible disturbance p The radius of the norm sphere.

12. The system according to claim 8, wherein: The input boundary data includes the upper and lower boundary data of the training data; The first boundary data includes the first upper boundary data and the first lower boundary data of the output of the first encoding component of the encoder network; The second boundary data includes the second upper boundary data and the second lower boundary data output by the second encoding component of the encoder network. The third boundary data includes the third upper boundary data and the third lower boundary data associated with the latent variables; and The fourth boundary data includes the fourth upper boundary data and the fourth lower boundary data of the output of the decoder network.

13. A non-transitory computer-readable medium comprising at least computer-readable data, which, when executed by an electronic processor, is operable to implement a method for training a machine learning system to be robust to perturbations, the method comprising: The input data includes image data based on sensor data and permissible perturbation l. p The radius of the norm sphere; Based on the input data, generate input boundary data; The first boundary data is generated by propagating the input boundary data onto the first output of the encoder network; The second boundary data is generated by propagating the input boundary data onto the second output of the encoder network; Generate third boundary data associated with latent variables based on the first and second boundary data; Fourth boundary data is generated by propagating third boundary data onto the output of the decoder network; Robust credentials are established about the input data by generating the lower boundary of the lower boundary of evidence (ELBO) based on the first boundary data, the second boundary data, the third boundary data, and the fourth boundary data. The encoder network and the decoder network are updated based on the robustness credentials, making the machine learning system including the encoder network and the decoder network robust against permissible perturbations; and Output data is generated via the machine learning system, wherein the output data is a reconstruction of the image data.

14. The non-transitory computer-readable medium according to claim 13, wherein: The ELBO includes a conditional log-likelihood function; and The steps of updating the encoder network and the decoder network include optimizing the lower boundary of the ELBO.

15. The non-transitory computer-readable medium of claim 13, wherein the step of updating the encoder network and the decoder network comprises: The parameters of the encoder network and the decoder network are updated based on the lower boundary of the ELBO to maximize the lower boundary of the ELBO.

16. The non-transitory computer-readable medium according to claim 13, wherein: The machine learning system includes a variational autoencoder, the variational autoencoder comprising an encoder network and a decoder network; and The latent variables are associated with the variational autoencoder.

17. The non-transitory computer-readable medium according to claim 13, wherein: The encoder network includes a first encoding component and a second encoding component; The step of generating the first boundary data includes generating first upper boundary data and first lower boundary data of the first output of the first encoding component; and The step of generating the second boundary data includes generating the second upper boundary data and the second lower boundary data of the second output of the second encoding component.

18. The non-transitory computer-readable medium of claim 13, wherein the robust credential generates the lower bound of the ELBO to ensure that for conditions satisfying ||δ|| ∞ ≤∈ train For all δ, log p(x+δ)≥L(x+δ)≥ L(x) , in x represents image data. δ represents the perturbation data. log p(x+δ) represents the log-likelihood of (x+δ). L(x+δ) represents the ELBO of the image data perturbed by the perturbed data. L(x) Indicates the lower boundary of the ELBO, and ∈ train l indicates the permissible disturbance p The radius of the norm sphere.

Images