Apparatus and method for generating image

Abstract

A computer-implemented method for generating an image wherein the image is generated by a neural network (70), the method comprising the steps of: providing a randomly rendered image or a representation of the randomly rendered image as input to a sequence of layers of the neural network (70), the sequence of layers comprising at least one cross-focus layer; providing a first input to the cross-attention layer, where the first input is a representation of an input of the layer sequence determined by a layer of the layer sequence preceding the cross-attention layer, or the first input is an input to the layer sequence, and providing a text embedding characterizing a description of the image to be generated as a second input to the cross-attention layer; determining, by the cross-attention layer, an attention map based on the first input and the second input; optimizing the input provided to the sequence of layers based on a loss function to determine an optimized input, the method being characterized in that the loss function comprises terms characterizing the negative total variation of the map of interest; determining an output of the layer sequence based on the optimized input; determining an image on the basis of the output of the determined sequence of layers.

Description

Technical Field

[0001] The present invention relates to a method for generating an image, a method for training or testing a machine learning system, a method for determining a control signal, a training system, a control system, a computer program and a machine-readable storage medium. Prior art

[0002] Chefer et al., “Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models” on January 31, 2023, discloses how to intervene in the generative process of a stable diffusion model on the fly during inference time to improve the realism of generated images, which is available at https: / / arxiv.org / abs / 2301.13826. Background Art

[0004] The need to create images with specific content is a recurring subtask in many different applications. In particular, when training or testing a machine learning system configured for image analysis, it is often not feasible to take pictures of the required scenes. For example, when testing an image classification system for robotics, it may not be possible to lay out a wide variety of critical scenes for the test robot in critical scenarios. Similarly, when training an image classification system, the variety of content required in the training images may be difficult to achieve by recording images in the real world.

[0005] Text-to-image generative models have particular advantages in these situations. Text-to-image generative models are able to generate diverse and creative images under the guidance of target text prompts (also known as text descriptions). However, current text-to-image models may not generate images that fully convey the semantics in a given text prompt. Specifically, the inventors have identified three areas where text-to-image models are prone to errors. First, the generated images do not necessarily include all objects described in the prompt. For example, when the prompt is "cats and dogs", only cats are shown in some images, while dogs are ignored. Second, the properties of the object described by the subject in the prompt may (partially) leak to another object depicted in the generated image or be misassigned. When the input prompt is "blue cat and red suitcase", blue may also leak to the suitcase, and the generated image may depict a blue suitcase. Third, the model may not follow spatial relationships such as left / right / up / down. For example, given the prompt "the dog is on the right of the cat", the model may generate a dog on the left.

[0006] All of these issues can result in the generation of images that do not resemble the content of the textual description of the image that was supposed to be generated, leading to inaccurate evaluations of machine learning systems when used during testing, and poor performance of machine learning systems when used during training.

[0007] The inventors have discovered that advantageously, specifically optimizing the intermediate representations determined during the generation of images using a generative model helps mitigate the above-mentioned disadvantages and allows the generation of images that more faithfully match the desired content provided by the textual description. Summary of the invention

[0008] In a first aspect, the present invention relates to a computer implemented method for generating an image, wherein the image is generated by a neural network, and wherein the method comprises the following steps:

[0009] providing a randomly drawn image or a representation of a randomly drawn image as input to a sequence of layers of a neural network, wherein the sequence of layers includes at least one cross-attention layer;

[0010] providing a first input to a cross-attention layer, wherein the first input is a representation of an input to a sequence of layers determined by a layer of the sequence of layers preceding the cross-attention layer, or

[0011] where the first input is the input to the layer sequence, and

[0012] providing a text embedding representing the description of the image to be generated as a second input to the cross-attention layer;

[0013] Determining, by a cross-attention layer, an attention map based on the first input and the second input;

[0014] Optimizing the input provided to the sequence of layers based on a loss function to thereby determine the optimized input, wherein the method is characterized in that the loss function includes a term characterizing the negative total change of the attention map;

[0015] Determine the output of the layer sequence based on the optimized input;

[0016] - Determining an image based on the output of the determined sequence of layers.

[0017] The method may be understood in particular as a method for generating images for training and / or testing an image classification system and / or an image regression system.

[0018] The term "cross-attention layer" may in particular be understood as a layer implementing a cross-attention mechanism. A cross-attention layer may be understood as injecting additional information into the image generation process, which may be used to guide the content of the generated image.

[0019] The cross-attention layer is part of a sequence of layers. Preferably, the sequence of layers may be characterized in the form of separate blocks of layers of the neural network (e.g., blocks in a backward diffusion process if the neural network is a diffusion model (or just the backward part of a diffusion model)).

[0020] The cross-attention layer receives a first input which may be a randomly drawn image (if the cross-attention layer is the first layer of the neural network) or may be a representation obtained by forwarding the randomly drawn image through a layer of the neural network before the cross-attention layer. The cross-attention layer further receives a text embedding as a second input. Both the first input and the second input may be understood as sequences. The first input may in particular comprise a sequence of pixel patches of a randomly generated image (in the case where the layer is the first layer of the neural network) or embeddings for the corresponding patches. Preferably, the cross-attention layer determines a key matrix and a value matrix from the text embeddings and determines a query matrix from the second input. Based on these matrices, the cross-attention layer determines an attention map which may be understood as assigning the relevance of elements of the text embedding to the corresponding patches.

[0021] The text embedding represents the description of the image to be generated. The description preferably includes words that describe the desired content of the image. The text embedding can be determined by a language model that maps the description to the embedding, for example by using a text encoder of the CLIP model. Therefore, the resulting text embedding is preferably a sequence of embeddings for each word or token in the description, possibly filled by controlling the embedding of the token, which is common in modern language models. In other words, the attention map can be understood as a word or token linked to the text description by determining it for a specific element of the text embedding, where the text embedding is determined for the word or token. In this way, the corresponding word of the description has a one-to-one relationship with the attention map determined by the cross-attention layer. In particular, the attention map can assign a value representing the probability to each patch of a randomly drawn image, where the value indicates the importance or relevance of the word corresponding to the attention map for different patches.

[0022] The effect of optimizing the loss function is that the total variation of the attention map increases (minimizing the negative total variation can be understood as maximizing the total variation, so their optimization step can instead maximize the positive total variation). The inventors found that this leads to a reduction in the smoothness of the attention map with respect to the subject included in the description of the image to be generated. Advantageously, optimizing the input to the sequence of layers leads to an improved semantic guidance of the generated image. In other words, the generated image more faithfully matches the semantic content of the description, in particular with respect to the appearance of objects. This is due to the fact that maximizing the total variation of the attention map leads to a reduction in the smoothness of the attention map. This leads to a more diverse and realistic image synthesis, which is able to generate multiple instances of a given object. In other words, image synthesis (or image generation) is improved.

[0023] After optimizing the input to the layer sequence, the optimized input can then be used again as input to the layer sequence to determine the output of the layer sequence. The cross-concern layer may be preceded by one or more other layers in the layer sequence, i.e., the first input can be determined by forwarding the input to the layer sequence through the previous layer. Likewise, the cross-concern layer may be followed by one or more layers, where the output of the sequence can then be determined by forwarding the output of the cross-concern layer through other layers.

[0024] Determining the image based on the output of the determined layer sequence can be understood as forwarding the output of the layer sequence through one or more other layers of the neural network, wherein the last layer of these other layers then provides the generated image as output. For example, when using a stable diffusion model as a neural network, the output can be forwarded through one or more other layers, wherein the output of the one or more other layers is then provided to the decoder to determine the generated image.

[0025] In other words, determining an image based on the output of the determined sequence of layers may be understood as the output being used as input to other layers, which in turn determine the generated image as output.

[0026] A subject may be understood in particular as a noun that characterizes one or more objects that should be present (or absent) in the generated image.

[0027] The term "text embedding" can be understood as a vector representation of a text description, which is usually generated using a pre-trained language model (e.g., BERT or GPT). In particular, a text embedding can be a sequence of vector representations, including the representation of each word or token in the text description.

[0028] The term "attention map" may be understood as a map of the areas of the image to be generated that are most relevant to the text description as determined by the cross-attention layer. The attention map may in particular be understood as corresponding to words or tokens from a word embedding, wherein the attention map is represented or given by a matrix comprising probability values ​​for corresponding words or tokens for patch positions along the width and height dimensions of the matrix. Since text may be represented by a text embedding, the attention map may also be understood as storing probabilities along the width and height dimensions with respect to the text embedding.

[0029] The term "negative total change" may be understood as a measure of the change in the attention map. In particular, it may be understood as the maximum absolute difference between adjacent elements in the attention map multiplied by -1.

[0030] The attention map can also be understood as a 3D tensor storing multiple separate attention maps for each word or token.

[0031] Preferably, the text embedding comprises embeddings of a plurality of subjects included in the description, and an attention map is determined for each of the plurality of subjects by a cross-attention layer, and a negative total change is determined for each attention map corresponding to a subject from the plurality of subjects, and wherein the first term characterizes a minimum negative total change among the determined negative total changes.

[0032] This preferred embodiment describes a specific implementation of the first term used in the loss function. In particular, an attention map may be determined for each of the plurality of subjects individually, wherein the minimum negative total change of these determined attention maps is then used as the result of evaluating the first term.

[0033] The first term can be characterized in particular by the following formula:

[0034]

[0035] Among them A t is the focus map, and expresses A t [i, j, s] represents the attention map at position i, j of the embedding s of a set of subjects S included in the text description.

[0036] Preferably, the neural network includes a plurality of layer sequences, thereby including a plurality of cross-attention layers, as described above. For each of these layer sequences, the input can be optimized with respect to the loss function and the input received from the randomly drawn image (if the layer sequence is the first layer sequence in the neural network).

[0037] Preferably, the neural network is a latent diffusion model, such as a stable diffusion model or a normalized flow.

[0038] The authors found that advantageously, when using a latent diffusion model such as a stable diffusion model, image generation is even more accurate.

[0039] Optimizing the inputs provided to the sequence of layers based on the loss function is preferably achieved by using a gradient descent method.

[0040] The inventors have found that a gradient descent based approach results in the fastest convergence time and thus reduces the computational cost of performing the method.

[0041] Preferably, the text embedding includes embedding for a subject included in the description and embedding for an attribute included in the description and describing the subject, and wherein the method further comprises the following steps:

[0042] A first attention map of attributes and a second attention map of the subject are determined by a cross-attention layer;

[0043] Optimizing the input provided to the sequence of layers based on a loss function, wherein the loss function includes a second term, wherein the second term characterizes a difference between the first attention map and the second attention map.

[0044] Advantageously, this term encourages better alignment between the subject and its attributes in the generated image. In other words, if the attribute has a visual representation, that representation is better assigned to the subject defined by the attribute. For example, if the text description includes the expression "blue bench", the neural network generates an image depicting a blue bench.

[0045] Hence, the second term leads to even better image generation, i.e., the generated images even more faithfully match the desired text description.

[0046] The definition of which attribute is connected to which subject can be provided from an external source, for example, a user forming the neural network. Alternatively, part-of-speech tagging methods can also be used to automatically determine the connection between attributes and subjects in the text description.

[0047] The subject involved in the second item may also be the subject already involved in the first item. Alternatively, it may be another subject.

[0048] The difference can be determined by any matrix norm. Preferably, the difference is the Jensen-Shannon divergence. For determining the Jensen-Shannon divergence, preferably all elements of the corresponding matrix are used as empirical distributions.

[0049] Preferably, the text description includes a spatial relationship connecting the first body and the second body of the text description, and wherein the method further comprises the steps of:

[0050] Determining a third attention map corresponding to the first subject and a fourth attention map corresponding to the second subject by the cross attention layer Figure 2

[0051] determining a first index corresponding to a position in the third attention map having a maximum value within the third attention map;

[0052] determining a second index corresponding to a position in the fourth attention map having a maximum value within the fourth attention map;

[0053] Optimize the sequence of layers (s t ,s t-1 , s1), wherein the third term of the loss function characterizes the difference between the first index and the second index, wherein whether to subtract the first index from the second index or to subtract the second index from the first index is determined based on the spatial relationship.

[0054] Advantageously, the third term leads to an even greater improvement in the quality of the generated image, since the spatial positions of objects in the image are encouraged to take into account the spatial relationships provided in the textual description.

[0055] The term "spatial relationship" may be understood to refer to the relative positions of two objects in an image, as described in the text description, for example, the relative positions of the objects to be generated.

[0056] The spatial relationship may in particular be any one from the list "left", "right", "below" or "above". In the method, the first index may be determined by determining the maximum value of the focus map corresponding to the first subject and using the position of the maximum value along the height and / or width dimension of the focus map as the first index. The second index may be determined by determining the maximum value of the focus map corresponding to the second subject and using the position of the maximum value along the height and / or width dimension of the focus map as the second index. The third term may be characterized by the following formula:

[0057]

[0058] where x1 is the first index along the width dimension of the attention map.

[0059] The first, second, and third loss terms can be used in any combination in the loss function, e.g., all three terms, or just the first and second, the first and third, or just the first alone.

[0060] Some or all of the preferred embodiments of the first aspect described above may be combined to form further embodiments.

[0061] In another aspect, the present invention relates to a computer-implemented method for training or testing an image classification or image regression system. The method comprises generating an image using the method of the first aspect, and using the generated image to train or test the image classification system and / or the image regression system.

[0062] Image classification systems and / or image regression systems may be understood in particular as models from the field of machine learning that predict a class depicted on an image (image classification) and / or determine a real value based on an image (image regression). Certain models in the field of machine learning perform both classification and regression, for example object detection models.

[0063] Image classification can be understood as the process of classifying an image into one of several predefined categories or classes. Image-based object detection or semantic segmentation can be understood as specific forms of image classification. Image regression refers to the process of predicting a continuous value associated with an image, such as the age or height of a person in the image. Training involves optimizing the model parameters using a set of training images, while testing involves evaluating the performance of the model on a set of test images.

[0064] In the context of this aspect, the generated images can be used as training or test examples for an image classification or image regression system. By using the generated images, the system can learn to classify or regress images based on their content, or its ability to generalize to new images can be evaluated.

[0065] Advantageously, the authors found that better content of the generated images leads to an improved training dataset and / or an improved test dataset for an image classification or image regression system. In this way, the image classification or image regression system can achieve better performance (when the generated images are used during training), or the performance (i.e., generalization ability) of the image classification or image regression system can be more accurately determined (when the generated images are used during testing).

[0066] The generated images may in particular be annotated by, for example, a human annotator or an automatic labeling method to be used as training data in supervised training of an image classification or image regression system. Alternatively, text descriptions may have been used as annotations (e.g., "photo of a dog" may be used to extract the label "dog" as the desired class label for the generated images).

[0067] In another aspect, the invention relates to a computer-implemented method for determining a control signal for an actuator and / or a display based on the output of a neural network, wherein the neural network has been trained using a training method as described above.

[0068] Advantageously, the method is able to determine better control signals because the machine learning system achieves better performance due to the improved training data set used in training the machine learning system.

[0069] The expression "wherein the neural network has been trained using a training method as described above" can be understood as a method providing a neural network trained using a training method. Alternatively, it can also be understood as a method for determining a control signal, including a method for training an image classification system and / or an image regression system.

[0070] All or some of the described preferred embodiments of the individual methods may be combined. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Embodiments of the present invention will be discussed in more detail with reference to the following drawings. The drawings show:

[0072] Figure 1 is the method used to generate the image;

[0073] Figure 2 It is a training system for training an image classification system and / or an image regression system;

[0074] Figure 3 is a control system including a machine learning system that controls an actuator in its environment using an image classification system and / or an image regression system;

[0075] Figure 4 is a control system for controlling an at least partially autonomous robot;

[0076] Figure 5 It is the control system that controls the manufacturing machines;

[0077] Figure 6 It is the control system that controls the imaging system. DETAILED DESCRIPTION

[0078] Figure 1 The image (x) is generated by the neural network (70). i The neural network (70) is preferably a generative part of a stable diffusion model. The neural network (70) is provided with a randomly sampled image (z t ) or randomly sampled vectors as input. In addition, for example, text embeddings (τ θ ), which is common for stable diffusion models. Text embedding (τ θ ) is preferably in the form of a sequence of vectors, each vector representing an image to be generated (x i ) is a token or word describing the text. Therefore, the text embedding (τ θ ) sequence and the words or tokens in the text description.

[0079] The neural network (70) comprises at least one cross-attention layer (CA t , C.A. t-1 , CA1), preferably multiple cross-attention layers (CA t ,CA t-1 , CA1). In the cross attention layer (CA t ,CA t-1 , CA1), there are preferably other layers that perform calculations. In addition, the cross-attention layer (CA t , C.A. t-1 , CA1) are preferably located in the layer sequence (s t ,s t-1, s1), which can be understood as a sub-network of the neural network (70). For example, when the neural network (70) is a stable diffusion model, the cross attention layer (CA t ,CA t-1 , CA1) are placed inside the transformer block. Each block can be understood as a sequence of layers (s t ,s t-1 , s1), where the entire neural network (70) also represents a sequence of layers. In the following, the first cross-attention layer (CA t ) describes the functionality of the approach and then generalizes it to include multiple cross-attention layers (CA t-1 , CA1) is an embodiment of a neural network (70).

[0080] Including the first cross-attention layer (CA t ) layer sequence (s t ) can be obtained from the cross-attention layer (CA t ). Alternatively, the cross-attention layer (CA t ) can also be located in the layer sequence (s t ) between layers or layer sequences (s t ) at the end of the layer. Regardless of where it is placed, the cross-attention layer (CA t ) is based on the Cross Attention Layer (CA t ) before one or more layers to process randomly drawn images (z t ) to receive a randomly drawn image (z t ) or randomly draw an image (z t ) is an intermediate representation (not shown). To this end, randomly draw an image (z t ) or the intermediate representation preferably takes the form of a vector sequence.

[0081] In addition, the cross attention layer (CA t ) is the text embedding (τ θ ), which can be understood as the image (x i )’s textual description.

[0082] Cross Attention Layer (CA t ) input (z t ) is optimized to determine the optimal input (o t ). Optimize input (o t ) is determined by the optimization process (Opt t ) is determined. Optimization process (Opt t ) Based on the optimization process (Opt t ) optimizes the loss function to determine the optimization input (o t ). In other words, optimize the input (o t) is an optimization result that at least characterizes a local minimum of the loss function. The loss function includes a first term that characterizes a negative total change of the attention map of the cross-attention. Preferably, the text embedding includes an embedding for a plurality of subjects included in the description, wherein the negative total change is determined for each attention map corresponding to a subject from the plurality of subjects, and wherein the first term characterizes a minimum negative total change among the determined negative total changes. In other words, based on the cross-attention layer (CA t ) input, determine the attention map, where the optimized input (o t ) is selected so that the loss function is minimized (at least locally). This is preferably achieved by an iteratively determined gradient descent method, where in each iteration, the optimization input (o t ), which is then used as input to determine the attention map. The interactive nature of this preferred variant is depicted by the dashed arrows in the figure.

[0083] The first term characterizing the negative total change may in particular be understood as the first term of the total change of the attention map multiplied by -1. In particular, the attention map may include a probability for each position in the attention map for a plurality or all vectors included by the optimized text embedding. The text description may be about the image to be generated (x i ) constitutes the body (i.e., for example, what words or tokens in the description of image (x i ) are annotated with visual concepts such as objects depicted in the text. Annotations can be provided by users of the neural network (70) or by automatic processes such as part-of-speech tagging methods. θ ), each attention map can also be assigned to a specific word or token in the text description, since the attention map can be specifically targeted at the text embedding (τ θ ) is determined by each vector included in the text description. The first term can then characterize the minimum negative total change in some or all attention maps corresponding to the body of the text description.

[0084] The first term can be characterized in particular by the following formula:

[0085]

[0086] Among them A t is the focus map, and expresses A t [i, j, s] represents the image (x i ) describes an attention map at positions i, j of the embeddings s of multiple subjects S included.

[0087] In the optimization process (Opt t ) to determine the optimal input (o t ) after the cross attention layer (CA t) can then be based on the optimized input (o t ) and text embedding (τ θ ) determines its output. The output can be provided to other blocks (e.g., a sequence of layers) of the neural network (70). The output can also be provided to an optional decoder (71) that decodes the determined output into the generated image (x i ), which is common for stable diffusion models. For other models (e.g., normalized flow), the output may already be in the generated image (x i ) space, and thus the decoder (71) may not be required, i.e., the output may be the generated image ((x i ), for example, for the normalized flow model).

[0088] Preferably, the cross-attention layer (CA t ) is provided as input to one or more further layers of the neural network (70). The one or more further layers may in particular represent a transformer block. The further layers may in particular also include further cross-attention layers (CA t-1 , CA1), where for the other cross-attention layer (CA t-1 , CA1), optimize the input (o t-1 , o1) can be determined as for the first cross-attention layer (CA t In other words, if the neural network (70) includes multiple cross-attention layers (CA t , C.A. t-1 , CA1), then optimize the input (o t , o t-1 , o1) can be individually targeted for each cross-attention layer (CA t , C.A. t-1 , CA1) is determined. Optimize input (o t , o t-1 , o1) can in particular be determined sequentially, i.e. once the corresponding cross-attention layer (CA t , C.A. t-1 , CA1) is available, the corresponding optimization process (Opt t , Opt t-1 , Opt1). Each optimization process (Opt t , Opt t-1 , Opt1) can perform separate optimizations using the loss functions described above.

[0089] The generated image (x i ) can be understood in particular to be suitable for training or testing another machine learning system. Thus, the image (x i) is provided to the database (St2) for training or testing. Preferably, the image (x i ) may be annotated by labels characterizing the object or other content of the image, wherein the labels may then be provided in the database (St2) as corresponding to the image (x i ) tag.

[0090] The label can represent the image (x i ) main categories, images (x i ) multiple different attribute classification, characterization image (x i ) in the bounding box of the object, image (x i ) in the object category, image (x i ) semantic segmentation.

[0091] Preferably, the text description may include a subject and an attribute, wherein the attribute describes the subject. For example, the attribute may be an adjective describing the subject (e.g., green dog). The loss function may then include a second term, wherein the second term characterizes the difference between an attention map corresponding to the cross-attention of the subject and an attention map corresponding to the cross-attention of the attribute. The second term may in particular characterize the Jensen-Shannon divergence between an attention map determined for the attribute and an attention map determined for the subject.

[0092] In particular, the second term can be characterized by:

[0093]

[0094] Where A[:,:,r] represents the matrix of attention values ​​of attributes included in the attention map, and A[:,:,s] represents the matrix of attention values ​​of subjects included in the attention map.

[0095] The connection of which attribute is connected to which subject can be provided by the user or through part-of-speech tagging.

[0096] Preferably, the text embedding includes a spatial relationship connecting the first subject and the second subject, and the loss function also includes a third term representing the difference between a first index of an attention map corresponding to the first subject and a second index of an attention map corresponding to the second subject, wherein the first index represents the maximum value of the attention map corresponding to the first subject, and the second index represents the maximum value of the attention map corresponding to the second subject, and wherein whether to subtract the first index from the second index or to subtract the second index from the first index is determined based on the spatial relationship.

[0097] The spatial relationship may in particular be any one from the list "left", "right", "below" or "above". In the method, the first index may be determined by determining the maximum value of the focus map corresponding to the first subject and using the position of the maximum value along the height and / or width dimension of the focus map as the first index. The second index may be determined by determining the maximum value of the focus map corresponding to the second subject and using the position of the maximum value along the height and / or width dimension of the focus map as the second index. The third term may be characterized by the following formula:

[0098]

[0099] where x1 is the first index along the width dimension of the attention map.

[0100] Figure 2 The generated image (x i ) for training a machine learning system (60). The training dataset (T) preferably includes a plurality of images (x i ), for example, where each image is based on a different randomly drawn image (z t ) and / or different text descriptions. i ) is used to train the machine learning system (60), wherein the training dataset (T) preferably also has for each image (x i ) includes the expected output signal (t i ), which corresponds to the image (x i ) and characterizes the image (x i ) is a real value. Therefore, the machine learning system (60) can be understood as an image classification system or an image regressor system. The machine learning system (60) can be given in the form of a neural network, such as a convolutional neural network, a field of view transformer, or a visual transformer.

[0101] For training, the training data unit (150) accesses a computer-implemented database (St2), which provides a training data set (T). The training data unit (150) preferably randomly determines at least one image (x i ) and corresponds to the image (x i ) of the expected output signal (t i ), and the image (x i ) is sent to a machine learning system (60). The machine learning system (60) processes the image (x) based on the image (x) i ) to determine the output signal (y i ).

[0102] Expected output signal (ti ) and the determined output signal (y i ) is sent to the modification unit (180).

[0103] Based on the expected output signal (t i ) and the determined output signal (y i ), the modification unit (180) then determines new parameters (Φ′) for the machine learning system (60). To this end, the modification unit (180) uses a loss function to compare the expected output signal (t i ) and the determined output signal (y i The loss function determines the output signal (y i ) and the expected output signal (t i ) deviates from the first loss value. In a given embodiment, a negative log-likelihood function is used as a loss function. In alternative embodiments, other loss functions are also conceivable.

[0104] Furthermore, it is conceivable that the output signal (y i ) and the expected output signal (t i ) each includes a plurality of sub-signals, for example in the form of a tensor, wherein the desired output signal (t i ) corresponds to the determined output signal (y i For example, it can be envisioned that the machine learning system (60) is configured for object detection, and the first sub-signal characterizes the object with respect to the image (x i ), and the second sub-signal characterizes the exact position of the object. If the output signal (y i ) and the expected output signal (t i ) includes multiple corresponding sub-signals, then a second loss value is preferably determined for each corresponding sub-signal by an appropriate loss function, and the determined second loss values ​​are appropriately combined to form the first loss value, for example by a weighted sum.

[0105] The modification unit (180) determines new parameters (Φ') based on the first loss value. In a given embodiment, this is done using a gradient descent method, preferably stochastic gradient descent, Adam or AdamW. In other embodiments, the training can also be based on an evolutionary algorithm or a second-order method for training neural networks.

[0106] In other preferred embodiments, the described training is iteratively repeated for a predefined number of iteration steps, or iteratively repeated until the first loss value is below a predefined threshold. Alternatively or additionally, it is also conceivable that the training is terminated when the average first loss value with respect to the test or validation data set is below a predefined threshold. In at least one iteration, the new parameters (Φ′) determined in the previous iteration are used as parameters (Φ) of the machine learning system (60).

[0107] Furthermore, the training system (140) may include at least one processor (145) and at least one machine-readable storage medium (146) containing instructions which, when executed by the processor (145), cause the training system (140) to perform a training method according to one of the aspects of the present invention.

[0108] Figure 3 An embodiment of a control system (40) is shown that controls an actuator (10) in its environment (20) based on the output of a machine learning system (60). The actuator (10) and its environment (20) will be collectively referred to as the actuator system. At preferably evenly spaced time points, a sensor (30) senses a condition of the actuator system. The sensor (30) may include several sensors. Preferably, the sensor (30) is an optical sensor that captures an image of the environment (20). An output signal (S) of the sensor (30) (or, in the case where the sensor (30) includes multiple sensors, an output signal (S) of each sensor) encoding the sensed condition is sent to the control system (40).

[0109] Thus, the control system (40) receives a stream of sensor signals (S). It then calculates a series of control signals (A) from the stream of sensor signals (S), which are then sent to the actuators (10).

[0110] The control system (40) receives a stream of sensor signals (S) from the sensor (30) in an optional receiving unit (50). The receiving unit (50) transforms the sensor signals (S) into images (x). Alternatively, in the absence of a receiving unit (50), each sensor signal (S) may be directly provided as an image (x). For example, the image (x) may be given as an excerpt from the sensor signal (S). Alternatively, the sensor signal (S) may be processed to generate the image (x), for example, by preprocessing the sensor signal (S) to determine the image (x). In other words, the image (x) is provided based on the sensor signal (S).

[0111] The image (x) is then passed to a machine learning system (60).

[0112] The machine learning system (60) is parameterized by parameters (Φ), which are stored in and provided by a parameter memory (St1).

[0113] The machine learning system (60) determines an output signal (y) based on the image (x). The output signal (y) includes information that assigns one or more labels to the image (x). The output signal (y) is sent to an optional conversion unit (80), which converts the output signal (y) into a control signal (A). The control signal (A) is then sent to the actuator (10) to control the actuator (10) accordingly. Alternatively, the output signal (y) can be directly used as the control signal (A).

[0114] The actuator (10) receives the control signal (A), is controlled accordingly, and performs an action corresponding to the control signal (A). The actuator (10) may include control logic that transforms the control signal (A) into another control signal, which is then used to control the actuator (10).

[0115] In further embodiments, the control system (40) may include the sensor (30). In still further embodiments, the control system (40) may alternatively or additionally include the actuator (10).

[0116] In yet further embodiments, it is contemplated that the control system (40) controls the display (10a) instead of or in addition to the actuator (10).

[0117] Furthermore, the control system (40) may include at least one processor (45) and at least one machine-readable storage medium (46) storing instructions thereon which, if executed, cause the control system (40) to perform a method according to an aspect of the present invention.

[0118] Figure 4 An embodiment is shown in which a control system (40) is used to control an at least partially autonomous robot, such as an at least partially autonomous vehicle (100).

[0119] The sensors (30) may include one or more video sensors and / or one or more radar sensors and / or one or more ultrasonic sensors and / or one or more LiDAR sensors. Some or all of these sensors are preferably, but not necessarily, integrated into the vehicle (100).

[0120] The machine learning system (60) may be configured to detect an object in the vicinity of the at least partially autonomous robot based on an input image (x). The output signal (y) may include information characterizing where the object is located in the vicinity of the at least partially autonomous robot. A control signal (A) may then be determined based on the information, for example to avoid a collision with the detected object.

[0121] The actuator (10), preferably integrated in the vehicle (100), may be provided by a brake, propulsion system, engine, drive train, or steering device of the vehicle (100). The control signal (A) may be determined so that the actuator (10) is controlled so that the vehicle (100) avoids colliding with the detected object. The detected objects may also be classified according to what the machine learning system (60) thinks they are most likely to be (e.g., pedestrians or trees), and the control signal (A) may be determined based on the classification.

[0122] Alternatively or additionally, the control signal (A) may also be used to control a display (10a), for example for displaying objects detected by the machine learning system (60). It is also conceivable that the control signal (A) may control the display (10a) so that it generates a warning signal if the vehicle (100) is close to colliding with at least one of the detected objects. The warning signal may be a warning sound and / or a tactile signal, such as vibration of the vehicle's steering wheel.

[0123] In further embodiments, the at least partially autonomous robot may be provided by another mobile robot (not shown), which may move, for example, by flying, swimming, diving or walking. The mobile robot may in particular be an at least partially autonomous lawn mower or an at least partially autonomous cleaning robot. In all of the above embodiments, the control signal (A) may be determined such that a propulsion unit and / or a steering device and / or a brake of the mobile robot are controlled such that the mobile robot can avoid a collision with the identified object.

[0124] In a further embodiment, the at least partially autonomous robot may be provided by a gardening robot (not shown) which uses a sensor (30), preferably an optical sensor, to determine the state of plants in the environment (20). The actuator (10) may control a nozzle for spraying a liquid and / or a cutting device such as a blade. Depending on the identified plant species and / or the identified state of the plant, a control signal (A) may be determined to cause the actuator (10) to spray an appropriate amount of an appropriate liquid on the plant and / or to cut the plant.

[0125] In yet another embodiment, at least part of the autonomous robot may be provided by a household appliance (not shown), such as a washing machine, stove, oven, microwave or dishwasher. A sensor (30), such as an optical sensor, may detect the state of an object to be processed by the household appliance. For example, where the household appliance is a washing machine, the sensor (30) may detect the state of the laundry in the washing machine. The control signal (A) may then be determined based on the detected laundry material.

[0126] Figure 5An embodiment is shown in which a control system (40) is used to control a manufacturing machine (11) (e.g., a punch tool, a tool, a gun drill, or a fixture) of a manufacturing system (200) (e.g., as part of a production line). The manufacturing machine may include a transport device, such as a conveyor belt or an assembly line, that moves the manufactured product (12). The control system (40) controls the actuator (10), which in turn controls the manufacturing machine (11).

[0127] The sensor (30) may be provided by an optical sensor that captures, for example, characteristics of the manufactured product (12).

[0128] The machine learning system (60) may determine the position of the manufactured product (12) relative to the transport device. The actuator (10) may then be controlled based on the determined position of the manufactured product (12) for subsequent manufacturing steps of the manufactured product (12). For example, the actuator (10) may be controlled to cut the manufactured product at a specific location on the manufactured product itself. Alternatively, it is contemplated that the machine learning system (60) classifies whether the manufactured product is broken and / or presents a defect. The actuator (10) may then be controlled to remove the manufactured product from the transport device.

[0129] Figure 6 An embodiment of a medical imaging system (500) controlled by a control system (40) is shown. The imaging system may be, for example, an MRI device, an x-ray imaging device, or an ultrasound imaging device. The sensor (30) may be, for example, an imaging sensor that takes at least one image of a patient, for example showing different types of body tissue of the patient.

[0130] The machine learning system (60) may then determine a classification of at least a portion of the sensed image. Thus, at least a portion of the image is used as an input image (x) to the machine learning system (60).

[0131] A control signal (A) may then be selected based on the classification to control the display (10a). For example, the machine learning system (60) may be configured to detect different types of tissue in the sensed image, for example by classifying the tissue displayed in the image as malignant or benign tissue. This may be accomplished by semantic segmentation of the input image (x) by the machine learning system (60). The control signal (A) may then be determined to cause the display (10a) to display different tissues, for example, by displaying the input image (x) and coloring different areas of the same tissue type with the same color.

[0132] In other embodiments (not shown), the imaging system (500) can be used for non-medical purposes, such as to determine material properties of a workpiece. In these embodiments, the machine learning system (60) can be configured to receive an input image (x) of at least a portion of the workpiece and perform semantic segmentation of the input image (x) to classify the material properties of the workpiece. A control signal (A) can then be determined to cause the display (10a) to display the input image (x) and information about the detected material properties.

[0133] The term "computer" may be understood to cover any device for the processing of predefined computing rules. These computing rules may take the form of software, hardware, or a mixture of software and hardware.

[0134] In general, a plurality may be understood to be indexed, that is, each element in the plurality is assigned a unique index, preferably by assigning consecutive integers to the elements included in the plurality. Preferably, if the plurality comprises N elements, where N is the number of elements in the plurality, the elements are assigned integers from 1 to N. It may also be understood that the elements of the plurality may be accessed by their indexes.

Claims

1. A method for generating an image (x i ), wherein the image (x i ) is generated by a neural network (70), and wherein the method comprises the following steps: Provide a random drawn image (z t ) or randomly draw an image (z t ) t-1 , z1) as the layer sequence (s) of the neural network (70) t ,s t-1 , s1), where the layer sequence includes at least one cross-attention layer (CA t , C.A. t-1 , CA1); Cross-Attention Layer (CA t , C.A. t-1 , CA1) providing a first input, wherein the first input is a representation of an input of a sequence of layers determined by a layer of the sequence of layers before the cross attention layer, or where the first input is the input to the layer sequence, and Provide a representation of the image to be generated (x i ) is a text embedding of the description (τ θ ) as the cross attention layer (CA t , C.A. t-1 , the second input of CA1); Determining, by a cross-attention layer, an attention map based on the first input and the second input; Optimize the sequence of layers (s t ,s t-1 , s1) input, thereby determining the optimal input (o t , o t-1 , o1), wherein the method is characterized in that the loss function includes a term characterizing the negative total change of the attention map; Based on optimized input (o t , o t-1 , o1) to determine the layer sequence (s t ,s t-1 , output of s1); Based on the determined layer sequence (s t ,s t-1 , s1) output to determine the image (x i ).

2. The method according to claim 1, wherein the text embedding (τ θ ) includes embeddings for multiple subjects included in the description, and By the cross attention layer (CA t , C.A. t-1 , CA1) determines an attention map (CA1) for each of the multiple subjects t , C.A. t-1 , CA1), and determining a negative total change for each of the interest maps corresponding to a subject from the plurality of subjects, and The first term represents the smallest negative total change among the determined negative total changes.

3. The method according to claim 1 or 2, wherein the first term is characterized by the following formula: Among them A t is the focus map, and expresses A t [i, j, s] represents the attention map at position i, j of the embedding s describing the set of subjects S included.

4. The method according to any one of claims 1 to 3, wherein the text embedding (τ θ ) comprises embedding for a subject included in the description, and embedding for an attribute included in the description and describing the subject, and wherein the method further comprises the steps of: The first attention map for the attribute and the second attention map for the subject are determined by the cross attention layer. Optimize the sequence of layers (s t ,s t-1 , s1), where the loss function includes a second term, where the second term characterizes the difference between the first attention map and the second attention map. The method of claim 4 , wherein the difference is the Jensen-Shannon divergence.

6. A method according to any one of the preceding claims, wherein the text description comprises a spatial relationship connecting a first body and a second body of the text description, and wherein the method further comprises the steps of: Determining, by the cross-attention layer, a third attention map corresponding to the first subject and a fourth attention map corresponding to the second subject; determining a first index corresponding to a position in the third attention map having a maximum value within the third attention map; determining a second index corresponding to a position in the fourth attention map having a maximum value within the fourth attention map; Optimize the sequence of layers (s t ,s t-1 , s1), wherein the third term of the loss function characterizes the difference between the first index and the second index, wherein whether to subtract the first index from the second index or to subtract the second index from the first index is determined based on the spatial relationship.

7. A method according to any one of the preceding claims, wherein the optimization input (o) is determined by minimizing a loss function using a gradient descent method. t , o t-1 , o1).

8. A method according to any one of the preceding claims, wherein: The neural network (70) is a potential diffusion model, preferably a stable diffusion model, or a normalized flow.

9. A computer-implemented method for training or testing an image classification system and / or an image regression system (60), comprising the following steps: The method according to any one of the preceding claims generates an image (x i ); Use the generated image (x i ) to train or test an image classification system and / or an image regression system (60).

10. A computer-implemented method for determining a control signal (A) for an actuator (10) and / or a display (10a) based on an output (y) of a machine learning system (60), wherein the machine learning system (60) has been trained using a method according to claim 9.

11. A training system (140) configured to perform the training method according to claim 9.

12. A control system (40) configured to perform the method according to claim 10.

13. A computer program configured to cause a computer to execute the method according to any one of claims 1 to 10 and all the steps thereof when the computer program is executed by a processor (45, 145).

14. A machine-readable storage medium (46, 146) having stored thereon a computer program according to claim 13.

Images