Apparatus and computer-implemented method for machine learning
By using the method of embedding and model selection token selection in machine learning, the problem of insufficient computing resources is solved, and efficient digital image representation and reconstruction on limited resource devices is realized, improving the performance of machine learning.
Patent Information
- Application Number
- CN202410767494.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-06-14
- Filing Date
- 2024-06-14
- Publication Date
- 2024-12-17
AI Technical Summary
In machine learning, the prior art is difficult to achieve good performance under affordable computing resource requirements, especially in memory and computing time, making it difficult to perform efficiently on devices with limited computing resources.
By providing embeddings associated with objects, using the model to select the token to represent the digital image, and determining the embedding and model parameters based on the difference between the token and its reconstruction, the quantization and reconstruction of the digital image is realized, using an encoder- Decoder structure trains embeddings to reduce computing resource requirements.
While reducing computing resources, this method realizes effective representation and reconstruction of digital images, and can be performed on computers with fewer computing resources, improving the performance and efficiency of machine learning.
Smart Images
Figure CN119151005A_ABST
Abstract
Description
Background Art
[0001] The present invention relates to apparatus and computer-implemented methods for machine learning. In machine learning, it is desirable to achieve good performance with affordable computational requirements. Summary of the invention
[0002] A computer-implemented method for machine learning, comprising: providing an embedding associated with an object, providing a token representing a portion of a digital image depicting at least a portion of the object, wherein the token represents the portion at a resolution lower than the resolution of pixels of the portion, selecting an embedding from the embeddings to represent the token in a representation of the digital image using a model, determining a reconstruction of the token representing the object using the model based on the representation of the digital image, and determining parameters defining at least one of the embeddings and / or parameters defining the model based on a difference between the token and the reconstruction of the token. The token represents a quantization of the digital image. The embedding represents a codebook. The representation of the digital image is a codebook-specific quantization of the digital image. The method trains the embedding, i.e., the codebook and / or the model based on the quantized digital image. This requires less computing resources, such as memory, computing time, computing power, and makes the method executable on a computer with fewer computing resources.
[0003] The method preferably comprises providing an embedding set for the object and selecting an embedding from the embedding set with the model. The set represents an object-specific codebook storing embeddings associated with the object.
[0004] The method preferably comprises providing an embedding set associated with the object by a corresponding label, determining the label of the object from pixels of the portion of the digital image using a model, and selecting the embedding set for the object associated with the label. This means that the codebook is selected according to the label.
[0005] The method preferably comprises determining tokens from pixels of said portion of the digital image, in particular using an encoder.The training may be based on an encoder-decoder structure, wherein the encoder determines features, ie tokens, which the decoder uses to determine reconstruction of the digital image.
[0006] The method preferably comprises determining a reconstruction of a digital image depicting a reconstruction of the object or a digital image depicting a reconstruction of the object based on the reconstruction of the tokens representing the object, in particular using a decoder.The decoder may determine the reconstruction of the digital image or a new digital image comprising the object.
[0007] The method preferably comprises providing a position of a reconstructed object in the reconstruction of a digital image, and determining the reconstructed object at the position in the reconstruction of the digital image, or providing a position of a reconstructed object in a digital image including the reconstructed object, and determining the reconstructed object at the position in the digital image including the reconstructed object. Providing the position enables the object to be reconstructed at an arbitrary position in the reconstructed digital image or in a new digital image.
[0008] The method preferably includes providing an embedding for a reconstructed object of a portion of a digital image, determining a token representing the reconstructed object of the digital image based on the embedding, and determining the reconstruction of the digital image based on the token representing the reconstructed object of the digital image, or providing an embedding for an object of the digital image, determining a token representing the object of the digital image based on the embedding, and determining the digital image based on the token representing the object of the digital image. Providing the object with the embedding creates a new object in the reconstruction of the digital image or in the new digital image.
[0009] The method preferably comprises determining a mapping associating an embedding representing a token associated with an object with a position of a portion of an image or a position of an object in a digital image. The mapping is position information enabling position specific reconstruction.
[0010] The method preferably comprises determining the position of the reconstructed object in the reconstruction of the digital image or in the reconstructed digital image depicting the object based on the mapping. This means that the reconstruction is position-specific.
[0011] The method preferably includes determining a plurality of embeddings to represent tokens representing a plurality of portions of the digital image, wherein the mapping associates the embeddings with corresponding locations in the digital image.
[0012] A device for machine learning, comprising at least one processor and at least one memory, wherein the memory is configured to store instructions, which are executable by the at least one processor and when executed by the at least one processor, cause the device to perform the method, and the device has advantages corresponding to the advantages of the method.
[0013] A program including instructions which, when executed by a computer, causes the computer to execute the method, has advantages corresponding to the advantages of the method. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] Further advantageous embodiments can be derived from the following description and the accompanying drawings. In the drawings: Figure 1 Schematically depicting a device used for machine learning, Figure 2 Schematically depicting a machine learning system, Figure 3 A first example of an encoder is schematically described, Figure 4 A first example of a decoder is schematically described, Figure 5 A second example of an encoder is schematically described, Figure 6 A second example of a decoder is schematically described, Figure 7 The inference is schematically described, Figure 8 The method for machine learning is schematically depicted. DETAILED DESCRIPTION
[0015] Figure 1 A device 100 for machine learning is schematically depicted.
[0016] The device 100 includes at least one processor 102 and at least one memory 104 .
[0017] The at least one memory 104 is configured to store instructions executable by the at least one processor 102 .
[0018] When executed by at least one processor 102, the instructions cause the device 100 to perform a method for machine learning.
[0019] The program may include instructions. The device 100 may be a computer.
[0020] The device 100 may include an input 106 for a digital image 108. The device 100 in this example may be connected to a camera 110 via the input 106. The device 100 may include a camera 110 for capturing the digital image 108.
[0021] The digital image 108 may be a video, radar, LiDAR, ultrasonic, motion, or thermal image.
[0022] The device 100 may comprise an output 112 of a reconstructed digital image 114. The device 100 in this example may be connected via the output 112 to a display 116 for displaying the reconstructed digital image 114. The device 100 may comprise a display 116.
[0023] The device 100 may include a controller 118 for a technical system 120. The technical system 120 may be a physical system in a real-world environment. The technical system 120 may be a robot, in particular a vehicle, a manufacturing machine, a power tool, an access control system, a medical imaging device, or a personal assistance system. The camera 110 may be configured to capture a digital image 108 depicting a portion of the technical system 120 or an environment of the technical system 120. The technical system 120 may be controllable by the controller 118. The controller 118 may be configured to control the technical system 120 based on a reconstruction 114 of the digital image 108 or a new digital image determined using the device 100.
[0024] Figure 2 A machine learning system 200 is depicted.
[0025] The machine learning system 200 includes an encoder 202 and a decoder 204 .
[0026] The encoder 202 is configured to map the digital image 108 to tokens 206 in a latent space Z representing a portion of the digital image 108. The encoder 202 may be configured to map at least a portion of the digital image 108 to a token associated with the at least a portion.
[0027] The machine learning system 200 includes a model 208. The model 208 is configured to map the tokens 206 to representations 210 of the digital image 108 in a latent space Z. The model 208 is configured to map the representations 210 of the digital image 108 to reconstructions of tokens 212. The model 208 may be configured to map representations of new digital images sampled in the latent space Z to new tokens for the new digital images.
[0028] The decoder 204 is configured to map the reconstruction of the tokens 212 from the latent space Z to the reconstructed digital image 114, or to map new tokens from the latent space Z to a new digital image. The decoder 204 may be configured to map the reconstruction of the tokens associated with at least one portion to a portion of the digital image 108.
[0029] The encoder 202 and the decoder 204 may be part of a first stage. The first stage may include a codebook dictionary of predetermined embeddings in the latent space Z. The codebook dictionary may include a codebook associated with an object or a label of an object. The codebook dictionary may include a codebook for at least one object or at least one label. The codebook dictionary may include several codebooks for several objects or several labels. The model 208 may be configured to select an embedding for at least one of the tokens 206 from the codebook and map the embedding to a representation 210 of the digital image 108.
[0030] Tokens 206 and embeddings may be vectors or tensors in a latent space Z.
[0031] The machine learning system 200 is configured, for example, to learn a codebook dictionary by tokenizing a digital image with an encoder 202, i.e., encoding the digital image into a finite set of tokens with the encoder 202, determining a reconstruction of the digital image from the finite set of tokens with a decoder 204, and determining a codebook dictionary that minimizes the difference between the digital image and the reconstruction of the digital image.
[0032] The embedding in the codebook may be defined by at least one parameter. The machine learning system 200 is, for example, configured to determine at least one parameter that minimizes the difference. The machine learning system 200 may be configured to learn the encoder 202 or the decoder 204. The encoder 202 may be defined by at least one parameter. The decoder 204 may be defined by at least one parameter. The machine learning system 200 may be configured to determine at least one parameter of the encoder 202. The machine learning system 200 may be configured to determine at least one parameter of the decoder 204.
[0033] In one example, the digital image 108 includes pixels. In one example, the pixel includes three dimensions of color, such as red, green, and blue. The pixel can include more or fewer color dimensions, such as one dimension for a monochrome image, or four dimensions for encoding cyan, magenta, yellow, and black (i.e., black).
[0034] The digital image 108 has a resolution defined by the following dimensions: width, for example 256, height, for example 256, and a dimension of color. The width or height is provided, for example, as a number of pixels. The resolution is, for example, 256×256×3.
[0035] The encoder 202 is configured to encode the digital image 108 into tokens 206 in a low-dimensional latent space Z. The latent space Z has, for example, the following dimensions: 16x16x'embed_dim'. The dimension embed_dim may be a predetermined dimension or parameter. The machine learning system 200 may be configured to learn the parameter embed_dim.
[0036] Tokens 206 are quantized. In this example, each token 206 is replaced by the nearest embedding from the codebook dictionary. The machine learning system 200 can be configured to determine the nearest embedding of a token using a distance metric in the latent space Z, such as cosine distance.
[0037] The machine learning system 200 may include a generative adversarial network GAN, which includes a discriminator configured to discriminate between the digital image 108 and the reconstruction 114 of the digital image 108. The decoder 204 may decode the token 206 with a perceptual loss, a reconstruction loss, and a GAN loss. The GAN may include at least one parameter. The machine learning system 200 may be configured to train at least one parameter of the GAN.
[0038] Model 208 may be part of a second level.
[0039] The machine learning system 200 may be configured to train the model 208 to utilize the learned codebook dictionary to determine the reconstruction 212 of the token 206 , i.e., to determine the reconstruction 212 of the token 206 using the embedding of the codebook dictionary instead of the token 206 .
[0040] The machine learning system 200 may be configured to generate new digital images at inference time by randomly sampling using the model 208. The model 208 may include a transformer decoder to map embeddings of the codebook dictionary determined for the tokens 206 of the digital image 108 to a representation 210 of the digital image 108 in the latent space Z.
[0041] The input to model 208 includes tokens 206 .
[0042] The model 208 may be configured to map the tokens 206 to the representations 210 in a diffusion process. The model 208 may be configured to map the representations 210 to the reconstruction of the tokens 212 in a denoising process.
[0043] The diffusion process may be a Markov chain to gradually add Gaussian noise q to the input of the model 208: where is N1 normally distributed, and is a fixed or known variance table.
[0044] The representation 210 of the digital image 108 is determined, for example, in the latent space Z as: where z0, z t ∈Z
[0045] The denoising process can be parameterized by a Gaussian distribution: p Θ (z t-1 , z t ):=N2(z t-1 ;μ Θ (z t , t), σΘ (z t , t)) where μ Θ (z t , t) is expressed as z t and prediction noise ε Θ (z t , t). The prediction noise ε Θ (z t , t) can be modeled by a convolutional neural network, such as the U-Net described in the following paper: U-Net: Convolutional Networksfor Biomedical Image Segmentation, Olaf Ronneberger, Philipp Fischer, Thomas Brox, arXiv:1505.04597.
[0046] The prediction noise ε can be learned according to the training objective Θ (z t , t) parameters Θ. The training goal in the example is
[0047] After the model 208 is trained, it means that z T It can be selected by the user or randomly sampled from Gaussian distribution N2. Given z from t=T to t=1 t , z can be obtained sequentially t-1 To determine the reconstructed digital image Token Reconstruction or new digital image New token
[0048] The method for machine learning may include training the machine learning system 200. The method for machine learning may include testing, verifying, or validating the machine learning system 200. The method for machine learning may include generating training data for training the machine learning system 200. The method for machine learning may include generating test data for testing, verifying, or validating the machine learning system 200.
[0049] Figure 3 A first example of an encoder 202 is schematically depicted. Figure 4 A first example of a decoder 204 is schematically depicted.
[0050] The encoder 202 according to the first example is object-aware. The encoder 202 according to the first example is trained to associate a portion of the representation 210 of the digital image 108 with a learned object representation 302. The encoder 202 according to the first example includes a storage unit. In this example, the learned object representation 302 is stored in the storage unit. According to the first example, the class or class label of the object is unknown or unused.
[0051] The encoder 202 according to the first example is position-aware. The encoder 202 according to the first example is trained to associate portions of the representation 210 of the digital image 108 with the object position map 304. The object positions in the object position map 304 may be learned. In this example, the object positions may be stored in a storage unit.
[0052] In the first example, the digital image 108 depicts a first object 108-1, a second object 108-2, a third object 108-3, and a background 108-4. In the first example, the object representation 302 includes a first object representation 302-1 associated with the first object 108-1, a second object representation 302-2 associated with the second object 108-2, a third object representation 302-3 associated with the third object 108-3, and a fourth object representation 302-4 associated with the background 108-4. The object representation 302 may include object representations of more or fewer objects.
[0053] In the first example, the digital image 108 depicts a first object 108-1 at a first position, a second object 108-2 at a second position, a third object 108-3 at a third position, and a background 108-4 at several other positions. In the first example, the object position map 304 includes a first position representation 304-1 associated with the first object 108-1, a second position representation 304-2 associated with the second object 108-2, a third position representation 304-3 associated with the third object 108-3, and several position representations 304-4 associated with the background 108-4. The object representation 302 may include position representations of more or fewer objects.
[0054] The encoder 202 according to the first example is trained to pick a vector from a codebook in a codebook dictionary.The encoder 202 according to the first example is trained to pick a vector for a portion of a digital image 108 depicting one of the objects from a codebook associated with the object based on an object position map 304 .
[0055] The decoder 202 according to the first example is trained to determine the reconstruction 114 of the digital image 108 from the object center representation 402 .
[0056] In a first example, the object center representation 402 includes a first object center representation 402-1 associated with the first object 108-1 and located at a first point in the object center representation 402 associated with a first portion of the reconstruction 114 of the digital image 108, a second object center representation 402-2 associated with the second object 108-2 and located at a second point in the object center representation 402 associated with a second portion of the reconstruction 114 of the digital image 108, a third object center representation 402-3 associated with the third object 108-3 and located at a third point in the object center representation 402 associated with a third portion of the reconstruction 114, and a fourth object center representation 402-4 associated with the background 108-4 and located at several points in the object center representation 402 associated with the background portion of the reconstruction 114 of the digital image 108. The object center representation 402 may include position representations of more or fewer objects.
[0057] The decoder 202 according to the first example is trained to determine a reconstruction of an object in a portion of the reconstruction 114 of the digital image 108 from the vector determined for the object. The decoder 202 according to the first example is trained to select the vector for the reconstruction of the portion of the reconstruction 114 from the latent space Z associated with the object based on the object position map 304.
[0058] According to a first example, the machine learning system 200 synthetically determines a reconstruction 114 of the digital image 108 using the object center representation and the corresponding position information.
[0059] Figure 5 A second example of an encoder 202 is schematically depicted. Figure 6 A second example of a decoder 204 is schematically depicted.
[0060] As described for the encoder 202 and decoder 204 according to the first example, the encoder 202 and decoder 204 according to the second example are object-aware and position-aware. According to the second example, a class or label of a class of objects is used. The encoder 202 according to the second example can be configured to determine the label of the object detected in the digital image 108 based on the digital image 108. The label can be determined by a classifier trained to classify objects. According to the second example, the codebook in the codebook dictionary is associated with the label identifying the object associated with the codebook. The encoder 202 according to the second example is configured to select a codebook of tokens 206 for determining a portion of the digital image 108 from the codebook dictionary based on the label assigned to the object detected in the portion of the digital image 108.
[0061] In this example, the digital image 108 includes a cat 502-1 and a dog 502-2 as well as grass 502-3 and a tree 502-4. The codebook dictionary 504 includes a first codebook 504-1 associated with a first label 506-1, a second codebook 504-2 associated with a second label 506-2, a third codebook 504-3 associated with a third label 506-3, and a fourth codebook 504-4 associated with a fourth label 506-4. In this example, the first label 506-1 is a cat, the second label 506-2 is a dog, the third label 506-3 is grass, and the fourth label 506-4 is a tree. The codebook dictionary 504 may include codebooks for more or fewer labels.
[0062] The encoder 202 according to the second example is trained to associate portions of the representation 210 of the digital image 108 with an object position map 508. Object positions in the object position map 508 are learned. In this example, the object position map 508 is stored in a storage unit.
[0063] In the second example, the digital image 108 depicts a cat 502-1 in a first portion of the digital image 108, a dog 502-2 in a second portion of the digital image 108, grass 502-3 in portions of the digital image 108, and trees in portions of the digital image 108. In the second example, the object location map 508 includes a first location representation 508-1 associated with a first tag, a second location representation 508-2 associated with a second tag, several third location representations 508-3 associated with a third tag, and several location representations 508-4 associated with a fourth tag. The object representations 508 may include location representations of more or fewer objects.
[0064] The encoder 202 according to the second example is trained to pick a vector from a codebook in a codebook dictionary.The encoder 202 according to the second example is trained to pick a vector from a codebook associated with the label for a portion of the digital image 108 classified into a class with the label.
[0065] The decoder 202 according to the second example is trained to determine the reconstruction 114 of the digital image 108 from the object center representation 602 .
[0066] In a second example, the object center representation 602 includes a first object center representation 602-1 associated with a first label and located at a first point in the object center representation 602 associated with a first portion of the reconstruction 114 of the digital image 108, a second object center representation 602-2 associated with a second label and located at a second point in the object center representation 602 associated with a second portion of the reconstruction 114 of the digital image 108, a third object center representation 602-3 associated with a third label and located at a third point in the object center representation 602 associated with a third portion of the reconstruction 114, and a fourth object center representation 602-4 associated with a fourth label and located at a fourth point in the object center representation 602 associated with a fourth portion of the reconstruction 114 of the digital image 108. The object center representation 602 may include position representations of more or fewer objects.
[0067] The decoder 202 according to the second example is trained to determine a reconstruction of an object in a portion of the reconstruction 114 of the digital image 108 from a vector determined for the object by the label. The decoder 202 according to the second example is trained to select a vector for the reconstruction of the portion of the reconstruction 114 from a latent space Z associated with the object by the label based on the object position map 504.
[0068] According to a second example, the decoder 204 is configured to reconstruct a cat in a first portion 604 - 1 of the reconstruction 114 , a dog in a second portion 604 - 2 of the reconstruction 114 , grass in a third portion 604 - 3 of the reconstruction 114 , and a tree in a fourth portion 604 - 4 of the reconstruction 114 .
[0069] According to a second example, the machine learning system 200 utilizes the object center representation and the corresponding position information and the labels to synthetically determine the reconstruction 114 of the digital image 108.
[0070] A specific instantiation can be to introduce a codebook dictionary instead of slots according to the slot attention mechanism in Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf, Object-centric learning with slot attention, 2020.
[0071] The machine learning system 200 can be configured to initialize a codebook dictionary instead of a slot, learn to bind a codebook of the codebook dictionary to a specific object in a digital image, and process the codebook dictionary according to a slot attention mechanism. This can be done by storing object representations in a per-object codebook instead of in slots, and using them to compute a position map as an attention map containing its position information. By storing the codebook dictionary and the representation of the object in the slot, the slot attention mechanism can be extended.
[0072] Figure 7 The inference of the decoder 202 according to the second example is schematically described.
[0073] The machine learning system 200 may be configured to provide an object map 702. The machine learning system 200 may be configured to provide a new digital image 704 according to the object map 702. The object map 702 in the example includes a first portion 702-1 representing a first portion 704-1 of the new digital image 704, a second portion 702-2 representing a second portion 704-2 of the new digital image 704, a third portion 702-3 representing a third portion 704-3 of the new digital image 704, and a fourth portion 702-4 representing a fourth portion 704-4 of the new digital image 704. The object map 702 identifies objects represented by corresponding tags in portions of the object map. In this example, the first portion 702-1 of the object map 702 is associated with a first tag (e.g., cat). In this example, the second portion 702-2 of the object map 702 is associated with a second tag (e.g., dog). In this example, the third portion 702-3 and the fourth portion 702-4 of the object map 702 are associated with a third tag (e.g., grass). The new digital image 704 includes a cat in a first portion 704 - 1 of the new digital image 704 , a dog in a second portion 704 - 2 of the new digital image 704 , and grass in third and fourth portions 704 - 3 and 704 - 4 of the new digital image 704 .
[0074] The machine learning system 200 may be configured to provide an object position map 706 based on the object map 702. In this example, the object position map 706 includes a finite number of portions corresponding to a finite number of tokens used to determine the new digital image 704.
[0075] In this example, the object position map 706 includes 3×3 portions labeled according to labels from the object map 702, i.e., a first label 706-1 for the first portion 702-1 identifying the object to be depicted as a cat, a second label 706-2 for the second portion 702-2 identifying the object to be depicted as a dog, and a third label 706-3 identifying the object to be depicted as grass.
[0076] The machine learning system 200 may be configured to sample an object-perceived noise map 708 that includes different vectors in different portions thereof, the vectors being sampled from the codebooks of the codebook dictionary 502 corresponding to the labels in the object position map 706. The vectors of the portion labeled with the first label 706-1 in the object position map 706 are, for example, sampled from the first codebook 504-1. The vectors of the portion labeled with the second label 706-1 in the object position map 706 are, for example, sampled from the second codebook 504-2. The vectors of the portion labeled with the third label 706-1 in the object position map 706 are, for example, sampled from the third codebook 504-3.
[0077] The object perceived noise map 708 in the example includes a vector 708-1 sampled from the first codebook 504-1 of the first portion 704-1 of the new digital image. The object perceived noise map 708 in the example includes six vectors 708-1 sampled from the second codebook 504-2 of the second portion 704-2 of the new digital image. In this example, other vectors 708-3 of the object perceived noise map 708 are sampled from the third codebook 504-3 of the third portion 704-3 of the new digital image.
[0078] Figure 8 The method for machine learning is schematically depicted.
[0079] The method includes step 802 .
[0080] At step 802, a token 206 is provided.
[0081] The token 206 may be determined from pixels of the digital image 108, in particular using the encoder 202. A predetermined token 206 may also be used.
[0082] The tokens 206 each represent a portion of the digital image 108, i.e., the pixels of the portion. The tokens 206 can each be determined based on the pixels in the portion of the digital image 108, i.e., without using other pixels of the digital image. The tokens 206 are determined at a lower resolution than the resolution of the pixels of the portion they represent.
[0083] The method includes step 804 .
[0084] At step 804, an embedding associated with the object is provided. In this example, a codebook dictionary is provided.
[0085] Step 804 may include providing embedding groups, such as separate codebooks for different objects. The groups may be associated with an object or a corresponding tag identifying the object.
[0086] In training, the method may include determining a mapping that associates an embedding with a location of a portion of the image 108 or a location of an object in the digital image 108. In this example, an embedding representing one of the tokens 206 is associated with an object, and the embedding is associated with a location of a portion of the image depicting the object or a location of the object in the digital image 108. The mapping is location information that enables location-specific reconstruction.
[0087] In training, providing the embeddings may include determining a plurality of embeddings to represent the tokens 206 that represent portions of the digital image 108 .
[0088] The map associates the embeddings with corresponding locations in the digital image 108 .
[0089] The method includes step 806 .
[0090] At step 806 , an embedding for the token 206 is selected using the model 208 .
[0091] Step 806 may include selecting a corresponding embedding from a set of embeddings associated with the object or tag.
[0092] The token 206 represents a quantization of the digital image 108. The embedding represents a codebook.
[0093] The embedding is selected from embeddings in a codebook of a codebook dictionary that is associated with an object of the digital image 108 that is included in the portion represented by the token 206 or has the same label as the portion.
[0094] The method includes step 808 .
[0095] In step 808 , a reconstruction 212 of a token 206 representing the object is determined from the representation 210 of the digital image 108 .
[0096] In inference, step 808 may include determining a location of a reconstructed object in the reconstruction 114 of the digital image 108 .
[0097] In inferring, step 808 may include providing a location of the reconstructed object in the digital image 704 including the reconstructed object.
[0098] In inference, step 808 may include providing an embedding for the reconstructed object of the portion of the digital image, and determining a token 212 representing the object of the reconstruction 114 of the digital image 108 based on the embedding.
[0099] In inferring, step 808 may include providing an embedding for an object of the digital image 704 and determining a token representing the object of the digital image based on the embedding.
[0100] The method includes step 810 .
[0101] In step 810 , the method includes determining a reconstruction 114 of the digital image 108 , the reconstruction 114 depicting a reconstruction of the object, in particular using the decoder 204 .
[0102] In inference, step 810 may instead include determining a reconstructed digital image 704 depicting the object from the reconstruction 212 of the tokens 206 representing the object. The decoder may determine a reconstruction of the digital image or a new digital image including the object.
[0103] In inference, step 810 may include determining a reconstructed object at a location in the reconstruction 114 of the digital image 108 .
[0104] Inferring, step 810 may include determining the reconstructed object at a location in the digital image 704 that includes the reconstructed object.
[0105] Step 810 may include determining a location of a reconstructed object in the reconstruction 112 of the digital image 108 or the reconstructed digital image 704 that depicts the object according to a mapping.
[0106] During training, the method may include step 812 .
[0107] Step 812 includes determining parameters defining at least one of the embeddings and / or parameters defining the model 208 based on the difference between the token 206 and the reconstruction 212 of the token 206 .
[0108] The steps may be repeated to perform training with training data including digital images 108 .
[0109] To achieve good performance with affordable computational requirements, the vector quantization (VQ) model or the latent diffusion model (LDM) can be modified. These models divide the learning process into two stages: (Stage 1) a compact representation of the image is learned; (Stage 2) this representation is used to train a generative model in a lower dimension, which saves computational cost and makes the process faster.
[0110] The VQ model modified according to the description herein can be used to first learn to encode and quantize images via a learnable codebook, and then learn how to compose such a codebook to generate new images. The LDM modified according to the description herein can be used to learn how to encode images, and learn how to generate images via a diffusion process at such an embedding level.
[0111] The first stage of these models can be modified as described herein to learn object-centric representations, which can be used to regulate the generation process at the object level. In this way, the modified model provides more control over the synthesis of scenes in the generated digital images.
Claims
1. A computer-implemented method for machine learning, characterized in that The method includes providing (804) an embedding associated with an object, providing (802) a token (206) representing a portion of a digital image (108) depicting at least a portion of the object, wherein the token (206) represents the portion at a resolution lower than a resolution of pixels of the portion, selecting (806) an embedding from the embeddings using a model (208) to represent the token in a representation (210) of the digital image (108), determining (808) a reconstruction (212) of the token (206) representing the object using the model (108) based on the representation (210) of the digital image (108), and determining (812) parameters defining at least one of the embeddings and / or defining parameters of the model (208) based on a difference between the token (206) and the reconstruction (210) of the token (206).
2. The method according to claim 1, characterized in that A set of embeddings is provided (804) for the object, and an embedding is selected (806) from the set of embeddings using the model (208).
3. The method according to claim 2, characterized in that Providing (804) a set of embeddings associated with the object by a corresponding label, determining (802) the label of the object based on pixels of the portion of the digital image (108) using the model (208), and selecting (806) an embedding set for the object associated with the label.
4. The method according to any one of the preceding claims, characterized in that A token is determined (802) from pixels of a portion of a digital image (108), in particular using an encoder (202).
5. The method according to any one of the preceding claims, characterized in that Based on the reconstruction of the tokens representing the object, a reconstruction (114) of a reconstructed digital image (108) depicting the object or a reconstructed digital image (704) depicting the object is determined (810), in particular using a decoder (204).
6. The method according to claim 5, characterized in that The method includes providing (808) a position of a reconstructed object in a reconstruction (114) of a digital image (108) and determining the position of the reconstructed object in the reconstruction (114) of the digital image (108), or providing a position of a reconstructed object in a digital image (704) including the reconstructed object and determining the position of the reconstructed object in the digital image (704) including the reconstructed object.
7. The method according to claim 5, characterized in that The method includes providing (808) an embedding for a reconstructed object of the portion of the digital image (108), determining a token (212) representing the object of the reconstruction (114) of the digital image (108) based on the embedding, and determining the reconstruction (114) of the digital image (108) based on the token (212) representing the object of the reconstruction (114) of the digital image (108), or providing (808) an embedding for an object of the digital image (704), determining a token representing the object of the digital image based on the embedding, and determining the digital image (704) based on the token representing the object of the digital image (704).
8. The method according to any one of claims 5 to 7, characterized in that A mapping is determined that associates embeddings representing tokens associated with the object with locations of portions of the image or locations of the object in the digital image.
9. The method according to claim 8, characterized in that A position of a reconstructed object in a reconstruction (112) of the digital image (108) or in a reconstructed digital image (704) depicting the object is determined (810) based on the mapping.
10. The method according to claim 8 or 9, characterized in that: The method includes determining (804) a plurality of embeddings to represent tokens (206) representing a plurality of portions of a digital image (108), wherein the mapping associates the embeddings with corresponding locations in the digital image (108).
11. A device (100) for machine learning, characterized in that: The device comprises at least one processor (102) and at least one memory (104), the memory being configured to store instructions executable by the at least one processor (102) and which, when executed by the at least one processor (102), cause the device (100) to perform a method according to one of claims 1 to 10.
12. A program characterized by The program comprises instructions which, when executed by a computer, cause the computer to perform the method according to one of claims 1 to 10 .