Method for finding matched semantic names for regions of digital images and related methods, data structures and devices

Abstract

An apparatus, data structure, and computer-implemented method for training for finding a matching semantic name for a region of a digital image and for finding a matching semantic name for a region of a digital image wherein the method for training comprises: providing a digital image and a category name and an indicator identifying a region in the digital image, providing a set of candidate names according to the category name, determining a code of the digital image, determining a multi-scale feature according to the code of the digital image, determining an embedding of the candidate names, and embedding the candidate names according to an output of a converter decoder for the embedding of the candidate names, the multi-scale feature and the masks, in one embodiment, the method includes determining an output of a prediction category including a prediction indicator and a corresponding candidate name, determining a pixel-by-pixel feature from a multi-scale feature, and training a transformer decoder from a loss including a difference between the indicator and the prediction indicator, the prediction indicator better reproducing the indicator than other prediction indicators.

Description

Background Art

[0001] The present invention relates to a computer-implemented method, data structure and device for finding matching semantic names for regions of a digital image, or for training, in particular training, a transformer decoder and / or a pixel decoder for finding matching semantic names for regions of a digital image.

[0002] The Transformer Decoder with Masked Attention assigns different semantic names to different regions of a digit image.

[0003] Invention Disclosure

[0004] The computer implemented method, data structure and apparatus according to the independent claims provide for a refinement of semantic names given to regions of digital images.

[0005] A computer-implemented method for finding matching semantic names for regions of a digital image comprises providing a digital image and a category name, providing an indicator (particularly a bounding box or a mask) identifying a region in the digital image, providing a set of candidate names based on the category name, determining an encoding of the digital image, determining multi-scale features based on the encoding of the digital image, determining embeddings for the candidate names, determining an output comprising a predicted indicator (particularly a predicted bounding box or a predicted mask) and a predicted category for the corresponding candidate name based on the embedding of a transformer decoder for the candidate names, the multi-scale features, and the indicator, and selecting, based on the output, a candidate name from a set associated with the predicted indicator that better reproduces the indicator than other predicted indicators as the semantic name for the region of the digital image. The method refines the name of each region and is therefore applicable to digital images having complex scenes.

[0006] Providing candidate names can include querying a suggestible language model, particularly a Generative Pre-Trained Transformer (GPT), to output candidate names for the category name. The GPT can generate multiple plausible names.

[0007] The method may include providing a caption for the digital image or determining a caption from the digital image (particularly using an image caption model), and wherein providing candidate names includes selecting candidate names for the category name from the caption. The caption includes a plurality of plausible names.

[0008] The method may include utilizing a visual encoder (particularly a contrastive language-image pretrained (CLIP) image encoder or method DINO) to determine an encoding for the digital image, and / or utilizing a text encoder (particularly a contrastive language-image pretrained text encoder) to determine an embedding for the candidate name. A paired visual encoder and text encoder (e.g., CLIP) may be used. It is not required to use a paired visual encoder and text encoder. For example, DINO may be used as the visual encoder, and the text encoder portion of CLIP may be used.

[0009] The method may include determining an output embedding using a transformer decoder, wherein the transformer decoder includes a masked crisscross attention layer having an input of an embedding and multi-scale features for the candidate names and a mask, wherein the masked crisscross attention layer is followed by a self-attention layer, wherein the self-attention layer is followed by a feed-forward network, wherein the feed-forward network is configured to output the output embedding. This order reduces computational resources required to implement the transformer decoder.

[0010] The method may comprise determining pixel-wise features based on the multi-scale features, in particular determining the multi-scale features and the pixel-wise features using a pixel decoder based on encoding of the digital image.

[0011] A computer-implemented method for training to find matching semantic names for regions of a digital image comprises providing a digital image and a class name and a mask, the mask identifying the region in the digital image, providing candidate names based on the class name, determining an encoding of the digital image, determining multi-scale features based on the encoding of the digital image, determining embeddings for the candidate names, determining an output comprising a predicted indicator (particularly a predicted bounding box or a predicted mask) and a predicted class for the corresponding candidate name based on the embeddings of a transformer decoder for the candidate names, the multi-scale features, and the indicators, determining pixel-wise features based on the multi-scale features, and training the transformer decoder based on a loss comprising the difference between the indicator and a predicted indicator, the predicted indicator reproducing the indicator better than other predicted indicators. The method trains the transformer decoder to refine the name of each region and is therefore applicable to digital images with complex scenes.

[0012] The method for training may comprise determining output embeddings for a plurality of candidate names for the same digit image and mask, wherein the loss comprises a linear mapping, particularly a normalization, of the output embeddings.

[0013] The method for training may include determining multi-scale features and pixel-wise features using a pixel decoder based on an encoding of a digital image, and training the pixel decoder based on a loss.

[0014] The training method may include using a visual encoder to determine an encoding for a digital image, using a text encoder to determine an embedding for a candidate name, and keeping the visual encoder and / or the text encoder unchanged during training. This means that a pre-trained, off-the-shelf text encoder or visual encoder may be used.

[0015] A device for finding matching semantic names for regions of a digital image, or for training, in particular training a transformer decoder and / or a pixel decoder, to find matching semantic names for regions of a digital image, comprising at least one processor and at least one memory, wherein the at least one memory stores instructions which, when executed by the at least one processor, cause the device to perform a method for finding matching semantic names or for training.

[0016] A data structure for finding matching semantic names for regions of a digital image, or for training, in particular training, a transformer decoder, to find matching semantic names for regions of a digital image, comprising at least one data field for the digital image, for a category name, for a mask for identifying regions in the digital image, for candidate names determined based on the category name, for an encoding of the digital image, for multi-scale features determined based on the encoding of the digital image, for embeddings for the candidate names, for outputting embeddings (in particular, for output embeddings of the transformer decoder of the embeddings of the candidate names, the multi-scale features and indicators), and for outputting outputs representing candidate names determined based on the output embeddings.

[0017] In particular, for training, the data structure may include at least one data field for a pixel-wise feature determined based on the multi-scale feature, and for a loss comprising a pixel-wise multiplication result of the pixel-wise feature with an output representing the candidate name.

[0018] The data structure may comprise at least one data field for outputting the embedded, in particular normalized, linear mapping.

[0019] Further advantageous embodiments can be derived from the following description and the accompanying drawings, in which:

[0020] Figure 1 Schematically depicted is a device for finding matching semantic names for regions of a digital image, or for training, in particular training a transformer decoder and / or a pixel decoder, for finding matching semantic names for regions of a digital image,

[0021] Figure 2 schematically depicts an overview of the training of a transformer decoder and / or pixel decoder,

[0022] Figure 3The generation of candidate names is schematically depicted.

[0023] Figure 4 The data structure is schematically depicted,

[0024] Figure 5 A flowchart depicting the steps of a method for finding a matching semantic name for a region of a digital image,

[0025] Figure 6 A flowchart depicting the steps of a method for training.

[0026] Figure 1 The device 100 is schematically depicted. The device 100 comprises at least one processor 102 and at least one memory 104.

[0027] According to an example, the device 100 is configured to find a matching semantic name for a region of a digital image.

[0028] According to an example, the device is configured for training, in particular training a transformer decoder and / or a pixel decoder, to find matching semantic names for regions of a digital image.

[0029] The at least one memory 104 stores instructions that, when executed by the at least one processor 102 , cause the apparatus to perform a method for finding matching semantic names or for training.

[0030] Figure 2 A model 200 for finding matching semantic names is schematically depicted. The model 200 comprises a transformer decoder 202 and a pixel decoder 204.

[0031] Figure 2 An overview of the training of the transformer decoder 202 and / or the pixel decoder 204 is schematically depicted.

[0032] Transformer decoder 202 in this example is an artificial neural network. Transformer decoder 202 in this example includes L layers. Transformer decoder 202 in this example includes a cross-attention layer 206, a self-attention layer 208, and a feedforward network 210 in each layer I. Cross-attention layer 206 is followed by self-attention layer 208. Self-attention layer 208 is followed by feedforward network 210. In this example, "followed" means that the output of cross-attention layer 206 is the input to self-attention layer 208. In this example, the output of self-attention layer 208 is the input to feedforward layer 210. There may be layers between these layers.

[0033] The crisscross attention layer 206 includes inputs of the mask 212 and the multi-scale features 214 from the pixel decoder 204 and the embedding 216 of the candidate name 218.

[0034] The mask 212 is an example of an indicator indicating a region. As an alternative to the mask 212, the indicator may be a bounding box.

[0035] In this example, a set 220 of candidate names 218 is provided. In this example, the set 220 of candidate names 218 is associated with category c. i The candidate name 218 is provided from a set 220 of candidate category names 218 .

[0036] For the embedding 216 of the candidate class name 218, the transformer decoder 202 is configured to output an output embedding 222. The model 200 is configured to determine the output of the model based on the output embedding 222. The output of the model includes a prediction mask and a predicted class for the corresponding candidate name 218.

[0037] In the case where the indicator is a bounding box, the model 200 may be configured to output an output of the model that includes a predicted bounding box rather than a predicted mask.

[0038] The transformer decoder 202 is configured to iteratively determine the output 222 in L iterations. According to this example, the embedding 216 of the candidate name 218 is the initial input X (0) And the output 222 is the result of the last iteration X (L) In this example, the iterative input X (l-1) and the iterative output X (l) The dimension is N×C.

[0039] The transformer decoder 202 in the example is configured as described in Bowen Cheng, Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar, “Masked-attention mask transformer for universal image segmentation. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022 (Mask2Former)”.

[0040] The embedding 216 of the candidate category name 218 in the example is determined using a text encoder 224. The text encoder 224 is configured to encode the candidate name 218 into its embedding 216. The text encoder 224 can be CLIP. The text encoder 224 in the example is pre-trained.

[0041] Figure 2Depicted are a plurality of candidate class names 218 , their respective embeddings 216 , and outputs 222 of the transformer decoder 202 for the embeddings 216 of the corresponding candidate names 218 .

[0042] The pixel decoder 204 is configured to determine the multi-scale features 214 based on an embedding 224 of a digital image 226. In this example, the embedding 224 of the digital image 226 is determined using a visual encoder 228.

[0043] The visual encoder 228 can be CLIP or DINO. The visual encoder 228 in this example is pre-trained.

[0044] The pixel decoder 204 is configured to output pixel-wise features 230 of dimensions H×W×C, denoted as F pix .

[0045] Output 222 of transformer decoder 202 for iteration 1, intermediate mask Determined as the pixel-wise product 232 of the pixel-wise feature 230 and the corresponding candidate name 218 .

[0046] The pixel-wise product 232 in this example is the intermediate mask It is determined to be:

[0047]

[0048] Where * refers to pixel-level multiplication.

[0049] For the output 222 of the iterative transformer decoder 202, a linear layer 234 is used to determine the predicted class The linear layer 234 outputs a linear mapping 236, i.e., the category

[0050]

[0051] According to the result X of the last iteration L (L) To make the final prediction, that is, the final mask in the output of model 200 and the final category

[0052] The mask 212 of the training data point i is denoted as M i Mask M i A region in the digital image 226 is identified.

[0053] For training, we use L for training data point i i The loss 238 depends on the mask M i , prediction category The best prediction category and mask The best predicted mask

[0054]

[0055] Among them L mask is the mask localization function in Mask2Former and L class is the classification loss function in Mask2Former.

[0056] The training data includes the true mask M i and category c i With the visual encoder and the text encoder both kept frozen, the transformer decoder 202 is trained together with the pixel decoder 204 for the segmentation task. i and category c i , the transformer decoder 202 utilizes i Multiple mask and category predictions are performed on the candidate names of M. i The prediction with the highest Intersection over Union (IoU) is used for loss calculation.

[0057] According to the example, to provide additional supervision on the quality of the names, the candidate names in the training data can be appended with a i A randomly selected "negative" name from different categories. If the IoU score of the "negative" name is the highest, the prediction result is supervised using the empty mask and the "empty" category. The empty category is an additional category in addition to the training categories:

[0058]

[0059] With both positive and negative supervision, we effectively incentivize the model to prefer high-quality names and penalize low-quality ones, thereby helping to accurately identify the best matching name for each segment.

[0060] According to the example, after training with the training data including negative names, the model 200 again performs well on the dataset with the true class c i And run on the training data without negative names, that is, use the true mask M i As an attention bias and no longer attached to the negative name. In this example, the generated i The candidate name of the mask prediction result with the highest IoU and the true mask M i associated.

[0061] According to the example, for the true category c i, the candidate names selected by the model 200 for the training data are summarized and ranked, e.g., based on their comparison with the ground-truth mask M i To keep the most accurate names and discard less common, possibly noisy candidate names, the top candidate names can be selected so that they cover, for example, at least 90% of the total frequency. The top candidate names are the true class c i The name of the update.

[0062] True category c i The top candidate names are higher than the true category c i The original category names are more descriptive and precise. When using the top candidate names as text prompts, the pre-trained open vocabulary model achieves much higher segmentation performance than using the original category names.

[0063] Figure 3 An overview of the generation of candidate names is schematically depicted.

[0064] Candidate names are generated based on category c i digital image.

[0065] Category c i The digital image 302 is provided to an image caption model 304. The image caption model 304 is configured to output caption nouns 306 for the digital image 302 from the collection. According to an example, the image caption model 304 outputs the nouns: tree, sidewalk, hedge, mountain bike, bike rack, grass, sidewalk, building, saddle bag. In this example, the caption nouns 306 are for the category c that includes the digital image 302. i is determined by a collection of digital images.

[0066] Image caption model 304 for category c i The nouns in the captions generated from the collection of digital images are aggregated and sorted into categories c i Context names 308. According to an example, context names 308 include nouns: building, road, person, park, sidewalk, car, walk, city street, bicycle, street corner.

[0067] Category c i Associated with category name 310. In this example, the category c associated with category name 310 i A set 220 of candidate names 218 is provided. According to this example, the category name "bicycle" is associated with category c i associated.

[0068] Category ci The category name 310 and the category c i The generated context name 308 is input to a Generative Pre-Trained Transformer (GPT) 312, which is configured to output the category c i A set 220 of candidate names 218 is provided.

[0069] In this example, GPT 312 outputs: city bike, bicycle, road bike, tandem bike, cruiser, sidewalk bike, leisure bike, mountain bike, mobility bike.

[0070] According to K categories c i For example, candidate name 218 is for the corresponding category c i , from the digital image collection and the corresponding category c i Determined in the category name.

[0071] This means that GPT 312 selects candidate names from the caption of the digital image. The caption includes multiple plausible names. Candidate names can also be selected directly from the caption.

[0072] Figure 4 A data structure 400 for finding matching semantic names for regions of a digital image 226 is schematically depicted.

[0073] The data structure 400 includes at least one data field 402 for the digital image 226, for the category name 310, a mask 212 for identifying regions in the digital image 226, at least one candidate name 218 determined based on the category name 310, an encoding 214 for the digital image 226, multi-scale features 214 determined based on the encoding 214 of the digital image 226, an embedding 216 for the candidate name 218, and an output 222 for representing the candidate name 218, the candidate name 218 being determined based on the embedding 216 of the candidate name 218, the multi-scale features 214, and the mask 212.

[0074] The data structure 400 can be particularly used for training, and the data structure 400 includes at least one data field 402, at least one data field 402 for the pixel-by-pixel feature 230 determined based on the multi-scale feature 214, and for the loss 238, the loss 238 including the pixel-level multiplication result 232 of the pixel-by-pixel feature 230 and the output 222 representing the candidate name 218.

[0075] The data structure 400 may include at least one data field 402 for outputting 222 , in particular the normalized linear mapping 236 .

[0076] Figure 5 A flow chart including steps of a computer-implemented method for finding a matching semantic name for a region of a digital image 226 is depicted.

[0077] The steps of the method are described with respect to a digital image 226 associated with a category name 310 .

[0078] The method includes step 502 .

[0079] Step 502 includes providing 502 the digital image 226 and the category name 310 .

[0080] In this example, category name 310 is associated with digital image 226. For example, classification of digital image 226 is performed such that category name 310 is associated with digital image 226.

[0081] The method includes step 504 .

[0082] Step 504 includes providing mask 212. Mask 212 identifies regions in digital image 226.

[0083] The method includes step 506 .

[0084] Step 506 includes providing at least one candidate name 218 based on the category name 310. In this example, a set 220 of candidate names 218 is provided.

[0085] According to an example, providing the at least one candidate name 218 includes querying the generative pre-trained transformer 312 to output the set 220 of candidate names 218 for the category name 310 . Providing the at least one candidate name can include selecting the candidate name 218 for the category name 310 from the description text 306 .

[0086] According to an example, the method includes providing a caption for the digital image 226. For example, the caption includes the caption noun 306. According to an example, the method includes determining a caption, such as the caption noun 306, based on the digital image 226.

[0087] The method may include determining the caption using a caption model. In the example, the caption noun 306 is determined using the image caption model 304 .

[0088] The method includes step 508 .

[0089] Step 508 includes determining the code 214 of the digital image 226 .

[0090] The method may include determining an encoding 224 of a digital image 226 using a visual encoder 228. The visual encoder 228 may be a CLIP neural network or a method DINO.

[0091] The method includes step 510 .

[0092] Step 510 includes determining the embedding 216 for the candidate name 218. The method may include determining the embedding 216 for the candidate name 218 using a text encoder 224, particularly a CLIP neural network.

[0093] For a candidate name 218 in the set 220 of candidate names 218 , step 510 is performed.

[0094] The method includes step 512 .

[0095] Step 512 includes determining the multi-scale features 214 based on the code 214 of the digital image 226 .

[0096] Step 512 includes determining pixel-wise features 230 based on multi-scale features 214 , which are dependent on encoding 224 of digital image 226 .

[0097] According to an example, the multi-scale features 214 and the pixel-by-pixel features 230 are determined using the pixel decoder 204 .

[0098] In this example, pixel-by-pixel features 230 are determined for number of candidate names 218 .

[0099] The method includes step 514 .

[0100] Step 514 includes determining an output of the model 200 . Determining the output of the model 200 includes determining an output 222 of the corresponding candidate names 218 of the set 220 based on the corresponding embeddings 216 , the multi-scale features 214 , and the masks 212 .

[0101] Determining the output of model 200 includes determining output 222 of transformer decoder 202 and the output of model 200 in an iteration.

[0102] The output of model 200 is the final prediction, i.e. the final mask and the final category The final prediction is based on the result X of the last iteration L. (L) And make.

[0103] The output 222 in this example is determined using the transformer decoder 202 .

[0104] This means that for at least one candidate name 218 , an output 222 is determined using the transformer decoder 202 , which includes a masked cross-attention layer 206 having inputs of the embedding 216 and multi-scale features 214 for the candidate name 218 , and the mask 212 .

[0105] This means that, for digital image 226 and mask 212 , output 222 is determined for candidate name 218 in set 220 of candidate names 218 .

[0106] The method includes step 516 .

[0107] Step 516 includes: selecting and predicting a mask based on the output of model 200 The candidate name 218 is associated as the semantic name of the region of the digital image 226, and the predicted mask than other prediction masks The mask 212 is better reproduced.

[0108] This means that, among the set of candidate names 220 , the candidate name 218 associated with the prediction indicator that replicates the indicator 212 better than the other prediction indicators is selected as the semantic name of the region of the digital image 226 according to the output of the model 200 .

[0109] According to the example, the corresponding prediction mask is calculated The correlation metric (e.g., the IoU ratio) with the mask 212 is used as the correlation metric. The correlation metric (e.g., IoU ratio) indicates the predicted mask The extent of the mask 212 is reproduced.

[0110] A relevance metric may be determined for the candidate names 218 in the set 220 of candidate names 218 .

[0111] According to an example, at least one candidate name 218 is selected as a semantic name for a region of the digital image 226 that produces a metric indicating a higher correlation than other metrics, such as a higher intersection over union.

[0112] According to an example, the candidate name 218 that produces a metric indicating the greatest relevance is selected.

[0113] Figure 6 A flowchart depicting the steps of a method for training.

[0114] The method for training includes step 602 .

[0115] Step 602 includes providing training data.

[0116] The training data includes a plurality of digital images associated with corresponding ground-truth categories and corresponding ground-truth masks.

[0117] The training data includes multiple real categories c i and the true mask M i .

[0118] Category name 310 is the true category c i Mask 212 is the real mask M i .

[0119] The method for training includes running the model 200 as described with respect to the example class names 310 and the example ground truth masks 212 .

[0120] The training data includes, for example, the digital image 226 and the class names 310 and the mask 212 identifying regions in the digital image 226 .

[0121] The method for training includes performing the following steps for each digital image in the training data: The following steps are described using the example of a digital image 226 associated with a class name 310 .

[0122] The method for training includes step 604 .

[0123] Step 604 includes providing the candidate names 218 based on the category name 310 .

[0124] Step 604 includes providing the set 220 of candidate names 218 based on the category name 310 .

[0125] As described in step 506 , the candidate category names 218 may be determined.

[0126] According to an example, for a ground truth class 310, candidate names 218 selected by the model 200 for training data from the set 220 of candidate names 218 are summarized and ranked, for example, based on how often they match the ground truth mask 212. To retain the most accurate names and discard less common, potentially noisy candidate names, the top-ranked candidate names can be selected so that they cover, for example, at least 90% of the total frequency. The top-ranked candidate names are the updated names for the ground truth class 310.

[0127] The method for training includes step 606 .

[0128] Step 606 includes determining the code 224 of the digital image 226 .

[0129] The method for training may include determining an encoding 224 of a digital image 226 using a visual encoder 228 .

[0130] The method for training includes step 608 .

[0131] Step 608 includes determining the embedding 216 of the candidate name 218 in the set 220 of candidate names 218 .

[0132] The method for training may include determining embeddings 216 for candidate names 218 using a text encoder 224 .

[0133] The method for training includes step 610 .

[0134] Step 610 includes determining the multi-scale features 214 based on the code 214 of the digital image 226 .

[0135] The method for training includes step 612 .

[0136] Step 612 includes determining pixel-by-pixel features 230 based on multi-scale features 214 .

[0137] According to an example, the multi-scale features 214 and the pixel-by-pixel features 230 are determined using the pixel decoder 204 based on the encoding 214 of the digital image 226 .

[0138] The method for training includes step 614 .

[0139] Step 614 includes determining an output of the model 200. Determining the output of the model 200 includes determining an output 222 representing the candidate name 218.

[0140] Determining the output of model 200 includes determining output 222 of transformer decoder 202 and the output of model 200 in an iteration.

[0141] The output of model 200 is the final prediction, i.e. the final mask and the final category The final prediction is based on the result X of the last iteration L. (L) And make.

[0142] In this example, the transformer decoder 202 is utilized to determine an output 222 for the corresponding candidate name 218 based on the embedding 216 , the multi-scale features 214 , and the mask 212 of the corresponding candidate name 218 .

[0143] This means that the output 222 may be determined for multiple candidate names 218 for the same digital image 226 and mask 212 .

[0144] The method for training includes step 616 .

[0145] Step 616 includes training the transformer decoder 202 according to the loss 238. Step 616 may include training the pixel decoder 204 according to the loss 238.

[0146] In this example, the generated mask M i The candidate name of the mask prediction result with the highest IoU is used to determine the loss 238.

[0147] Methods for training may include keeping the visual encoder 228 and / or the text encoder 224 constant during training.

[0148] Category name 310 is the true category c i Mask 212 is the real mask M i .

[0149] The methods used for training include: i and the true mask M i The following steps of the training method are described for running the model 200.

[0150] According to an example, the method for training includes running the model 200 with training data including negative names.

[0151] According to the example, after training with the training data including negative names, the model 200 again performs well on the dataset with the true class c i And run on the training data without negative names, that is, use the true mask M i As an attention bias and no longer has a negative name attached to it.

[0152] The method for training may include determining parameters of the transformer decoder 202 and / or the pixel decoder 204 using a gradient descent method based on a loss determined for the training data.

Claims

1. A computer-implemented method for finding a matching semantic name for a region of a digital image (226), characterized in that The method comprises providing (502) the digital image (226) and a category name (310), providing (504) an indicator identifying the region in the digital image (226), the indicator being in particular a bounding box or mask (212), providing (506) a set (220) of candidate names (218) based on the category name (310), determining (508) an encoding (214) of the digital image (226), determining (512) multi-scale features (214) based on the encoding (214) of the digital image (226), determining (510) an embedding (216) of the candidate names (218), and Based on the embedding (216) of the candidate class (218) name, the multi-scale features (214) and the output embedding (222) of the transformer decoder (202) of the indicator (212), determining (514) an output comprising a prediction indicator and a predicted class of the corresponding candidate name (218), the prediction indicator being in particular a prediction bounding box or a prediction mask, and selecting (516) from the set (220) associated with the prediction indicator the candidate name (218) that reproduces the indicator (212) better than other prediction indicators as the semantic name of the region of the digital image (226).

2. The method according to claim 1, characterized in that Providing (506) the candidate names (218) includes querying a suggestable language model, particularly a generative pre-trained transformer (312), to output candidate names (218) for the category name (310).

3. The method according to claim 1, characterized in that The method comprises providing a caption (306) for the digital image (226) or determining the caption (306) based on the digital image (226), in particular using an image caption model (304), and wherein providing (506) the candidate names comprises selecting the candidate names (218) for the category name (310) from the caption (306).

4. The method according to any one of the preceding claims, wherein the method comprises: An encoding (224) of the digital image (226) is determined (508) using a visual encoder (228), in particular a contrastive language-image pre-training (CLIP) neural network or method DINO, and / or an embedding (216) of the candidate name (218) is determined (510) using a text encoder (224), in particular a contrastive language-image pre-training (CLIP) neural network.

5. The method according to any one of the preceding claims, wherein the method comprises: The output embedding (222) is determined (514) using the transformer decoder (202), wherein the transformer decoder (202) includes a masked crisscross attention layer (206) having inputs of the embedding (216) and the multi-scale features (214) for the candidate name and the mask (212), wherein the masked crisscross attention layer (206) is followed by a self-attention layer (208), wherein the self-attention layer (208) is followed by a feed-forward network (210), wherein the feed-forward network (212) is configured to output the output embedding (222).

6. The method according to any one of the preceding claims, wherein the method comprises: A pixel-by-pixel feature (230) is determined (512) based on the multi-scale feature (214), in particular by using a pixel decoder (204) to determine the multi-scale feature (214) and the pixel-by-pixel feature (230) based on the encoding (224) of the digital image (226).

7. A computer-implemented method for training to find matching semantic names for regions of a digital image (226), characterized in that The method comprises providing (602) the digital image (226) and a category name (310) and an indicator identifying the region in the digital image (226), the indicator being in particular a bounding box or mask (212), providing (604) a set (220) of candidate names (218) based on the category name (310), determining (606) an encoding (224) of the digital image (226), determining (610) multi-scale features (214) based on the encoding (214) of the digital image (226), determining (608) an embedding (216) of the candidate name (218), and determining (609) an embedding (216) of the candidate name (218) based on the embedding (216) for the candidate name (218). ), the output embedding (222) of the transformer decoder (202) of the multi-scale features (214) and the indicator (212), determining (614) an output of a predicted category comprising a predicted indicator and a corresponding candidate name (218), the predicted indicator being in particular a predicted bounding box or a predicted mask, determining (612) pixel-wise features (230) based on the multi-scale features (214), and training (616) the transformer decoder (202) based on a loss (238), the loss (238) comprising the difference between the indicator (212) and the predicted indicator, the predicted indicator reproducing the indicator (212) better than the other predicted indicators.

8. The computer-implemented method of claim 7, wherein: The method comprises determining (614) the output embeddings (222) of a plurality of candidate names (218) for the same digital image (226) and mask (212), wherein the loss (238) comprises a linear mapping (236), in particular a normalization, of the output embeddings (222).

9. The computer-implemented method according to claim 7 or 8, wherein: The method includes determining (610, 612) the multi-scale features (214) and the pixel-wise features (230) using a pixel decoder (204) based on the encoding (214) of the digital image (226), and training (616) the pixel decoder (204) based on the loss (238).

10. The computer-implemented method according to any one of claims 7 to 9, wherein: The method includes determining (606) the encoding (224) of the digital image (226) using a visual encoder (228), determining (608) the embedding (216) of the candidate name (218) using a text encoder (224), and maintaining the visual encoder (228) and / or the text encoder (224) unchanged during training.

11. A device (100) for finding matching semantic names for regions of a digital image, or for training, in particular training, a transformer decoder (202) and / or a pixel decoder (204) to find matching semantic names for regions of a digital image, wherein the device (100) comprises at least one processor (102) and at least one memory (104), wherein the at least one memory (104) stores instructions which, when executed by the at least one processor (102), cause the device to perform a method according to any one of claims 1 to 10.

12. A data structure (400) for finding a matching semantic name for a region of a digital image (226), or for training, in particular training, a transformer decoder (204), to find a matching semantic name for a region of a digital image (226), wherein the data structure comprises at least one data field (402) for the digital image (226), a category name (310), an indicator, in particular a bounding box or mask (212), for identifying the region in the digital image (226), a candidate name (218) determined based on the category name (310), an encoding ( 214), multi-scale features (214) for determining based on the encoding (214) of the digital image (226), embeddings (216) for the candidate names (218), output embeddings (222) - in particular, the embeddings (216) for the candidate names (218), output embeddings of the transformer decoder (204) of the multi-scale features (214) and the indicators (212), and output, the output comprising a prediction indicator and a predicted category for the corresponding candidate name (218), the prediction indicator being in particular a prediction bounding box or a prediction mask, the corresponding candidate name (218) being determined based on the output embeddings (222).

13. The data structure (400) according to claim 12, in particular a data structure for training, characterized in that The data structure (400) includes at least one data field (402) for a pixel-by-pixel feature (230) determined from the multi-scale feature (214) and for a loss (238) comprising a difference between the indicator (212) and a predicted indicator that better replicates the indicator (212) than the other predicted indicator.

14. The data structure (400) according to claim 13, characterized in that The data structure (400) comprises at least one data field (402) for a linear mapping (236), in particular a normalization, of the output embedding (222).

Images