Representation-based image retrieval based on text query

Abstract

A method (100) for retrieving one or more images (2) from a set of images (3) based on a textual query (1), comprising the steps of: computing (110) a representation of the textual query (1) in a work space (4) as a query representation (1a); providing (120) for each image (2) in the set of images (3) a plurality of representations in the work space (4) as image aspect representations (2a1-2a5); determining (130) a set of coefficients (5) in a predetermined manner based at least in part on the query representation (1a); for each image (2) in the set of images (3), aggregating (140) the respective image aspect representations (2a1-2a5) into a respective image representation (2a) in a predetermined manner based on the coefficients (5); comparing (150) the query representation (1a) with the image representations (2a); and determining (160) one or more images (2) to be returned as a result (6) of the textual query (1) based at least in part on a result (150a) of the comparison (150).

Description

Technical Field

[0001] This invention relates to retrieving images from a large collection of pre-stored images using text queries. For example, this retrieval can be used to obtain suitable training examples for downstream machine learning models. Background Technology

[0002] For many applications, such as combining training datasets for machine learning models, it is necessary to start with a given text query and find suitable images from a large collection of pre-stored images. Representation-based retrieval methods map ("embed") both the text query and the pre-stored images into a common workspace, and then compare the representation of the query with the representation of the images to determine which image representation is closest to the query representation. The image corresponding to that representation is then returned as a result of the query. Examples of this method are well known by the names CLIP, BLIP, BLIP2, FILIP, SIGLIP, and LLIP.

[0003] Computing image representations is computationally expensive for each query processing, especially if the given image set contains millions of images. Therefore, it is advantageous to compute these representations only once and then use those stored representations repeatedly. However, some of the more advanced methods mentioned above, such as BLIP2 or LLIP, require representations of either the image or the text query. These representations cannot be computed offline in advance. Summary of the Invention

[0004] This invention provides a method for retrieving one or more images from an image collection based on a text query.

[0005] In this method, the representation of the text query in the workspace is computed as a query representation. For example, this query representation can be computed by a trained encoder. Additionally, for each image in the image set, multiple representations in the workspace are provided as image aspect representations. Specifically, these image aspect representations may have been pre-computed and can be retrieved from memory. Each image aspect representation can be considered to be associated with a specific aspect of the image, even if that aspect does not need to be explicitly expressed when creating the image representation. For example, if a trained encoder is used to create the image representation, it will learn to encode the n most important aspects of the image content into a set of n representations.

[0006] The workspace is typically chosen as a space in which the similarity or dissimilarity between query content and image content, and between different image contents, is itself represented as similarity or dissimilarity, or as proximity or distance, or as distinctness. For example, the workspace can be a vector space into which a trained encoder encodes both the query and the image. The workspace typically has lower dimensionality than the text query space and the image space. That is, representations in the workspace typically contain less information than either a text query or an image, i.e., they depend on fewer independent variables.

[0007] At least in part, a set of coefficients is computed in a predetermined manner based on the query representation. Based on these coefficients, for each image in a given set of images, the corresponding image aspect representations are aggregated into a corresponding image representation. That is, now each image has only one image representation.

[0008] The query representation is compared with the image representation. Based at least in part on the result of this comparison, one or more images are determined to be returned as the result of the text query.

[0009] The inventors have discovered that, in this way, an image representation compared to a query representation can be made dependent on the query representation, and therefore dependent on the text query, which is computationally much cheaper than merging the content of the text query with the image content and creating its representation in the workspace. In particular, the "forward pass" from the content space to the workspace through such a trained encoder (i.e., a machine learning model) is expensive if the representation is created by such an encoder. In contrast, coefficients can be determined much faster because this already occurs in a workspace that typically has a much lower dimension. Aggregating image aspect representations can be done, for example, using efficient matrix or vector operations, and is therefore inexpensive.

[0010] The end result is that the benefits of image representations that rely on text queries known from the literature, such as the potential for better fine-grained search, can be obtained without having to endure the fact that image representations can no longer be pre-computed offline and retrieved quickly from memory in real time.

[0011] Furthermore, the use of multiple image aspect representations allows for more precise evaluation of images containing several objects or subjects. A prime example of such images is traffic scene images, which typically contain multiple traffic participants of various types (cars, trucks, buses, pedestrians, cyclists), road markings, and different obstacles. This means that sentences like "A pedestrian is crossing the road on the left," "A bus is standing at a traffic light, waiting for the light to change," and "Many cars are parked on the side of the road next to a large building" could all be full descriptions of the same image. Therefore, if one of these sentences is used as a text query, that image should be returned in the result set. This means that the image representation in the workspace needs to approximate several query representations that may be very different from each other and therefore very far apart in the workspace. This is very difficult to achieve without making the image representation dependent on the text query. And as mentioned above, this method allows the image representation to be dependent on the text query in a particularly computationally efficient manner.

[0012] Different aspects of the image can be represented in any suitable way.

[0013] For example, vector representations of parts of an image (such as image tiles) can be combined in different ways to form different image aspect representations. For example, vector representations can be generated as in the known CLIP method, and they can be combined as a whole into an image representation (i.e., an image aspect representation in the context of this method) through cross-attention, as in the known BLIP method. In this method, this is a one-off effort because, as mentioned above, generating the final image representation from the image aspect representation is inexpensive.

[0014] In another example, representations obtained from an image by encoders trained differently and / or by encoders with different architectures can be used to create different image aspect representations. This allows different image aspect representations to be tuned toward aspects known to be important in some way. For example, when processing images of traffic scenes, one encoder can place more emphasis on traffic signs, a second encoder can place more emphasis on traffic participants, and a third encoder can place more emphasis on road markings.

[0015] In another example, representations obtained as different samples generated from an image by a probabilistic encoder can be used as image aspect representations. These image aspect representations are then aggregated using coefficients, providing a better incentive to use the aggregation from the probabilistic encoder in the first place: without such aggregation, it might be problematic to ask why use that particular sample drawn from the encoder instead of another.

[0016] In a particularly advantageous embodiment, coefficients are determined from the query representation using a trained machine learning model. In this way, combining image aspect representations into a single image representation by training a machine learning model can be particularly well-suited to the application at hand. This is especially advantageous if the creation of individual image aspect representations is not explicitly directed toward a particular aspect. Because the machine learning model maps from the workspace to a small number of coefficients, it can be small, making the forward pass of the query representation through that model inexpensive. In one example, the machine learning model may include a neural network with at least one fully connected layer. After training, the machine learning model can “select” which image aspect representations are relevant based on the text query and capture the concepts presented in the text query. In particular, using a machine learning model allows for significant differences between the final image representations created for the same image based on different text queries. Each such final representation can be an “expert” on different concepts or topics.

[0017] In a particularly advantageous embodiment, aggregation comprises calculating a weighted sum of image aspect representations using coefficients as weights. This is computationally very lightweight, as it involves only one additional matrix multiplication, an operation that is very efficient to implement.

[0018] In another particularly advantageous embodiment, a first vector is established, which is a concatenation of the image aspect representation. A second vector can then be established, which is a concatenation of as many copies of the query representation as there are image aspect representations, where each copy of the query representation is multiplied by the corresponding weight of the image representation. The dot product of the first and second vectors can then be computed as the desired weighted sum. This has a dual advantage: first, the dot product is implemented in a computationally very efficient manner; second, the first vector with the image aspect representation can be pre-stored in a vector database and retrieved based on the second vector using an efficient search algorithm. That is, the vector database can be reused within the context of this method for further acceleration and memory savings.

[0019] In another particularly advantageous embodiment, the comparison of the query representation with the image representation involves calculating the cosine similarity between the query representation and the image representation. This is a continuous measure of similarity and proximity in the workspace, which is particularly inexpensive to compute. Based on this, or any other suitable distance or similarity measure, image representations of the images to be returned as part of the result set can then be selected. For example, it is possible to select the top N (e.g., the top 1) image representations that are closest / most similar to the query representation. Optionally, a threshold for similarity / proximity may also exist. For example, if an image representation does not meet the threshold, it can be excluded from the result set.

[0020] In another particularly advantageous embodiment, the image set comprises frames from the video data stream. Based on the results of the text query, the positions in the video data stream where the situation described in the text query appears are calculated. In this way, interactive real-time analysis of the same video stream can be performed on different aspects. For example, during a test drive of a vehicle that is at least partially automated, many hours of video footage are recorded. Within this footage, only a few scenes may be important as interesting and novel content, such as further machine learning-based training for the automated driving system. Using this method, such content can be found much more easily. A manual review of the video footage might even completely miss it.

[0021] This also applies to another example of recording video data streams in a surveillance system. In such applications, typically very little interesting activity occurs within the monitored area. Using this method, such interesting events can be searched using text queries, such as "a person is trying to climb a fence," "a person is working on a fence," or "a person is scouting along the fence line."

[0022] In another particularly advantageous embodiment, for each image returned as a result of a text query, a text description associated with the corresponding image can be obtained. A description of the scene described in the text query can then be determined from the text description. In this way, pre-existing knowledge in the text description can be utilized. The scene description thus obtained may then be more realistic and / or more specific to the application at hand than a description generated "from scratch" using a generative machine learning model, such as a large language model (LLM).

[0023] In another particularly advantageous embodiment, a machine learning model can be trained using at least one image returned as a result of a text query. This training often suffers from "imbalance" in the training dataset because there are too few training examples for some categories or situations. For example, when assembling a training dataset for recognizing traffic signs, some traffic signs (such as those related to speed limits or priority) will be abundant, while warning signs, such as those for roads leading to waterways, will appear very rarely. Nevertheless, if it does appear, the warning sign is very important and must be correctly identified. In another example, cargo theft (such as luggage or even some furniture) is rare on highways, but it does sometimes occur, and if it does, a rapid reaction (braking or evasive maneuver) is required. Using this method, it is easy to search through many hours of video footage for rare traffic signs, lost cargo, or other things that appear very infrequently. In this way, then even for such rare cases, there is sufficient training data.

[0024] Specifically, once the machine learning model has been trained, at least one image captured by at least one sensor can be input into the trained machine learning model. An actuation signal can be calculated based on the output of the trained machine learning model. The vehicle, driver assistance system, robot, quality inspection system, monitoring system, and / or medical imaging system can then be actuated using the actuation signal. In this context, the use of this method improves the probability that the action performed by the corresponding actuated system in response to the actuation signal is appropriate given the context represented by the input image.

[0025] The present invention also provides a system for retrieving one or more images from an image collection based on a text query. Specifically, this system can be used to perform the methods described above. Therefore, it provides one way to implement the methods described above, but other ways are also possible.

[0026] The system includes a first trainable encoder configured to compute a representation of a text query in a workspace as a query representation. Additionally, a memory exists configured to store multiple representations in the workspace as image aspect representations for each image in the image set. In this way, the pre-computed image aspect representations are available on top of the newly generated query representation.

[0027] The system also includes a trainable coefficients calculation module configured to determine a set of coefficients based at least in part on a query representation generated by a first trainable encoder. As previously described, the trainable coefficients calculation module may include a neural network, such as a network having at least one fully connected layer.

[0028] The system also includes a representation calculation module. This representation calculation module is configured to aggregate the corresponding image aspect representations into a corresponding image representation for each image in the image set based on the coefficients.

[0029] The system also includes a comparison module configured to compare a query representation computed by a first trainable encoder with an image representation computed by a representation computation module.

[0030] The system also includes a result calculation module, which is configured to determine one or more images to be returned as the result of a text query based at least in part on the result of the comparison generated by the comparison module.

[0031] As previously mentioned, the system proposed in this paper allows for real-time image retrieval even from large collections of millions of images because, upon receiving a text query, the text query only needs to be forwarded once by a first trainable encoder. Generating image representations based on image aspect representations is significantly cheaper than generating new representations dependent on both image content and text query through a forward pass via an encoder.

[0032] In a particularly advantageous embodiment, the system further includes a second trainable encoder. This second trainable encoder is configured to compute multiple representations in the workspace as image aspect representations for the input image. In this way, the image aspect representations can be obtained in a one-time effort and then stored in the system's memory.

[0033] The present invention also provides a method for training the above-described system.

[0034] Throughout this method's development, a set of training examples is provided. Each such training example consists of a text query. The text query is labeled with the image to be retrieved as the ground truth. This is the image that the system should ideally return given the text query.

[0035] For each training example, the system processes the corresponding text query into a result that includes one or more returned images. Then, a predetermined loss function is used to evaluate the degree of correspondence between this result and the retrieved image labeled with that training example. The first encoder parameters, characterizing the behavior of the first trainable encoder, and the coefficient parameters, characterizing the behavior of the coefficient calculation module, are then optimized towards improving the rating of the loss function when processing further training examples. In this paper, it is possible to simultaneously optimize both the first encoder parameters and the coefficient parameters, optimizing one first and then the other, or changing the optimization of the first encoder parameters and the coefficient parameters multiple times.

[0036] In this way, for each text query and each image in the image set, the coefficient calculation module learns which aspects of each image need to be considered relevant to that specific text query. Simultaneously, the first trainable encoder learns how to extract the salient features needed to be found in a given image from the text query.

[0037] In another particularly advantageous embodiment, training images are provided to a second trainable encoder. In particular, these training images may include one or more images to be retrieved, which are used to label training examples.

[0038] Then, the second encoder parameters, characterizing the behavior of the second trainable encoder, are optimized with the goal of improving the rating of the loss function when processing further training examples and / or training images. This optimization can be performed in parallel, sequentially, or in any suitable mixture with the optimization of the first encoder parameters and the coefficient parameters. In this way, the second trainable encoder can learn to encode aspects of the corresponding images in the image aspect representation that are salient to queries in the domain given by the training examples.

[0039] In another particularly advantageous embodiment, the coefficients used to activate the loss function for a single training example are different from each other. In this way, different image aspect representations are made specifically for different concepts. Alternatively, or in combination, taking into account the diversity of coefficients in the loss function, the temperature parameter for calculating the softmax score of the coefficients in the coefficient calculation module can be reduced so as to force the coefficient calculation module to output a sharper distribution in the weights.

[0040] In another particularly advantageous embodiment, the loss function incentivizes a pool of coefficients collected over a given batch of training examples, where the coefficients exhibit a uniform statistical distribution. In this way, the incentivizes each aspect of the image at a certain point in the batch of training examples. That is, there are no unimportant "isolated" aspects within the entire domain of the text query given by the batch. Instead, the image aspect representation can be tailored to this domain of the text query.

[0041] The method can be implemented wholly or partially by a computer and embodied in software. Therefore, the invention also relates to computer programs having machine-readable instructions that, when executed by one or more computers and / or computing instances, cause one or more computers and / or computing instances to perform the methods described above. In this document, control units for vehicles or robots, as well as other embedded systems capable of executing machine-readable instructions, are also considered computers. Computing instances include virtual machines, containers, or other execution environments that permit the execution of machine-readable instructions in the cloud.

[0042] Non-transitory storage media and / or downloadable products may include computer programs. A downloadable product is an electronic product that can be sold online and transferred via a network for immediate fulfillment. One or more computers and / or computing instances may be equipped with the computer programs and / or the non-transitory storage media and / or downloadable products. Attached Figure Description

[0043] In the following text, figures are used to illustrate the invention, but are not intended to limit the scope of the invention in any way. The figures show: Figure 1 An exemplary embodiment of a method 100 for retrieving one or more images 2 from an image set 3 based on a text query 1; Figure 2 The image representation aggregates the dependency queries of 2a1-2a5 into a single image representation of the diagram of 2a; Figure 3 An exemplary embodiment of a system 20 for retrieving one or more images 2 from an image set 3 based on a text query 1; Figure 4 Used for training Figure 3 An exemplary embodiment of the method 300 of the system 20 shown. Detailed Implementation

[0044] Figure 1 This is an illustrative flowchart of an embodiment of a method 100 for retrieving one or more images 2 from an image set 3 based on a text query 1.

[0045] According to box 105, image set 3 may include frames of video data stream.

[0046] In step 110, the representation of text query 1 in workspace 4 is calculated as query representation 1a.

[0047] In step 120, for each image 2 in the image set 3, multiple representations in the workspace 4 are provided as image aspect representations 2a1-2a5.

[0048] According to box 121, these different image aspect representations 2a1-2a5 may include Different combinations of vector representations of parts of Image 2, and / or Representations 2a1-2a5 obtained from image 2 by encoders trained with different methods and / or by encoders with different architectures, and / or The representations 2a1-2a5 are obtained as different samples generated from image 2 by a probabilistic encoder.

[0049] In step 130, a set of coefficients 5 are determined in a predetermined manner, at least in part based on query representation 1a.

[0050] According to box 131, coefficient 5 can be determined by a trained machine learning model based on query representation 1a. Specifically, according to box 131a, the trained machine learning model can include a neural network with at least one fully connected layer.

[0051] In step 140, for each image 2 in the image set 3, based on coefficient 5, the corresponding image aspect representations 2a1-2a5 are aggregated into the corresponding image representation 2a in a predetermined manner.

[0052] According to box 141, aggregation 140 may include calculating a weighted sum of image aspect representations 2a1-2a5 with a coefficient of 5 as the weight.

[0053] Specifically, according to box 141a, a first vector can be constructed, which is a concatenation of image aspect representations 2a1-2a5. According to box 141b, a second vector can then be constructed, which is a concatenation of as many copies of query representation 1a as there are image aspect representations 2a1-2a5. In this paper, each copy of query representation 1a is multiplied by the corresponding weight of image representation 2a. According to box 141c, the dot product of the first and second vectors can then be computed as the sought weighted sum.

[0054] In step 150, the query represents image representation 1a to image representation 2a. The output is result 150a.

[0055] According to box 151, the comparison 150 between query representation 1a and image representation 2a may include calculating the cosine similarity between query representation 1a and image representation 2a.

[0056] In step 160, one or more images 2 to be returned as the result 6 of text query 1 are determined at least in part based on the result 150a of comparison 150.

[0057] exist Figure 1 In the example shown, if image set 3 includes frames from the video data stream according to box 105, then in step 170, the position 7 in the video data stream where the situation described in text query 1 occurs is calculated based on the result 6 of text query 1.

[0058] exist Figure 1 In the example shown, for each image 2 returned as a result 6 of text query 1, a text description 2b associated with the corresponding image 2 is obtained in step 180. In step 190, a description 8 of the scene described in text query 1 is then determined from the text description 2b.

[0059] exist Figure 1 In the example shown, in step 200, at least one image 2 returned as the result 6 of text query 1 is used to train machine learning model 9. The result of this training is the training state 9 of machine learning model 9. .

[0060] In step 210, at least one image 10 captured by at least one sensor 11 can then be input into a trained machine learning model 9. Then, in step 220, based on the trained machine learning model 9 Output 12 can be used to calculate actuation signal 13. In step 230, vehicle 50, driver assistance system 51, robot 60, quality inspection system 70, monitoring system 80 and / or medical imaging system 90 can then be actuated using actuation signal 13.

[0061] Figure 2 The diagram illustrates how the different image aspect representations 2a1-2a5 of image 2 in image set 3 are aggregated in a query-dependent manner. Image aspect representations 2a1-2a5 exist in the same workspace 4 as representation 1a of text query 1. Based on this representation 1a of text query 1, coefficient 5(1a) is determined. Figure 1In step 140 of method 100, image aspect representations 2a1-2a5 are aggregated into a single image representation 2a of image 2 based on these coefficients 5 (1a). Workspace 4 contains many such image representations 2a. In step 150 of method 100, query representation 1a is compared with all these image representations 2a.

[0062] Figure 3 This is a schematic block diagram of an exemplary embodiment of a system 20 for retrieving one or more images 2 from an image set 3 based on a text query 1.

[0063] System 20 includes a first trainable encoder 21 configured to compute a representation of text query 1 in workspace 4 as query representation 1a. The memory 22 of system 20 is configured to store multiple representations in workspace 4 as image aspect representations 2a1-2a5 for each image 2 in the image set 3. System 20 also includes a trainable coefficient computation module 23 configured to determine a set of coefficients 5 based at least in part on query representation 1a.

[0064] The representation calculation module 24 of system 20 is configured to aggregate the corresponding image aspect representations 2a1-2a5 into a corresponding image representation 2a for each image 2 in the image set 3 based on the coefficients 5 received from the coefficient calculation module 23.

[0065] System 20 also includes a comparison module 25 configured to compare query representation 1a with image representation 2a. The result calculation module 26 of system 20 can then determine, at least in part, one or more images 2 to be returned as the result 6 of text query 1 based on the result 150a of the comparison in comparison module 25.

[0066] exist Figure 3 In the example shown, system 20 also includes a second trainable encoder 27, which is configured to compute multiple representations in workspace 4 as image aspect representations 2a1-2a5 for input image 2.

[0067] Figure 4 It is used for training Figure 3 The illustrated flowchart is a schematic flowchart of an embodiment of the method 300 of the system 20.

[0068] In step 310, a set of training examples 14 is provided. Each training example 14 includes a text query 1, which is labeled with an image 2 to be retrieved. As a benchmark of reality.

[0069] In step 320, for each training example 14, the system 20 processes the text query 1 into a result 6 that includes one or more returned images 2.

[0070] In step 330, the predetermined loss function 15 is applied to the result 6 thus obtained and the image to be retrieved labeled with the corresponding training example 14. The degree of correspondence was rated. The result was a rating of 15a.

[0071] Specifically, according to box 331, loss function 15 can activate coefficients 5 used for a single training example 14 that are distinct from each other. Alternatively or in combination with this, according to box 332, loss function 15 can activate a pool of coefficients 5 collected on a given batch of training examples 14, where the coefficients 5 exhibit a uniform statistical distribution.

[0072] In step 340, the first encoder parameters 21a, characterizing the behavior of the first trainable encoder 21, and the coefficient parameters 23a, characterizing the behavior of the coefficient calculation module 23, are optimized towards the goal of improving the rating 15a of the loss function 15 when processing further training examples 14. This result includes the training state 21, characterizing the first trainable encoder 21. Optimization parameters 21a and the training state 23 of the characterization coefficient calculation module 23 Optimization parameters 23a .

[0073] exist Figure 4 In the example shown, in step 350, training image 16 is presented to second trainable encoder 27, which then outputs image aspect representations 2a1-2a5 to the processing in step 320. According to box 341, the second encoder parameters 27a, characterizing the behavior of second trainable encoder 27, can then also be optimized towards the goal of improving the rating 15a of loss function 15 when processing further training examples 14 and / or training images 16. The result includes a training state 27 characterizing second trainable encoder 27. Optimization parameters 27a .

Claims

1. A method (100) for retrieving one or more images (2) from an image set (3) based on a text query (1), comprising the following steps: Calculate (110) the representation of the text query (1) in the workspace (4) as the query representation (1a); Provide (120) multiple representations in the workspace (4) as image aspect representations (2a1-2a5) for each image (2) in the image set (3); At least in part based on the query representation (1a), a set of coefficients (5) is determined (130) in a predetermined manner; For each image (2) in the image set (3), based on these coefficients (5), the corresponding image aspect representations (2a1-2a5) are aggregated (140) into the corresponding image representation (2a) in a predetermined manner. The query representation (1a) is compared with the image representation (2a) (150); and Based at least in part on the result (150a) of the comparison (150), determine (160) one or more images (2) to be returned as the result (6) of the text query (1).

2. The method (100) according to claim 1, wherein the different image aspect representations (2a1-2a5) include (121). Different combinations of vector representations of portions of the image (2), and / or The representations (2a1-2a5) obtained from the image (2) by encoders trained with different methods and / or by encoders with different architectures, and / or As representations (2a1-2a5) obtained from different samples generated by the probabilistic encoder from the image (2).

3. The method (100) according to any one of claims 1 to 2, wherein the coefficient (5) is determined (131) by a trained machine learning model based on the query representation (1a).

4. The method (100) of claim 3, wherein the trained machine learning model comprises (131a) a neural network having at least one fully connected layer.

5. The method (100) according to any one of claims 1 to 4, wherein the aggregation (140) comprises (141) calculating a weighted sum of the image aspect representations (2a1-2a5) using the coefficients (5) as weights.

6. The method (100) according to claim 5, wherein the calculation of the weighted sum comprises: Establish a first vector (141a), which is a connection of the image aspect representation (2a1-2a5); Establish a second vector (141b), which is a concatenation of as many copies of the query representation (1a) as the image aspect representations (2a1-2a5), wherein each copy of the query representation (1a) is multiplied by the corresponding weight of the image representation (2a); and Calculate the dot product of the first and second vectors (141c) as the weighted sum sought.

7. The method (100) according to any one of claims 1 to 6, wherein the comparison (150) between the query representation (1a) and the image representation (2a) comprises calculating (151) a cosine similarity between the query representation (1a) and the image representation (2a).

8. The method (100) according to any one of claims 1 to 7, wherein The image set (3) includes (105) frames of the video data stream, and Based on the result (6) of the text query (1), calculate (170) the position (7) where the situation described in the text query (1) appears in the video data stream.

9. The method (100) according to any one of claims 1 to 8, further comprising: For each image (2) returned as the result (6) of the text query (1), obtain (180) the text description (2b) stored in association with the corresponding image (2); and Determine (190) the description (8) of the scenario described in the text query (1) from the text description (2b).

10. The method (100) according to any one of claims 1 to 9, further comprising: The machine learning model (9) is trained (200) using at least one image (2) returned as the result (6) of the text query (1).

11. The method (100) according to claim 10, further comprising: At least one image (10) captured by at least one sensor (11) is input (210) into a trained machine learning model (9). ); According to the trained machine learning model (9) The output (12) of the actuation signal (13) is calculated (220); and The actuation signal (13) actuates (230) the vehicle (50), the driver assistance system (51), the robot (60), the quality inspection system (70), the monitoring system (80), and / or the medical imaging system (90).

12. A system (20) for retrieving one or more images (2) from an image set (3) based on a text query (1), the system (20) comprising: A first trainable encoder (21) is configured to compute the representation of the text query (1) in the workspace (4) as a query representation (1a); The memory (22) is configured to store multiple representations in the workspace (4) as image aspect representations (2a1-2a5) for each image (2) in the image set (3); A trainable coefficients calculation module (23) is configured to determine a set of coefficients (5) based at least in part on the query representation (1a); The representation calculation module (24) is configured to: based on the coefficients (5), for each image (2) in the image set (3), aggregate the corresponding image aspect representations (2a1-2a5) into a corresponding image representation (2a); A comparison module (25) is configured to compare the query representation (1a) with the image representation (2a); as well as The result calculation module (26) is configured to determine one or more images (2) to be returned as the result (6) of the text query (1) based at least in part on the result (150a) of the comparison.

13. The system (100) according to claim 12, further comprising: A second trainable encoder (27) is configured to compute multiple representations in the workspace (4) as image aspect representations (2a1-2a5) for the input image (2).

14. A method (300) for training a system (20) according to any one of claims 12 to 13, comprising the following steps: Provided (310) a set of training examples (14), each training example (14) including a text query (1) which is labeled with an image to be retrieved (2). (This serves as the baseline reality;) For each training example (14), the text query (1) is processed (320) by the system (20) according to any one of claims 12 to 13 into a result (6) including one or more returned images (2); The result (6) is evaluated (330) by a predetermined loss function (15) and compared with the image to be retrieved using the corresponding training example (14) labeled therewith. The degree of correspondence; and The first encoder parameters (21a) characterizing the behavior of the first trainable encoder (21) and the coefficient parameters (23a) characterizing the behavior of the coefficient calculation module (23) are optimized (340) towards the goal of improving the rating (15a) of the loss function (15) when processing further training examples (14).

15. The method (300) of claim 14, further comprising: Provide (350) images (2) from a set of training images (16) to the second trainable encoder (27); as well as The second encoder parameters (27a) characterizing the behavior of the second trainable encoder (27) are further optimized (341) towards the goal of improving the rating (15a) of the loss function (15) when processing further training examples (14) and / or training images (16).

16. The method (300) according to any one of claims 14 to 15, wherein the coefficients (5) of the loss function (15) activation (331) for a single training example (14) are different from each other.

17. The method (300) according to any one of claims 14 to 16, wherein the loss function (15) activates (332) a pool of coefficients (5) collected on a given batch of training examples (14), the coefficients (5) exhibiting a uniform statistical distribution.

18. A computer program comprising machine-readable instructions that, when executed by one or more computers and / or computing instances, cause the one or more computers and / or computing instances to perform the method (100, 300) according to any one of claims 1 to 11 or 14 to 17.

19. A machine-readable data carrier and / or downloadable product having the computer program according to claim 18.

20. A non-transitory machine-readable data carrier having a computer program as claimed in claim 18 and / or having a machine-readable data carrier and / or downloadable product as claimed in claim 19.

Images