Device and computer-implemented method for training first encoder for mapping radar spectrum to encoding, in particular encoding for training, testing, validating, or verifying first model that is configured for object detection, for event recognition, or for segmentation

Abstract

To provide a device and a computer-implemented method for training an encoder for mapping radar spectra to encodings, for training, testing, validating, or verifying a model that is configured for object detection, for event recognition, or for segmentation.SOLUTION: A method comprises: providing a first encoder 202 configured to map a radar spectrum to an encoding of a radar spectrum 204 in a feature space, and a second encoder 210 configured to map a digital image to an encoding of the digital image; providing a radar spectrum and a digital image 208, the radar spectrum comprising a radar reflection of at least a part of an object, the digital image depicting at least a part of the object; mapping the radar spectrum with the first encoder to a first encoding 206; mapping the digital image with the second encoder to a second encoding 212; and training the first and second encoders depending on a distance between the first encoding and the second encoding.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] background The present invention relates to an apparatus and computer-implemented method for encoding, in particular for training a first encoder for mapping a radar spectrum to an encoding for training, validating, verifying or validating a first model configured for object detection, event recognition or segmentation. [Background technology]

[0002] US Patent Application Publication No. 2020 / 174112 A1 and US Patent Application Publication No. 2020 / 082219 A1 disclose sensor fusion. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2020 / 174112A1 [Patent Document 2] US Patent Application Publication No. 2020 / 082219A1 Summary of the Invention [Means for solving the problem]

[0004] An apparatus and computer-implemented method according to the independent claims inserts information from digital images into radar spectrum encoding for object detection, event recognition or segmentation.

[0005] This means that information from the first digital image is inserted into the first model for object detection, event recognition or segmentation. The third encoder is used for fine-tuning. The second encoder may be part of a pre-trained image language model. The second encoder may be fine-tuned together with the training of the first encoder in training. The class-named statements are automatically created from the content of the first digital image or the second digital image. The third encoder may be part of a pre-trained image language model. The third encoder may be fine-tuned together with the training of the first encoder in training.

[0006] Providing a caption for the first digital image may include providing a set of class names and providing a set of categories, where at least one class name of the set of class names is associated with at least one category of the set of categories, and providing a template includes providing a template having a first placeholder for a category of the set of categories. Providing a caption for the first digital image may include associating at least one portion of the first digital image, in particular at least one pixel or at least one segment of pixels, with a class name of the set of class names, replacing the placeholder for the category with a first class name associated with the category and associated with the at least one portion of the first digital image, in particular at least one pixel or at least one segment of pixels, to create a first statement of the caption for the first digital image, and replacing the placeholder for the category with a second class name associated with the category and associated with the at least one portion of the first digital image, in particular at least one pixel or at least one segment of pixels, to create a second statement of the caption for the first digital image. Providing a caption for the second digital image may include associating at least one portion of the second digital image, in particular at least one pixel or at least one segment of pixels, with a class name from the set of class names; replacing a placeholder for a category with a first class name associated with the category and associated with at least one portion of the second digital image, in particular at least one pixel or at least one segment of pixels, to create a first statement of the caption for the second digital image; and replacing the placeholder for the category with a second class name associated with the category and associated with at least one portion of the second digital image, in particular at least one pixel or at least one segment of pixels, to create a second statement of the caption for the second digital image.This means that multiple statements containing different class names of the same category can be automatically generated from the same template.

[0007] Providing the first digital image includes providing depth information associated with pixels of the first digital image, and providing a caption for the first digital image includes providing a set of attributes describing a position of the object relative to another object depicted in the first digital image, determining a position of the first object relative to another object depicted in the first digital image in response to the depth information associated with at least a portion of the pixels depicting the first object and the depth information associated with at least a portion of the pixels depicting the other object, selecting an attribute from the set of attributes in response to the position, providing a template having two first placeholders and one second placeholder for attributes of the set of attributes, replacing the second placeholder with the attribute and replacing the two first placeholders with class names of the first object and the other object, respectively. The depth information further improves the caption for the first digital image.

[0008] Providing the second digital image includes providing depth information associated with pixels of the second digital image, and providing a caption for the second digital image includes providing a set of attributes describing a position of the object relative to another object depicted in the second digital image, determining a position of the second object relative to another object depicted in the second digital image in response to the depth information associated with at least a portion of the pixels depicting the second object and the depth information associated with at least a portion of the pixels depicting the other object, selecting an attribute from the set of attributes in response to the position, providing a template having two first placeholders and one second placeholder for attributes of the set of attributes, replacing the second placeholder with the attribute and replacing the two first placeholders with class names of the first object and the other object, respectively. The depth information further improves the caption for the second digital image.

[0009] Providing the caption may comprise providing a set of templates for the statement, where the templates in the set of templates comprise a first placeholder and a part of the respective statement, and selecting a template for the caption from the set of templates in a particularly random manner, meaning that the set of statements comprise statements that are different from each other and the statements used are randomized.

[0010] The method may include mapping a first radar spectrum to a first encoding in a second feature space by a first portion of a first model configured to map the radar spectrum to an encoding in the second feature space; mapping the first encoding in the first feature space to a second encoding in the second feature space by a second model configured to map the encoding in the first feature space to an encoding in the second feature space; mapping the first encoding in the second feature space and the second encoding in the second feature space by the second portion of the first model to an output of the first model; providing a ground truth for the output; and training the first model as a function of a difference between the output and the ground truth, wherein the output and the ground truth characterize at least one object detected in the radar spectrum, or an event recognized in the radar spectrum, or a segmentation of the radar spectrum. It uses stored information from the digital images used to train the first encoder to train a first model based on the first digital image or specifically a previously unseen third digital image.

[0011] The method may include mapping a second digital image to a first encoding in a second feature space by a first portion of a first model configured to map the digital image to an encoding in a second feature space, mapping the first encoding to a second encoding in the second feature space by a second model configured to map an encoding from the first feature space to an encoding in the second feature space, mapping the first encoding in the second feature space and the second encoding in the second feature space by a second portion of the first model to an output of the first model, providing a ground truth for the output, and training the first model as a function of a difference between the output and the ground truth, the output and the ground truth characterizing at least one object detected in the radar spectrum, or an event recognized in the radar spectrum, or a segmentation of the radar spectrum. This trains the first model based on the second digital image using stored information from the digital image used to train the first encoder.

[0012] The method includes providing a third radar spectrum, mapping the third radar spectrum to an encoding of the third radar spectrum in a first feature space by a first encoder, mapping the third radar spectrum to a first encoding in a second feature space by a first portion of a first model configured to map the radar spectrum to an encoding in a second feature space, and encoding the encoding of the third radar spectrum in the first feature space by a second model 410 configured to map the encoding in the first feature space to an encoding in the second feature space. the first encoding in the second feature space by the second portion of the first model and the second encoding in the second feature space to an output of the first model, providing a ground truth for the output, and training the first model as a function of a difference between the output and the ground truth, where the output and the ground truth characterize at least one object detected in the radar spectrum, or an event recognized in the radar spectrum, or a segmentation of the radar spectrum. The first model may be trained against a previously unseen radar spectrum.

[0013] The method includes capturing a first radar spectrum with a radar sensor, mapping a first encoding in the first feature space to a first encoding in the second feature space by a second model, mapping the first radar spectrum to a second encoding of the first radar spectrum in the second feature space by a part of the first model, and mapping the first encoding of the first radar spectrum in the second feature space and the second encoding of the first radar spectrum in the second feature space by a second part of the first model to an output of the first model characterizing at least one object detected in the first radar spectrum, in particular a traffic sign, a road surface, a person, a pedestrian, an animal, a plant, a vehicle, a road object, a building, or characterizing an event recognized in the first radar spectrum, in particular a traffic sign state, a human gesture, or characterizing a spectral segmentation with respect to, in particular, a traffic sign, a road surface, a pedestrian, an animal, a vehicle, a road object, a building. This means that the first radar spectrum is analyzed based on stored information from the digital images on which the first encoder was trained.

[0014] The method may comprise operating, in particular moving or stopping, a technical system, in particular a computer controlled device, preferably a robot, a vehicle, a manufacturing device, a household appliance, a power tool, an access control system, a personal assistant, a medical imaging system, which means that the technical system operates based on stored information from the digital images on which the first encoder was trained.

[0015] An apparatus for encoding, in particular for training a first encoder for mapping a radar spectrum to an encoding for training, verifying, certifying or validating a first model configured for object detection, event recognition or segmentation, comprises at least one processor and at least one memory, where the at least one processor is configured to execute instructions that, when executed by the at least one processor, cause the apparatus to perform steps of the method, and where the at least one memory is configured to store the instructions.

[0016] The apparatus comprises a radar sensor configured to capture a radar spectrum, the apparatus is configured to determine an output of a first model characterizing at least one object detected in the spectrum, in particular a traffic sign, a road surface, a person, a pedestrian, an animal, a plant, a vehicle, a road object, a building, or characterizing an event recognized in the spectrum, in particular a state of a traffic sign, a human gesture, or characterizing a segmentation of the spectrum, in particular with respect to a traffic sign, a road surface, a pedestrian, an animal, a vehicle, a road object, a building, and to operate, in particular move or stop, a technical system, in particular a computer controlled device, preferably a robot, a vehicle, a manufacturing device, a household appliance, a power tool, an access control system, a personal assistant, a medical imaging system, depending on the output.

[0017] The computer program comprises computer readable instructions which, when executed by a computer, cause the computer to perform a method.

[0018] Further embodiments can be derived from the following description and drawings. [Brief description of the drawings]

[0019] [Figure 1] FIG. 13 is a diagram that generally depicts an apparatus for encoding, in particular for training a first encoder for mapping a radar spectrum to an encoding for training, testing, verifying or validating a first model configured for object detection, event recognition or segmentation. [Diagram 2] FIG. 2 illustrates a schematic depiction of a first part of the training of a first encoder. [Diagram 3] FIG. 10 illustrates a schematic depiction of a second part of the training of the first encoder. [Figure 4] FIG. 2 is a schematic depiction of the training of a first model. [Diagram 5] FIG. 1 depicts a flowchart of a method for training a first encoder for mapping a radar spectrum to an encoding. [Figure 6] FIG. 1 depicts a flowchart of a method for training, testing, validating or verifying a first model. [Figure 7] FIG. 1 depicts a flowchart of a method for object detection, event recognition or segmentation using a first model. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] FIG. 1 illustrates a schematic representation of an apparatus 100 .

[0021] The apparatus 100 comprises at least one processor 102 and at least one memory 104 .

[0022] The at least one processor 102 is configured to execute instructions that, when executed by the at least one processor 102, cause the apparatus 100 to perform steps of a method for training an encoder for mapping a radar spectrum to a coding, in particular a coding for training, verifying, validating or verifying a model configured for object detection, event recognition or segmentation. The at least one memory 104 is configured to store instructions.

[0023] The apparatus 100 is configured to determine an output of the model.

[0024] The device 100 is configured to operate, in particular move or stop, the technical system 106 depending on the output of the model.

[0025] The technical system 106 is, for example, a computer controlled device, preferably a robot, a vehicle, a manufacturing device, a household appliance, a power tool, an access control system, a personal assistant, or a medical imaging system.

[0026] The apparatus 100 may include a radar sensor 108 configured to capture a radar spectrum.

[0027] The output can characterize at least one object detected in the spectrum, which can be a traffic sign, a road surface, a person, a pedestrian, an animal, a plant, a vehicle, a road object, or a building.

[0028] The output can characterize the events recognized in the spectrum, which can be the state of a traffic sign or a human gesture.

[0029] The output can characterize a spectral segmentation, which can be for traffic signs, road surfaces, pedestrians, animals, vehicles, road objects, and buildings.

[0030] 2 illustrates generally a first part of training a first encoder 202 to map a radar spectrum 204 to an encoding 206 in a first feature space. The first part of training includes mapping a digital image 208 by a second encoder 210 to a second encoding 212 in the first feature space.

[0031] The first part of the training is based on the first loss 214.

[0032] Denoting the radar spectrum as S and the digital image 208 as I, the first part of the training is i The radar spectra 204 and I i The training data includes pairs of digital images, denoted as {(I i ,S i )}.

[0033] Denoting the first encoder 202 as h and the second encoder 210 as f, the first loss 214 is, for example, a loss function L that depends on the distance between the first encoding 206 and the second encoding 212 to which the pair of radar spectrum 204 and digital image 208 are mapped. An example of the loss function L is:

number

number

[0034] 3 illustrates a schematic representation of a second part of the training of the first encoder 202. The second part of the training includes mapping the radar spectrum 204 to an encoding 206 in the first feature space. The second part of the training includes mapping a caption 302 of a digital image 208 to a third encoding 306 in the first feature space by a third encoder 304.

[0035] The first part of the training is based on the second loss 308.

[0036] The second loss 308 depends on the distance between the first encoding 206 and the third encoding 306. The second loss 308 is, for example, the cosine similarity of the first encoding 206 and the third encoding 306.

[0037] 4 illustrates in schematic form the training of a first model 402 configured for object detection, event recognition or segmentation. The first model 402 comprises a first portion 404 and a second portion 406.

[0038] A first portion 404 of the first model 402 is configured to map the first radar spectrum 204 to a first encoding 408 in a second feature space.

[0039] Training the first model 402 involves mapping the first radar spectrum 204 to a first encoding 206 in a first feature space.

[0040] The first feature space and the second feature space may be of comparable size. The first feature space and the second feature space may have different sizes.

[0041] Optionally, the second model 410 is configured to map the first encoding 206 from the first feature space to a second encoding 412 in the second feature space, particularly when the first feature space and the second feature space have different dimensions.

[0042] Training the first model 402 includes mapping the first encoding in the second feature space 408 and the second encoding in the second feature space 412 to an output 414 of the first model 402 by a second portion 406 of the first model 402.

[0043] A second part of the training includes providing a ground truth for the output and training the first model 402 according to the difference between the output 414 and the ground truth.

[0044] The output 414 and the ground truth characterize at least one object detected in the first radar spectrum 204, or an event recognized in the first radar spectrum 204, or a segmentation of the first radar spectrum 204.

[0045] 5 depicts a flow chart including method steps for training the first encoder 202 for mapping a radar spectrum to a coding, which may be used to train, test, validate, or verify the first model 402.

[0046] The training, testing, certification or validation is described as an example of a first radar spectrum 204, a first digital image 208 and a caption 302 of the first digital image 208. The first part of the training and the second part of the training may use different pairs of radar spectra and digital images. Some of the pairs may be used in both parts of the training and others may not be used. The training of the first model 402 may use similar or different radar spectrum and digital image pairs as the first part and / or the second part of the training. The pairs of radar spectra and digital images used in the training represent the same or essentially the same real-world scene and are preferably captured at the same time or essentially the same time.

[0047] According to one example, the training batches are selected from a driving dataset, which includes a series of frames of radar spectra and digital images. According to one example, the training batches are selected such that no two frames that are less than T seconds apart are selected for the same batch, where T is a parameter.

[0048] The method includes step 502.

[0049] Step 502 includes providing a first encoder 202 .

[0050] The method includes step 504.

[0051] Step 504 includes providing a second encoder 210 .

[0052] The second encoder 210 is, for example, pre-trained.

[0053] The method includes step 506 .

[0054] Step 506 includes providing a first radar spectrum 204 and a first digital image 208 .

[0055] The first radar spectrum 204 includes radar returns of at least a portion of a first object.

[0056] The first digital image 208 depicts at least a portion of a first object.

[0057] The first radar spectrum 204 and the first digital image 208 preferably represent the same or essentially the same real-world scene at the same time or essentially the same time.

[0058] The first digital image 208 may be provided with depth information associated with pixels of the first digital image 208. The depth information may be provided by a LiDAR sensor.

[0059] The method includes step 508.

[0060] Step 508 includes mapping the first radar spectrum 204 to a first encoding 206 by the first encoder 202 .

[0061] The method includes step 510.

[0062] Step 510 involves mapping the first digital image 208 to a second encoding 212 by the second encoder 210 .

[0063] The method includes step 512.

[0064] Step 512 includes training the first encoder 202 as a function of the distance between the first encoding 206 and the second encoding 212. Step 512 may include training the second encoder 210 as a function of the distance between the first encoding 206 and the second encoding 212. The second encoder 210 is, for example, fine-tuned as a function of the distance between the first encoding 206 and the second encoding 212.

[0065] Training may include repeating steps 502-512 for multiple pairs of the first radar spectrum and the first digital image. Training may include evaluating a first loss 214, such as one of the example loss functions.

[0066] The method may end after step 512 with the first encoder 202 pre-trained.

[0067] The method may include step 514 .

[0068] The method may begin at step 514 and with the first encoder 202 pre-trained.

[0069] Step 514 includes providing a third encoder 304 .

[0070] The third encoder 304 is configured to classify the set of class names according to, for example, a template for the caption 302. The template for the caption 302 includes, for example, a portion of a statement of the caption 302 and at least one placeholder. According to one example, the statement includes a placeholder for the class name. According to one example, the third encoder 304 is configured to output a classification of the class names in the set of class names with respect to the likelihood that the placeholder should be replaced with the respective class name, and the third encoding 306 represents the classification.

[0071] The third encoder 304 is, for example, pre-trained.

[0072] The method includes step 516 .

[0073] Step 516 includes providing a caption 302 for the first digital image 208 .

[0074] The caption 302 includes at least one statement that describes at least one object depicted in the first digital image 208. The statement in an example includes text.

[0075] The statement may be determined according to a template for the caption 302 .

[0076] The statement is determined, for example, by replacing the placeholder in the template with the class name to create the statement.

[0077] The class names may be selected from the set of class names depending on a semantic segmentation of the first digital image 208 that associates parts of the first digital image 208 with class names. The parts of the first digital image 208 may be pixels or segments of pixels. According to one example, the second encoder 210 is configured to output a classification of the class names in the set of class names with respect to the likelihood that a placeholder should be replaced with the respective class name, and the second encoding 212 represents the classification.

[0078] For example, the class name having the greatest joint likelihood is selected from the set of class names in the multimodal embedded space for the classification represented by the second encoding 212 and the classification represented by the third encoding 306 .

[0079] The second encoder 210 and the third encoder 304 may be trained together to predict the correct pairing of a batch of (digital image, caption) training examples.

[0080] A class name in the set of class names may be associated with each category in the set of categories.

[0081] For example, a set of categories might be vehicle, Infrastructure, Vegetation, Pedestrians, background Includes.

[0082] For example, the set of class names is: roads, sidewalks, buildings, walls, fences, poles, traffic lights, traffic signs, terrain, sky, people, riders, cars, trucks, buses, trains, motorcycles, bicycles, trees, grass Includes.

[0083] An exemplary association of class names to categories is: Vehicles: cars, trucks, buses, trains, motorcycles, bicycles Infrastructure: roads, sidewalks, buildings, walls, fences, poles Vegetation: trees, grass Animals: Dogs, cats, horses, birds Pedestrians: People, riders Background: sky, terrain It is.

[0084] A placeholder may be a placeholder for a category from a set of categories, which means that the placeholder may be replaced with any class name from the set of class names associated with the category.

[0085] According to one example, the semantic segmentation associates a first portion of the digital image 208 with a first class name from the set of class names and associates a second portion of the digital image 208 with a second class name from the set of class names. According to one example, the first class name and the second class name are associated with categories.

[0086] The placeholder for the category may be replaced with a first class name and a second class name. The caption 302 may include two statements, one created by replacing the placeholder for the category with the first class name and one created by replacing the placeholder for the category with the second class name.

[0087] The caption 302 may include at least one attribute, which may be selected from a set of attributes that describe the position of the object relative to other objects depicted in the first digital image 208.

[0088] For example, the digital image may depict a first object and another object, and the position of the first object relative to the other object may be determined as a function of depth information associated with at least some of the pixels depicting the first object and depth information associated with at least some of the pixels depicting the other object.

[0089] The attributes describing the location are selected from a set of attributes depending on the location.

[0090] The template may include two first placeholders and one second placeholder for attributes of the set of attributes.

[0091] The second placeholder may be replaced with an attribute, and the two first placeholders may be replaced with the class names of the first object and the other object, respectively.

[0092] Providing the caption 302 may include providing a set of templates for statements, where a template in the set of templates includes a first placeholder and a portion of a respective statement, and selecting a template for the caption 302 from the set of templates, particularly randomly.

[0093] An example caption generation process for an example digital image may include determining a semantic segmentation of the example digital image. The example digital image may include depth information provided by, for example, a LiDAR sensor.

[0094] The semantic segmentation may be determined by a neural network that is pre-trained for semantic segmentation.

[0095] The semantic segmentation is an output segmentation that assigns category and / or class names to objects, which may be, for example, infrastructure, or the environment, e.g., buildings, walls, vegetation, and objects, which may be actors, e.g., cars, pedestrians. The semantic segmentation is configured, for example, for the perception of driving scenes.

[0096] The output segmentation is filtered, for example, according to the following criteria: Filtering refers to ignoring or discarding segments in the segmentation that do not match the criteria for further steps in the caption generation process.

[0097] Confidence: A threshold on the classification confidence for each pixel may be used to discard or ignore pixels with confidence below the threshold.

[0098] Stability of neural network output: Multiple forward passes may be performed by the neural network on an input based on an exemplary digital image, where different iid samples of Gaussian noise are added to the exemplary digital image to create the input. The exemplary digital image may also be used in multiple forward passes without noise. This results in a pixel-by-pixel distribution of classification vectors that characterize the stability of the segmentation network output in the neighborhood of the exemplary digital image. The pixel-by-pixel distribution may be approximated by a Gaussian distribution. A maximum threshold for covariance and a minimum threshold for the probability of the output class name may be set to discard or ignore the segmentation output if the covariance of the multiple forward passes exceeds a maximum threshold or if the probability of the output class name is below a minimum threshold.

[0099] Physical size of the segment: A minimum size threshold may be defined for the size of the segment. A minimum size may be assigned per class name. The physical size of the segment is determined, for example, using depth information for an exemplary digital image.

[0100] No maximum size threshold may be defined since a single segment may contain multiple objects of the same class and thus violate such a maximum threshold.

[0101] An exemplary caption generation process may include collecting category names that correspond to segments that were not discarded in the previous step.

[0102] An exemplary caption generation process may include constructing a caption 302 by applying randomized templates to the detected categories in the following manner.

[0103] The overall structure of the statements in caption 302 is: Photos of {} {}[quality] photos Rendering {} Snapshot of {} There are multiple {} in the scene The scene contains {} This is a depiction of {} Scene contains {} Driving scene photos with {} Randomized from a set of templates including (but not limited to) [quality] is a placeholder for one of the attributes, e.g., bad, good, blurry, clean, cropped, close up, nice, large, small. where {} is a placeholder for a listing of the contents of an exemplary digital image.

[0104] The placeholders {} for listings are replaced, for example, as described below.

[0105] A random category is selected from the collected categories, and the class names associated with the selected category are concatenated, separated by commas, and added randomly.

[0106] In statements that use the parts "there is {}" or "there are multiple {}", the placeholder {} is replaced by the text that replaced the placeholder in the statement already created for the caption 302.

[0107] The following additional criteria: At least two different classes are associated with the categories, and at least one segment in the exemplary digital image is associated with each of the at least two classes; or A class name is associated with the category, and at least two segments in the exemplary digital image are associated with the class name. A random category may be selected from the collected categories that satisfies:

[0108] The placeholder {} is replaced with text containing the name of a category that meets the additional criteria, e.g., "a group of [category name]" or "multiple [category names]". According to one example, a category name is used only if the physical size of the segment exceeds a per-category threshold.

[0109] The placeholder {} may include at least one class name or at least one category name and may be modified with a geometric relationship based on the three-dimensional position of the segment in the real world relative to, for example, a camera that captured the exemplary digital image. The geometric relationship may be "further back," "left," or "right," according to the geometric relationship detected in the real world. The geometric relationship may be determined according to the depth associated with pixels within the respective segment of the exemplary digital image.

[0110] A geometric relationship may be expressed as a mathematical formula or a distance, e.g. <n>There is a car meters ahead," or " <m>There is a car meters to your left," <m>This may include "there is a car meters to the right".

[0111] Filtering the output segmentation based on physical object size, stability and / or reliability improves the resulting caption 302. The caption 302 is based on object type and, when depth is used, also on the shape of the scene.

[0112] Randomization of various caption templates automatically generates captions 302 for individual scene components.

[0113] This allows for large-scale randomization of captions during contrastive training.

[0114] This does not require rapid human tuning or annotation, or paid access to pre-trained models.

[0115] An example caption generation process may include constructing a caption 302 to further include the output of the image caption model.

[0116] For example, the image portion corresponding to the selected segment may be mapped by the image caption model, e.g., as a transformer input token. For example, the output of the image caption model is then concatenated with the caption 302 obtained according to the steps above.

[0117] This tends to produce caption output that is more focused on the object of interest.

[0118] The caption 302 may be determined for digital images used to train the first encoder 202. While randomization may be performed during training, other steps of the process for generating the caption 302 may be performed prior to training on the digital images used for training.

[0119] The method includes step 518.

[0120] Step 518 involves mapping, by the third encoder 304, the caption 302 of the first digital image 208 to a third encoding 306 in the first feature space.

[0121] The method includes step 520.

[0122] Step 520 includes training the first encoder 202 as a function of the distance between the first encoding 206 and the third encoding 306 .

[0123] Step 520 may include training the third encoder 304 as a function of the distance between the first encoding 206 and the third encoding 306. The third encoder 304 is, for example, fine-tuned as a function of the distance between the first encoding 206 and the third encoding 306.

[0124] Training may include repeating steps 514-520 for multiple pairs of the first radar spectrum and the first digital image. Training may include evaluating a second loss 308, for example cosine similarity.

[0125] The method ends after step 520 with the first encoder 202 trained.

[0126] FIG. 6 depicts a flowchart of a method for training, testing, validating, or verifying the first model 402.

[0127] A method for training, testing, validating, or verifying a first model 402 may include step 602 .

[0128] The method may start with a trained first encoder 202.

[0129] Step 602 includes mapping the first radar spectrum 204 to a first encoding 408 in a second feature space by a first portion 404 of a first model 402 .

[0130] The method includes step 604 .

[0131] Step 604 involves mapping the first encoding 206 in the first feature space to a second encoding 412 in the second feature space by a second model 410 .

[0132] The method includes step 606 .

[0133] Step 606 includes mapping the first encoding in the second feature space 408 and the second encoding in the second feature space 412 to an output 414 of the first model 402 by the second part 406 of the first model 402.

[0134] The method includes step 608.

[0135] Step 608 includes providing a ground truth for the output 414 .

[0136] The method includes step 610 .

[0137] Step 610 involves training, testing, validating or verifying the first model 402 as a function of the difference between the output 414 and the ground truth.

[0138] The output 414 for the first radar spectrum 204 and the ground truth characterizes at least one object detected in the radar spectrum, or an event recognized in the radar spectrum, or a segmentation of the radar spectrum.

[0139] Training may include repeating steps 602-610 for multiple pairs of the first radar spectrum and the first digital image. Training may include evaluating a second loss 308, for example cosine similarity.

[0140] The method ends after step 610 with the first model 402 trained.

[0141] FIG. 7 depicts a flowchart of a method for object detection, event recognition or segmentation by the first model 402.

[0142] A method for object detection, event recognition, or segmentation may include step 702 .

[0143] The method may begin at step 702 and train a first model 402 .

[0144] Step 702 involves capturing a spectrum with a radar sensor.

[0145] The method includes step 704 .

[0146] Step 704 involves mapping the spectrum by the first encoder 202 to an encoding of the spectrum in the first feature space.

[0147] The method includes step 706 .

[0148] Step 706 includes mapping the encoding in the first feature space to a first encoding of the spectrum in a second feature space by a second model 410 .

[0149] The method includes step 708 .

[0150] Step 708 includes mapping the spectrum with a portion of the first model 402 to a second encoding of the spectrum in a second feature space.

[0151] The method includes step 710 .

[0152] Step 710 includes mapping the first encoding of the spectrum in the second feature space and the second encoding of the spectrum in the second feature space by the second part 406 of the first model to an output of the first model 402.

[0153] The output 414 may characterize at least one object detected in the spectrum, which may be a traffic sign, a road surface, a person, a pedestrian, an animal, a plant, a vehicle, a road object, or a building.

[0154] The output 414 can characterize the events recognized in the spectrum. The events can be the state of a traffic sign, a person's gestures.

[0155] The output 414 may characterize the spectral segmentation with respect to traffic signs, road surfaces, pedestrians, animals, vehicles, road objects, and buildings, among others.

[0156] The method may include step 712 .

[0157] Step 712 involves operating, in particular moving or stopping, the technical system 106 in response to the output 414 .

[0158] For example, the technical system 106 is a vehicle and the method includes stopping the vehicle if the detected object is a traffic sign indicating to stop, such as a stop sign, and moving the vehicle otherwise or if the detected object is a traffic sign indicating to move, such as a yield sign.

[0159] For example, the technical system 106 is a vehicle and the method includes stopping the vehicle if the detected event is a traffic sign indicating to stop, such as a traffic light changing to red, and moving the vehicle otherwise or if the detected event is a traffic sign indicating to move, such as a traffic light changing to green.

[0160] Step 712 includes operating the technical system 106 to display the spectral segmentation according to the output 414 .

[0161] The first encoder 202 may be a neural network. The second encoder 210 may be a neural network. The third encoder 304 may be a neural network. The second encoder 210 may be an image encoder for an image language model (ILM). The third encoder 304 may be a CLIP text encoder for an ILM. ILM and CLIP are described in Radford, Alec, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry et al., "Learning Transferable Visual Models From Natural Language Supervision.", arXiv, 26 Feb. 2021, https: / / doi.org / 10.48550 / arXiv.2103.00020.

[0162] Using the trained first encoder and the trained third encoder 304, the associated caption 302 and radar spectrum 204 generate similar embeddings, particularly in the sense of cosine similarity. The trained first encoder and the trained third encoder 304 represent a spectral language model that can be used, for example, for data retrieval or classification.< / m> < / m> < / n>

Claims

1. 1. A computer-implemented method for training a first encoder (202) for mapping a radar spectrum to a coding for training, testing, validating or verifying a first model (402) configured for encoding, in particular object detection, event recognition or segmentation, comprising: The method comprises: providing (502) the first encoder (202), the first encoder (202) configured to map (502) a first radar spectrum (204) to a first encoding (206) in a first feature space; providing (504) a second encoder (210), the second encoder (210) configured to map (504) the first digital image (208) to a second encoding (212) in the first feature space; providing (506) the first radar spectrum (204) and the first digital image (208), the first radar spectrum (204) including radar returns of at least a portion of a first object, the first digital image (208) depicting at least a portion of the first object, the first radar spectrum (204) and the first digital image (208) preferably representing the same real world scene simultaneously; mapping (508) the first radar spectrum (204) to the first encoding (206) by the first encoder (202); Mapping (510) the first digital image (208) to the second encoding (212) by the second encoder (210); training (512) the first encoder (202) and / or the second encoder (210) as a function of the distance between the first encoding (206) and the second encoding (212); A method comprising: The method comprises: providing (514) a third encoder (304) configured to map captions to an encoding in the first feature space; Providing (516) a caption (302) for the first digital image; mapping (518) the caption (302) of the first digital image (208) to a third encoding (306) in the first feature space by the third encoder (304); training (520) the first encoder (202) and / or the third encoder (304) as a function of a distance between the first encoding (206) and the third encoding (306), wherein providing (516) the caption (302) of the first digital image (208) comprises determining a semantic segmentation, the semantic segmentation associating (520) a first portion of the first digital image (208), in particular a first pixel or a first segment of pixels, with a class name; providing a template for the caption (302), the template including a portion of a statement of the caption and a first placeholder; replacing the first space holder in the template with the class name to create the statement; A method comprising:

2. 1. A computer-implemented method for training a first encoder (202) for mapping a radar spectrum to a coding for training, testing, validating or verifying a first model (402) configured for encoding, in particular object detection, event recognition or segmentation, comprising: The method comprises: providing (502) the first encoder (202), the first encoder (202) configured to map (502) a first radar spectrum (204) to a first encoding (206) in a first feature space; providing (504) a second encoder (210), the second encoder (210) configured to map (504) the first digital image (208) to a second encoding (212) in the first feature space; providing (506) the first radar spectrum (204) and the first digital image (208), the first radar spectrum (204) including radar returns of at least a portion of a first object, the first digital image (208) depicting at least a portion of the first object, the first radar spectrum (204) and the first digital image (208) preferably representing the same real world scene simultaneously; mapping (508) the first radar spectrum (204) to the first encoding (206) by the first encoder (202); Mapping (510) the first digital image (208) to the second encoding (212) by the second encoder (210); training (512) the first encoder (202) and / or the second encoder (210) as a function of the distance between the first encoding (206) and the second encoding (212); A method comprising: The method comprises: providing a third encoder (304) configured to map captions to an encoding in the first feature space; providing a second radar spectrum and a second digital image, the second digital image depicting at least a part of an object, in particular the first object or a second object, the second radar spectrum including radar reflections of at least a part of the object, the second radar spectrum and the second digital image preferably representing the same real world scene simultaneously; providing a caption for the second digital image; mapping the caption of the second digital image to a third encoding in the first feature space by the third encoder; training the first encoder (202) and / or the third encoder (304) according to a distance between the first encoding (206) and the third encoding, wherein providing the caption for the second digital image comprises determining a semantic segmentation, the semantic segmentation associating a first portion of the second digital image, in particular a first pixel or a first segment of pixels, with a class name; providing a template for the caption, the template including a portion of a statement of the caption and a first placeholder; replacing the first space holder in the template with the class name to create the statement; A method comprising:

3. Providing (516) the caption (302) of the first digital image (208) includes: Providing a set of class names, Providing a set of categories; Including, At least one class name in the set of class names is associated with at least one category in the set of categories; Providing the template comprises: providing the template having the first placeholder for a category of the set of categories; Providing (516) the caption (302) of the first digital image (208) includes: - associating at least one portion of said first digital image (208), in particular at least one pixel or at least one segment of pixels, with a class name from said set of class names; replacing the placeholder for the category with a first class name associated with the category and associated with at least one portion of the first digital image (206), in particular at least one pixel or at least one segment of pixels, to create a first statement of the caption (302) of the first digital image (206); replacing the placeholder for the category with a second class name associated with the category and associated with at least one portion of the first digital image (208), in particular at least one pixel or at least one segment of pixels, to create a second statement of the caption (302) of the first digital image (208); Including, 2. The method according to claim 1 .

4. Providing the caption for the second digital image includes: - associating at least one portion of said second digital image, in particular at least one pixel or at least one segment of pixels, with a class name from said set of class names; replacing the placeholder for the category with a first class name associated with the category and associated with at least one portion of the second digital image, in particular at least one pixel or at least one segment of pixels, to create a first statement of the caption of the second digital image; replacing the placeholder for the category with a second class name associated with the category and associated with at least one portion of the second digital image, in particular at least one pixel or at least one segment of pixels, to create a second statement of the caption for the second digital image; Including, 3. The method according to claim 2 .

5. Providing (508) the first digital image (208) includes: providing depth information associated with the pixels of the first digital image, where providing (516) the caption (302) of the first digital image (208) includes providing a set of attributes that describe a position of an object relative to another object depicted in the first digital image (208); determining a position of the first object relative to another object depicted in the first digital image (208) in response to the depth information associated with at least some of the pixels depicting the first object and the depth information associated with at least some of the pixels depicting the other object; selecting an attribute from the set of attributes according to the location; providing the template having two first placeholders and one second placeholder for attributes of the set of attributes; replacing the second placeholder with the attribute and the two first placeholders with the class names of the first object and the other object, respectively; Including, 4. The method according to claim 1 or 3.

6. Providing the second digital image includes: providing depth information associated with the pixels of the second digital image, wherein providing the caption for the second digital image includes providing a set of attributes that describe a position of an object relative to another object depicted in the second digital image; determining a position of the second object relative to another object depicted in the second digital image in response to the depth information associated with at least some of the pixels depicting the second object and the depth information associated with at least some of the pixels depicting the other object; selecting an attribute from the set of attributes according to the location; providing the template having two first placeholders and one second placeholder for attributes of the set of attributes; replacing the second placeholder with the attribute and the two first placeholders with the class names of the first object and the other object, respectively; Including, 5. The method according to claim 2 or 4.

7. Providing (516) the caption (302) comprises: providing a set of templates for statements, the templates in the set of templates including the first placeholder and a portion of a respective statement; selecting said template for said caption (302) from a set of templates, in particular randomly; Including, 7. The method according to claim 1, wherein the first and second electrodes are connected to a first electrode.

8. The method comprises: Mapping (602) the first radar spectrum (204) to a first encoding (408) in the second feature space by a first portion (404) of the first model (402) configured to map the radar spectrum to an encoding in a second feature space; Mapping (604) the first encoding (206) in the first feature space to a second encoding (412) in the second feature space by a second model (410) configured to map the encoding in the first feature space to an encoding in the second feature space; Mapping (606) the first encoding (408) in the second feature space by a second portion (406) of the first model (402) and the second encoding (412) in the second feature space to an output (414) of the first model (402); Providing (608) a ground truth for the output (414); training (610) the first model (402) as a function of differences between the output (414) and the ground truth, the output (414) and the ground truth characterizing at least one object detected in the radar spectrum, or an event recognized in the radar spectrum, or a segmentation of the radar spectrum; 8. The method according to claim 1 , further comprising:

9. The method comprises: mapping the second digital image to a first encoding in the second feature space by a first portion of the first model (402) configured for mapping a digital image to an encoding in the second feature space; mapping the first encoding to a second encoding in the second feature space by a second model (410) configured to map an encoding from the first feature space to an encoding in the second feature space; mapping the first encoding in the second feature space and the second encoding in the second feature space by a second part of the first model (402) to an output of the first model (402); providing a ground truth for the output; training the first model (402) as a function of a difference between the output and the ground truth, the output and the ground truth characterizing at least one object detected in the radar spectrum, or an event recognized in the radar spectrum, or a segmentation of the radar spectrum; 5. The method according to claim 2 or 4, characterized in that it comprises:

10. The method comprises: providing a third radar spectrum; mapping, by the first encoder (204), the third radar spectrum to an encoding of the third radar spectrum in the first feature space; mapping the third radar spectrum to a first encoding in the second feature space by a first portion (404) of a first model (402) configured to map to an encoding in a second feature space; mapping the encoding of the third radar spectrum in the first feature space to a second encoding in the second feature space by a second model (410) configured to map an encoding in the first feature space to an encoding in the second feature space; mapping the first encoding in the second feature space and the second encoding in the second feature space by a second part (406) of the first model (402) to an output (414) of the first model (402); Providing (608) a ground truth for the output (414); training (610) the first model (402) as a function of differences between the output (414) and the ground truth, the output (414) and the ground truth characterizing at least one object detected in the radar spectrum, or an event recognized in the radar spectrum, or a segmentation of the radar spectrum; 10. The method according to claim 1, further comprising:

11. The method comprises: Capturing (702) the first radar spectrum with a radar sensor; Mapping (706) the first encoding in the first feature space to a first encoding in the second feature space by the second model (410); Mapping (708) the first radar spectrum to a second encoding of the first radar spectrum in the second feature space using the portion of the first model (402); and Mapping (710) the first encoding of the first radar spectrum in the second feature space and the second encoding of the first radar spectrum in the second feature space by the second part (406) of the first model to the output of the first model (402) characterizing at least one object detected in the first radar spectrum, in particular a traffic sign, a road surface, a person, a pedestrian, an animal, a plant, a vehicle, a road object, a building, or characterizing an event recognized in the first radar spectrum, in particular a state of a traffic sign, a human gesture, or characterizing a segmentation of the spectrum, in particular with respect to a traffic sign, a road surface, a pedestrian, an animal, a vehicle, a road object, a building; 11. The method according to claim 8, further comprising:

12. The method comprises:

12. The method according to claim 11, characterized in that it comprises operating (712), in particular moving or stopping, a technical system, in particular a computer controlled device, preferably a robot, a vehicle, a manufacturing device, a household appliance, a power tool, an access control system, a personal assistant, a medical imaging system.

13. An apparatus (100) for training a first encoder (202) for mapping a radar spectrum to a coding for training, testing, validating or verifying a first model (402) configured for encoding, in particular object detection, event recognition or segmentation, comprising: The device (100) comprises: At least one processor (102); At least one memory (104); Equipped with The at least one processor (102) is configured to execute instructions which, when executed by the at least one processor (102), cause the apparatus (100) to perform steps of a method according to any one of claims 1 to 12, The at least one memory (104) is configured to store the instructions.

1. An apparatus (100).

14. The apparatus (100) comprises a radar sensor (108) configured to capture a radar spectrum; The device (100) comprises: determining an output of said first model (402) characterizing at least one object detected in said spectrum, in particular a traffic sign, a road surface, a person, a pedestrian, an animal, a plant, a vehicle, a road object, a building, or characterizing an event recognized in said spectrum, in particular a state of a traffic sign, a human gesture, or characterizing a segmentation of said spectrum, in particular with respect to a traffic sign, a road surface, a pedestrian, an animal, a vehicle, a road object, a building, Depending on said output, a technical system (106), in particular a computer controlled device, is operated, in particular moved or stopped, preferably a robot, a vehicle, a manufacturing device, a household appliance, a power tool, an access control system, a personal assistant, a medical imaging system.

14. The device (100) according to claim 13, characterized in that it is configured to:

15. A computer program comprising computer readable instructions for causing a computer to carry out a method according to any one of claims 1 to 12, when the computer program is executed by the computer.

Images