Technique for extracting feature of environment from image data

Abstract

To provide a technique for extracting features of an environment from image data.SOLUTION: A computer implemented method comprises the steps of: receiving data (804) indicative of a visual domain of an environment; and generating a visual domain textual prompt (812) based on the received data (804) indicative of the visual domain of the environment. The method further comprises a step of receiving image data representing the environment. The method further comprises a step of extracting, in particular local, features (826) of the environment from the received image data. The extracting of the, in particular local, features is performed by conditional feature extracting models (814, 822). The extracting of the, in particular local, features (826) is conditioned by the generated visual domain textual prompt (812).SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] Techniques for extracting environmental features from image data and for training a conditional feature extraction model for extracting environmental (particularly local) features from image data are provided, including methods, computing devices, systems, computer program products, and computer-readable storage media. [Background technology]

[0002] Prior art Recent large-scale text-to-image generation models, such as stable diffusion (SD; described in Rombach et al. in “High-Resolution Image Synthesis with Latent Diffusion Models”, arXiv:2112.10752v2 [cs.CV], incorporated herein by reference), are trained on a large number of text-image pairs and have the ability to generate diverse and creative images with astonishing performance when given text prompts. In particular, SD is a state-of-the-art open-source visual language generation model trained on billions of text-image pairs. SD is a latent diffusion model, which is a special case of a diffusion model trained in a predetermined latent space instead of the original image space. More specifically, following P. Esser et al., "Taming Transformers for High-Resolution Image Synthesis," arXiv:2012.09841v3 [cs.CV] (also known by the abbreviation VQ-GAN), which is incorporated herein by reference, SD first trains an autoencoder, where images are encoded into a latent space Z and decoded to reconstruct a given image, as shown schematically in FIG. 5. In the second stage, a diffusion model is trained in the latent space Z. Because it is trained on a large amount of data, SD has a rich learned representation.

[0003] In a recent work, W. Zhao et al. in “Unleashing Text-to-Image Diffusion Models for Visual Perception (VPD)”, arXiv:2303.02153v1 [cs.CV], which is incorporated herein by reference, various visual recognition tasks, such as semantic segmentation and deep estimation, are provided, building on high-level SD and leveraging pre-trained knowledge of denoising UNets. W. Zhao et al. explore fine-tuning the SD and simultaneously training task-specific decoders for various visual recognition tasks (e.g., semantic segmentation, depth estimation), as shown schematically in Figure 6. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Rombach et al. in “High-Resolution Image Synthesis with Latent Diffusion Models”, arXiv:2112.10752v2 [cs.CV] [Non-patent document 2] P. Esser et al., “Taming Transformers for High-Resolution Image Synthesis”, arXiv:2012.09841v3 [cs.CV] [Non-patent document 3] W. Zhao et al. in “Unleashing Text-to-Image Diffusion Models for Visual Perception (VPD)”, arXiv:2303.02153v1 [cs.CV] Summary of the Invention [Problem to be solved by the invention]

[0005] However, according to W. Zhao et al., SD is fine-tuned for a specific training set and is only evaluated on the same training domain during inference. [Means for solving the problem]

[0006] Disclosure of the Invention In the following, techniques for extracting environmental features from image data and for training a conditional feature extraction model for extracting environmental (especially local) features from image data are described in the context of the claimed methods and the claimed computing devices. Features, advantages, or alternative embodiments herein may be mapped to other claimed subject matter (e.g., a computing device, a computer program, or a system including a computer program product), and vice versa. In other words, a claim relating to a computing device may be realized by features described or claimed in the context of a method. In this case, functional features of the method may be embodied by structural units of the system, and vice versa.

[0007] According to a first method aspect, a computer-implemented method for extracting features of an environment from image data is provided. The method includes receiving data indicative of a visual domain of the environment. The method further includes generating a visual domain text prompt based on the received data indicative of the visual domain of the environment. The method further includes receiving image data representing the environment. The method further includes extracting (particularly local) features of the environment from the received image data. The extraction of the (particularly local) features is performed by a conditional feature extraction model and is conditioned by the generated visual domain text prompt.

[0008] The method can be used to provide an image generation model that can be flexibly adapted based on visual domain text prompts that indicate the visual domain of the environment.

[0009] The computer-implemented techniques enable performing environmental feature extraction from received image data and preferably performing visual recognition tasks based on the extracted features, which are more robust and generalizable to various visual domains (e.g., different weather conditions, lighting conditions, and / or new countries). Alternatively or additionally, overfitting to a training data set (e.g., a data set including and / or based on features extracted conditioned by visual domain text prompts, particularly ground truth) for training a model (e.g., a neural network NN and / or an artificial intelligence AI) for performing the feature extraction and / or visual recognition tasks can be avoided. For example, accurate results of feature extraction and / or visual recognition tasks (e.g., object detection, depth estimation, classification, and / or semantic segmentation) may be important for safe autonomous driving (AD) and / or for safe robotic operation (e.g., in manufacturing environments and / or home automation).

[0010] AD driving functions may include full automation (also described as fully autonomous driving, specifically L5), at least partial automation (e.g., highly automated L4, conditional automated L3, or partial automated L2), and / or driver assistance automation (e.g., L1, including, for example, adaptive cruise control (ACC)).

[0011] Text prompts may also be described as sentence prompts and / or verbal prompts. Alternatively or additionally, text prompts may be provided and / or processed in text- and / or audio-based formats.

[0012] Visual domain text prompts (also domain indicators) can improve the (particularly visual) domain flexibility of feature extraction and / or visual recognition tasks (e.g., by using knowledge of predicted underexposure or overexposure).

[0013] Models trained (e.g., pre-trained) on training data sets (including, in particular, the results of feature extraction conditioned by visual-domain text prompts and / or the results of visual recognition tasks based on feature extraction) can be significantly improved, in particular by enabling the transfer of (particularly rich) prior knowledge and / or rapid training convergence.

[0014] The environment may also be described as a scene, setting, and / or ambience (especially physical, real, and / or natural). The environment represented by the received image data may include the environment of a (e.g., AD) vehicle, a robot, a surveillance system, and / or a human. In particular, the environment here may represent a traffic scene. Alternatively or additionally, the environment may be a real environment, a natural environment, a suburban environment, a built environment, an urban environment, and / or a spatial environment. For example, the environment of a (e.g., AD) vehicle may include a road layout, the environment of a robot may include a manufacturing floor, and the environment of a surveillance system and / or a human may include buildings. For example, in the case of an AD application, the environment may include an external environment, in particular a street view. Further exemplary, in the case of a robot, the environment may include a factory floor with an on-site area and / or an off-site area.

[0015] The image data representing the environment can be received as digital (and / or analog) data, in particular from a (e.g., video) camera. Alternatively or additionally, the image data of the environment can be acquired using a (e.g., video and / or thermal) camera, radar, LiDAR, ultrasonic sensor, and / or motion sensor.

[0016] The visual domain may be described as visibility conditions, optical conditions, and / or may include an indication of the visual (i.e., optical) settings of the environment (e.g., taking into account lighting conditions and / or weather conditions). Alternatively or additionally, the visual domain may be generated from geographic identifiers and / or may be country-specific conditions, e.g., lighting conditions based on the time of image data acquisition relative to local sunrise and sunset times, and / or traffic signs and / or street layout for exterior environments, and / or types of natural and / or artificial lighting for indoor environments. Lighting conditions (abbreviated as illumination) may alternatively be described as illumination amount and / or luminance.

[0017] The data indicative of the visual domain may comprise digital data, for example the result of a brightness determination of the imaging data. Alternatively or additionally, the digital data may be converted from (e.g. manual) input (e.g. selected brightness and / or weather conditions) and / or compressed (e.g. arranged as a table), for example with respect to a selected one of a plurality of predefined visual domains (e.g. especially manually and / or automatically).

[0018] The step of generating the visual domain text prompt can be performed at least in part by a Contrastive Language-Image Pretraining (CLIP) encoder. In some embodiments, a pre-trained visual question answering (VQA) model, such as mPLUG, can first receive data indicating the visual domain of the environment and provide results, particularly as probabilities of visual domain classes among a distinct set of visual domain classes. In one embodiment, the visual domain class with the highest probability can be selected as the visual domain for which the visual domain text prompt is generated. In an alternative embodiment, the visual domain for which the visual domain text prompt is generated can correspond to a weighted combination of multiple K visual domain classes having the K highest probabilities (e.g., K=2 for the two highest probabilities).

[0019] Conditional feature extraction models may include generative models and / or (especially large-scale) text-to-image generation artificial intelligence (AI).

[0020] The extracted features may locally represent the environment. Alternatively or additionally, the extracted features may include objects, shapes and / or colors.

[0021] Since the feature extraction process is conditioned on text prompts indicative of the visual domain, for example, location and / or country information may be associated with the extracted features. For example, some local traffic signs may vary from place to place, and / or driving on the left or right may be used as prior knowledge for model prediction. Feature extraction can, for example, additionally be conditioned on text prompts of environment classes, where the environment classes relate to, for example, countries (e.g., typical road layouts, and / or the layout and / or appearance of traffic signs and / or traffic lights), suburban environments (e.g., with respect to typical cultivated fields and / or buildings), and / or urban environments.

[0022] The (particularly locally) extracted features can be complemented by cross-attention maps, each of which can relate to a different patch of (particularly locally) extracted features. Alternatively or additionally, the cross-attention maps can enable consistency across environments contained in the image data, particularly as a result of a subsequently performed visual recognition task.

[0023] The extracted environmental features, optionally complemented by a Cross-Attention Map, can be provided for output.

[0024] The method may further comprise the step of performing a visual recognition task on the received image data representing the environment based on the extracted (especially local) features. The result of the performed visual recognition task may in particular comprise a (e.g. weighted) average over the features at different time steps. Alternatively or additionally, the result of the performed visual recognition task may also be provided and / or output.

[0025] The visual recognition task can be performed by a task-specific decoder.

[0026] By performing visual recognition tasks, the environment captured in the image data can be analyzed in a sufficiently accurate manner to enable, for example, AD and / or autonomous robot use and / or access control.

[0027] Visual recognition tasks may include depth estimation, object detection, classification, and / or semantic segmentation. Thus, the method can be used for image classification. Image classification may include object detection and segmentation, especially semantic segmentation. Classification may be based on low-level features of the image data, such as pixels, edges, and / or corners.

[0028] Depth estimation can perform distance estimation (e.g., to obstacles, particularly for AD vehicles and / or autonomous robots). Alternatively or additionally, object detection and / or classification can enable detection of paths, road surfaces, traffic signs, pedestrians, vehicles, (e.g., other) obstacles (e.g., trees), sky, and / or destinations. Further alternatively or additionally, classification can be performed to, for example, recognize traffic signs and / or obstacles and plan a path (and / or trajectory) accordingly. Further alternatively or additionally, semantic segmentation can provide a (e.g., complete) semantic content of the environment, which can improve the accuracy of path (and / or trajectory) planning for (e.g., AD) vehicles and / or (e.g., autonomous) robots, particularly depending on the real-time situation in the environment.

[0029] Alternatively or additionally, the visual recognition task may include, for example, performing regression analysis to identify (especially continuous) values, such as distance (e.g., by depth estimation), speed (i.e., velocity), acceleration, and / or tracking objects and / or people by considering edge and / or pixel attributes of the image data and / or by analysing (especially image) data, such as scalar time series, to perform corresponding (velocity and / or acceleration) measurements and / or tracking.

[0030] By performing visual recognition tasks, anomaly detection and, optionally, control of e.g. (particularly AD) vehicles and / or (particularly autonomous) robots after anomaly detection in a fail-safe mode may be provided.

[0031] For example, in an embodiment applied to AD, some anomalies may appear on the road, such as those described in Dan Zhang et al. in "Anomaly-Aware Semantic Segmentation via Style-Aligned OoD Augmentation," arXiv:2308.09965v1 [cs.CV], which is incorporated herein by reference. For example, if an accident occurs, "potential debris on the road" may be indicated in the text prompt (e.g., environment class).

[0032] Performance of a visual recognition task on image data can be conditioned on generated visual domain text prompts.

[0033] According to some embodiments, the extraction of (particularly local) features can be performed in multiple time steps, particularly with respect to generating latent state data (briefly, noisy latent values) each having different noise levels, and the visual recognition task can be performed based on a combination (and / or aggregation and / or weighting) of features extracted from the multiple time steps.

[0034] Alternatively or additionally, conditioning the generated visual domain text prompts may include conditioning on visual domain text prompts associated with and combined (and / or aggregated and / or weighted) visual domain text prompts across multiple time steps, particularly in connection with generating latent state data.

[0035] The received data indicative of the visual domain may include environmental sensor data, position data determined by a positioning system (particularly a satellite navigation system), manually entered data, and / or electronically available information data.

[0036] The environmental sensor data can be acquired by environmental sensors. The environmental sensors can include optical sensors, acoustic sensors, rain sensors, humidity sensors, heat sensors, temperature sensors, wind sensors, radar sensors, LiDAR sensors, ultrasonic sensors, and / or motion sensors. Alternatively or additionally, the environmental sensors can be attached to a (e.g., AD) vehicle, a (e.g., autonomous) robot, and / or can be stationary within the environment. The environmental sensors can advantageously determine information related to the visual domain, such as weather conditions (e.g., sunny, rainy, and / or foggy, particularly using humidity sensors) and / or lighting conditions (e.g., daytime, nighttime, dawn, and / or dusk), locally and in real time relative to (and / or within) the environment.

[0037] Positioning systems, in particular satellite navigation systems, may include global navigation satellite systems (GNSS), such as the Global Positioning System (GPS) and / or Galileo. Alternatively or additionally, positioning systems may also include regional systems (e.g., using radio signals, in particular), local positioning systems (LPS) (e.g., using radio and / or optical signal beacons, Wi-Fi and / or Li-Fi), and / or indoor systems (e.g., using ultrasound and / or multilateration, ActiveBat). These positioning systems may advantageously determine information about the visual domain, such as terrain type, construction developments and / or fixed installations (here, for example, information about lighting settings, such as dimming under bridges and / or in tunnels), in relation to the environment.

[0038] The manually entered data can be received by a user interface (UI), in particular a graphic user interface (GUI), and / or by a microphone (and / or acoustic interface) from a user (such as a human operator, e.g., a driver of an AD vehicle, in particular, and / or an operator of a robot). The manually entered data can include data selected from drop-down menus and / or any form of individual selection (e.g., sunny, foggy, rainy, snowy, and / or dark for lighting conditions). Alternatively or additionally, the manually entered data can include text (e.g., in natural language) describing the visual domain. The manually entered data allows a user to provide information describing the visual domain, in particular based on their own perceptual perception and / or based on readouts of information from one or more sensors without requiring a data connection of the sensors (e.g., wireless and / or wired).

[0039] The electronically available information data may be received via an interface (e.g., wireless and / or wired) (e.g., via the Internet or via cloud-based data storage), and / or may include data retrieved from the Internet, e.g., local weather forecasts, thereby advantageously allowing a wide range of, particularly existing, electronically available information to be utilized, e.g., for generating visual-domain text prompts.

[0040] The data indicative of the visual domain may include any combination of data types, for example, location data may be used to determine appropriate electronically available information data, including local weather forecasts for the environment.

[0041] The method may include receiving data indicative of a class related to the environment. The method may further include generating an environment class text prompt based on the received data indicative of the class related to the environment. Extraction of (particularly local) features of the environment from the received image data may further be conditioned by the generated environment class text prompt.

[0042] The generation of the environment class text prompts can be performed, at least in part, by the CLIP encoder.

[0043] Conventionally, CLIP encoders are frozen (and / or made unchangeable), particularly after the training phase, in contrast to the computer-implemented methods presented herein, but according to the computer-implemented methods presented herein, the generation of environmental class text prompts and / or visual domain text prompts is at least partially time-variable (e.g., after initial processing of the corresponding received data by the CLIP encoder) and / or can be performed by a trainable text adapter (e.g., including several layers of a multi-layer perceptron (MLP)).

[0044] By further conditioning the extraction of features of the generated environmental class text prompts, for example classification and / or semantic segmentation tasks can be enabled as visual recognition tasks.

[0045] Feature extraction can be further conditioned by environmental class text prompts. Conditioning on environmental class text prompts can enable and / or improve classification and / or semantic segmentation as visual recognition tasks.

[0046] The steps of generating visual domain text prompts and / or generating environmental class text prompts can be performed by an adaptive Contrastive Language-Image Pretraining (CLIP) encoder. The use of an adaptive CLIP encoder to generate (particularly visual domain and / or environmental class) text prompts enables and / or improves the generalizability of the computer-implemented method, particularly in terms of previously untrained (and / or unobserved) visual domains and / or previously untrained (and / or unobserved) environmental classes.

[0047] The conditional feature extraction model may include a generative image-to-feature model configured to extract environmental (particularly local) features from received environmental image data, and a conditioning model configured to encode generated visual-domain text prompts to condition and / or control the generative image-to-feature model.

[0048] The generative image-to-feature model is trainable (and / or operable) in a unified visual language space. Generative image-to-feature models include visual transformers (ViTs, e.g., as described in P. Esser et al. in “Taming Transformers for High-Resolution Image Synthesis”, arXiv:2012.09841v3 [cs.CV]), generative adversarial networks (GANs, particularly VG-GANs, e.g., as described in Yong Zheng Ong et al. in “VG-GAN: Conditional GAN ​​Framework for Graphical Design Generation”, 2022 IEEE International Conference on Image Processing (ICIP), Bordeaux, France, 2022, pp. 781-785, DOI: 10.1109 / ICIP46576.2022.9897935, which is incorporated herein by reference), and diffusion models for visual perception (VPDs, e.g., as described in Wenliang Zhao et al. in “Unleashing Text-to-Image Diffusion Models for Visual Perception”, arXiv:2303.02153 [cs.CV]). v1 [cs.CV]), and / or Stable Diffusion (SD), in particular SD denoising UNet.

[0049] When the method is performed, the generative image-to-feature model can be frozen and / or no further adaptation of the generative image-to-feature model is required during the fine-tuning and / or inference phases, in the sense that the weights of the generative image-to-feature model remain unchanged.

[0050] The optional environmental class text prompts can condition the Image-to-Feature model (e.g., directly) (e.g., by providing the environmental class text prompts to the encoding and / or decoding layers and / or to optional intermediate layers).

[0051] The generative image-to-feature model may include a diffusion model, particularly an SD network. The SD network may include a UNet architecture with an encoder and a decoder (particularly skip-connected).

[0052] The diffusion model (diffusion probabilistic model and / or score-based generative model) may include a machine learning (ML) model, in particular a generative model, configured to perform image generation, image denoising, inpainting, and / or superresolution by using a diffusion process, in particular a forward process (e.g., adding noise, in particular Gaussian noise, to an image), a backward process (e.g., predicting Gaussian noise in an image and compensating and / or subtracting accordingly), and a sampling procedure.

[0053] SD may include latent diffusion models. SD may relate to or include neural network (NN) architectures (models) for text-to-image (especially diffusion) generation. The goal of SD is to learn a diffusion process that generates a probability distribution for a given (image) data set.

[0054] SD NN architectures may include architectures based on convolutional neural networks (CNNs), particularly UNets with encoders and decoders with skip connections (e.g., between the encoder and decoder layers) and / or (e.g., cross-)attention mechanisms (e.g., within the encoder and / or decoder layers). SD NN architectures may also include a diffusion process in which input images are transformed into latent space images. With UNets, latent space images can be transformed back into images in non-latent (and / or image) space.

[0055] The denoising UNet can utilize latent diffusion and / or can be trained in latent space instead of the original image space. For example, SD can be trained specifically in latent space. Alternatively or additionally, a generalized diffusion model can be trained in the original (e.g., image) space.

[0056] The training may include two stages, with the first stage training an autoencoder that encodes and / or transforms images in a latent space and then decodes and / or reconstructs the images. In the second stage, a diffusion model can be trained in the latent space by forward diffusion, which gradually adds noise (e.g., Gaussian noise) (in particular according to a Markov chain), and by backward denoising (and / or UNet), which further predicts the noise (e.g., Gaussian noise).

[0057] SD can advantageously generate high-quality (e.g., high-resolution and / or photorealistic) synthetic images based on text prompts (particularly visual domains and / or environment classes) at low computational cost.

[0058] The computer-implemented method generalizes conventional generative image-to-feature models, particularly SD and / or VPD, by maintaining generalizability and / or facilitating adaptation to different visual domains and, optionally, different environment classes via corresponding textual prompts.

[0059] Compared to common "diffusion models," SD allows training diffusion models in latent space (especially pre-trained and / or, e.g., post-frozen autoencoders) instead of, e.g., image or pixel space. Because the latent space is much smaller than traditional pixel space, e.g., by a factor of 8, training in the latent space can reduce computational costs.

[0060] Alternatives to SD for generative image-to-feature models (and / or conditional feature extraction models) exist (briefly written as LCM), such as those described by Simian Luo et. al in “Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference”, arXiv:2310.04378v1 [cs.CV]. In general, techniques for extracting environmental features from image data are not limited to SD and can be implemented using LCM, for example. A major advantage of SD can be considered its open-source availability.

[0061] The conditioning model may include a ControlNet, which may include an encoder layer and a convolutional layer with a cross-attention mechanism for the image-to-feature model, particularly for the encoder of the SD network.

[0062] Generative Image-to-Feature models, particularly SD networks, can be conditioned by the visual domain using the conditioning model as a specially trainable branch, particularly by using ControlNet. For a fundamental discussion of conditioning SD networks using ControlNet, see L. Zhang et al. in "Adding Conditional Control to Text-to-Image Diffusion Models" (2023), available at the Computer Vision Foundation and IEEE Xplore, which is incorporated herein by reference.

[0063] The SD may include a frozen (e.g., after initial training) UNet, particularly to preserve trained (and / or prior) knowledge during fine-tuning. The ControlNet may include a trainable copy of the SD's UNet encoder for conditioning with visual domain text prompts. The ControlNet may further include one or more zero convolutional layers, through which visual domain text prompts are fed to the SD's decoder. The zero convolutional layers may include convolutional layers with both weights and biases initialized as 0 (e.g., 1×1).

[0064] A connection between an SD network and a ControlNet or a combination of an SD network and a ControlNet can provide efficient conditioning, especially in terms of computation, thereby saving time and (e.g., graphics processing unit, GPU) memory.

[0065] The combination of the SD network and ControlNet with visual domain text prompts and optionally environmental class text prompts can provide a particularly rich and generalizable framework for the performance of feature extraction and / or visual recognition tasks.

[0066] Conventional models for perceptual recognition tasks are trained on specific data sets that may contain limited data and / or scenarios due to costly annotation processes (e.g., for segmentation tasks). In contrast, techniques for extracting environmental features from image data, for example, according to a first method aspect, leverage the rich prior knowledge of pre-trained, large-scale text-to-image models that have observed a variety of scenarios and have a unified visual-linguistic latent space. Techniques for extracting environmental features from image data particularly take advantage of the control potential of language and / or text prompts, powerful pre-training knowledge to improve downstream tasks (e.g., AD, robotic motion planning, and / or access control), and the generalization and adaptation capabilities of models in various operational domains during deployment. Alternatively or additionally, only a lightweight decoder needs to be trained for a specific task, such as semantic segmentation and / or object detection.

[0067] Regarding a second method aspect, a computer-implemented method is provided for training a conditional feature extraction model for extracting environmental (especially local) features from image data conditioned by visual domain text prompts. The method includes receiving a training data set. The training data set includes a visual domain of the environment, image data of the environment, and visual domain text prompts indicating the environmental (especially local) features. The method further includes training the conditional feature extraction model based on the received training data set. Training the conditional feature extraction model includes receiving the environmental image data as input, receiving the visual domain text prompts as conditions, and receiving the (especially local) features as ground truth.

[0068] The training data set may include synthetic data (e.g., synthetic image data generated based on, among other things, local features and visual domain text prompts). Alternatively or additionally, the training data set may include real data, particularly real image data (e.g., having features, particularly local features and / or visual domain prompts as annotations).

[0069] By training a conditional feature extraction model on multiple training datasets, including in particular synthetic training datasets, the performance of feature extraction and / or visual recognition tasks can be adapted (and / or generalized) to visual domains and / or environment classes (for which it has not been specifically pre-trained) and / or for which real training datasets are lacking (e.g., due to a lack of real image data).

[0070] An example SD may be pre-trained on a large dataset, e.g., LAION-5B. Conditional feature extraction models can be trained on small, real-world datasets that are labeled for a specific task, e.g., semantic segmentation. For AD use cases, for example, the Cityscapes dataset (containing 4,000 datasets with high-quality annotations and 20,000 datasets with coarse annotations; https: / / www.cityscapes-dataset.com / ) can be used. Alternatively or additionally, synthetic data can also be used, especially if annotations (e.g., segmentation label maps) are generated along with the images.

[0071] The method according to the first method aspect may be used for autonomous driving (AD) and / or for planning the movement of a robot and / or for operating a domestic device and / or for controlling an access control system.

[0072] By extracting (especially local) features and / or by performing visual recognition tasks based on image data and visual domain text prompts (and optionally environment class text prompts), sufficiently accurate data about the environment can be provided to enable AD and / or path planning (and / or trajectory planning) of (especially AD) vehicles and / or (especially autonomous) robots and / or domestic devices (e.g., autonomously moving vacuum cleaners and / or AI butlers).

[0073] According to a first device aspect, a computing device is provided for extracting environmental features from image data. The computing device includes a visual domain instruction receiving interface configured to receive data indicating a visual domain of the environment. The device further includes a visual domain text prompt generation module configured to generate a visual domain text prompt for the environment based on the received data indicating the visual domain of the environment. The computing device further includes an environmental image data receiving interface configured to receive image data representing the environment. The computing device further includes a conditional feature extraction model configured to extract environmental features, particularly local features, from the received image data. The extraction of the (particularly local) features is conditioned by the generated visual domain text prompt. Optionally, the computing device according to the first device aspect may include a visual recognition task execution module configured to perform a visual recognition task for the environment based on the extracted (particularly local) features. Further optionally, the computing device according to the first device aspect may include an environmental class receiving interface configured to receive data indicating a class related to the environment. The computing device may further include an environmental class text prompt generation module configured to generate an environmental class text prompt based on the received data indicating a class related to the environment. The extraction of environmental (especially local) features from the received image data can further be conditioned by the generated environmental class text input prompt.

[0074] A computing device according to the first device aspect is configurable to perform any one of the steps and / or include any one of the features disclosed in connection with the first method aspect.

[0075] Regarding a second device aspect, a computing device is provided for training a conditional feature extraction model for extracting environmental (particularly local) features from image data conditioned by visual domain text prompts. The computing device includes a training data receiving interface configured to receive a training data set. The training data set includes a visual domain of the environment, image data of the environment, and visual domain text prompts indicating the environmental (particularly local) features. The computing device further includes a training module configured to train the conditional feature extraction model based on the received training data set. Training the conditional feature extraction model includes receiving the environmental image data as input, receiving the visual domain text prompts as conditions, and receiving the (particularly local) features as ground truth.

[0076] A computing device according to the second device aspect is configurable to perform any one of the steps and / or include any one of the features disclosed in connection with the second device aspect.

[0077] The computing device according to the first method aspect and the computing device according to the second method aspect may be the same.

[0078] In accordance with a system aspect, a system for extracting environmental features from image data is provided. The system includes a computing device according to the first device aspect, at least one sensor, and / or at least one interface for receiving data indicative of the visual domain and image data representative of the environment. The data indicative of the visual domain and / or the image data representative of the environment may be received wirelessly (e.g., from a sensor and / or a web server external to the AD vehicle). Alternatively or additionally, the data indicative of the visual domain and / or the image data representative of the environment may be received via a wired connection (e.g., to a fixedly installed access control system).

[0079] In another aspect, a computer program product is provided, the computer program product comprising program elements which, when loaded into a memory of a computing device, cause the computing device to perform the method steps for extracting environmental features from image data and / or the method steps for training a conditional feature extraction model for extracting environmental (particularly local) features from image data conditioned by visual-domain text prompts according to the first and / or second method aspects.

[0080] In yet another aspect, there is provided a computer readable medium having stored thereon program elements readable and executable by a computing device, which when executed by the computing device causes the method steps of extracting environmental features from image data and / or training a conditional feature extraction model to extract environmental features, in particular local features, from image data conditioned by visual domain text prompts according to the first method aspect and / or the second method aspect. [Brief explanation of the drawings]

[0081] [Figure 1] 1 is a flowchart of a method for extracting environmental features from image data. [Figure 2] 1 is a flowchart of a method for training a conditional feature extraction model to extract environmental (especially local) features from image data conditioned by visual domain text prompts. [Figure 3] 1 is a schematic diagram of the architecture of a computing device for extracting environmental features from image data. [Figure 4] 1 is a schematic diagram of the architecture of a computing device for training a conditional feature extraction model for extracting environmental (particularly local) features from image data conditioned by visual domain text prompts. [Figure 5] FIG. 1 is a diagram illustrating a conventional SD architecture. [Figure 6] FIG. 1 is a diagram illustrating a conventional VPD architecture. [Figure 7] FIG. 1 is a schematic diagram of a conventional ControlNet architecture. [Figure 8] FIG. 1 illustrates a schematic diagram of a technique for extracting environmental features from image data. [Figure 9A] FIG. 1 is an exemplary diagram illustrating data showing the visual domain of an environment along with a predetermined set of probability analyses for generating (particularly automatically) visual domain text prompts. [Figure 9B] FIG. 1 is an exemplary diagram illustrating data showing the visual domain of an environment along with a predetermined set of probability analyses for generating (particularly automatically) visual domain text prompts. [Figure 10A] FIG. 1 illustrates a schematic use case of technology for extracting environmental features from image data, particularly for autonomous driving. [Figure 10B] FIG. 1 shows a schematic diagram of a use case of the technique of extracting environmental features from image data, particularly for robot motion planning. [Figure 10C] FIG. 1 illustrates a schematic use case of the technique for extracting environmental features from image data, particularly for access control. DETAILED DESCRIPTION OF THE INVENTION

[0082] Detailed Description FIG. 1 shows a schematic diagram of a computer-implemented method 100 for extracting environmental features from image data.

[0083] The method 100 includes a step S102 of receiving data indicative of a visual domain of an environment. The method 100 further includes a step S104 of generating a visual domain text prompt based on the received data S102 indicative of the visual domain of the environment. In a step S106, image data representing the environment is received. The method further includes a step S108 of extracting (particularly local) features of the environment from the image data received in step S106. The extraction of (particularly local) features S108 is performed by a conditional feature extraction model. The extraction of (particularly local) features S108 is conditioned by the visual domain text prompt generated in S104.

[0084] Optionally, in step S110, a visual recognition task is performed on the environment based on the extracted S108 (especially local) features.

[0085] Further optionally, the method 100 includes a step S101 of receiving data indicative of an environment class for the environment, and a step S103 of generating a text prompt based on the data indicative of the class for the environment received in step S101. The extraction S108 of (particularly local) features of the environment from the image data received in step S106 can further be conditioned by the environment class text prompt generated in step S103.

[0086] FIG. 2 shows a schematic diagram of a computer-implemented method 200 for training a conditional feature extraction model for extracting environmental (particularly local) features from image data conditioned by visual-domain text prompts.

[0087] Method 200 includes step S202 of receiving a training data set. The training data set includes a visual domain of an environment, image data of the environment, and visual domain text prompts that indicate (especially local) features of the environment. Method 200 further includes step S204 of training a conditional feature extraction model based on the training data set received in step S202. Step S204 of training the conditional feature extraction model includes receiving the image data of the environment as input, receiving the visual domain text prompts as conditions, and receiving the (especially local) features as ground truth.

[0088] FIG. 3 shows a schematic architecture of a computing device 300 for extracting environmental features from image data.

[0089] The computing device 300 includes a visual domain instruction receiving interface 302 configured to receive data indicative of a visual domain of an environment. The computing device 300 further includes a visual domain text prompt generation module 304 configured to generate a visual domain text prompt based on the received data indicative of the visual domain of the environment. The computing device 300 further includes an environment image data receiving interface 306 configured to receive image data representing an environment. The computing device 300 further includes a conditional feature extraction model 308 configured to extract (particularly local) features of the environment from the received image data. The extraction of the (particularly local) features is conditioned by the generated visual domain text prompt.

[0090] Optionally, the computing device 300 may comprise a visual recognition task execution module 310 configured to execute a visual recognition task on the environment based on the extracted (especially local) features.

[0091] Further optionally, the computing device 300 may comprise an environment class receiving interface 301 configured to receive data indicative of a class related to the environment. The computing device may further comprise an environment class text prompt generating module 303 configured to generate an environment class text prompt based on the received data indicative of a class related to the environment. The extraction of (particularly local) features of the environment from the received image data may further be conditioned by the generated environment class text input prompt.

[0092] Any one of the visual domain instruction receiving interface 302, the optional environment class receiving interface 301, and the environment image data receiving interface 306 may be embodied by the input / output interface 312. Alternatively or additionally, any one of the visual domain text prompt generation module 304, the optional environment class text prompt generation module 303, the conditional feature extraction model 308, and the optional visual recognition task execution module 310 may be embodied by the processing unit 314. Further alternatively or additionally, the computing device 300 may include at least one memory 316.

[0093] Computing device 300 is configurable to perform any one of the steps and / or include any one of the features disclosed in connection with method 100.

[0094] FIG. 4 shows a schematic architecture of a computing device 400 for training a conditional feature extraction model for extracting environmental (particularly local) features from image data conditioned by visual domain text prompts.

[0095] The computing device 400 comprises a training data receiving interface 402 configured to receive a training data set. The training data set includes a visual domain of an environment, image data of the environment, and visual domain text prompts indicating features of the environment, particularly local features. The computing device 400 further comprises a training module 404 configured to train a conditional feature extraction model based on the received training data set. Training the conditional feature extraction model includes receiving the image data of the environment as input, receiving the visual domain text prompts as conditions, and receiving the (particularly local) features as ground truth.

[0096] The training data receiving interface 402 may be embodied by an input / output interface 406. Alternatively or additionally, the training module 404 may be embodied by a processing unit. Further alternatively or additionally, the computing device 400 may include at least one memory 410.

[0097] Computing device 400 is configurable to perform any one of the steps and / or include any one of the features disclosed in connection with method 200.

[0098] Either of the processing units 314; 408 may comprise a central processing unit (CPU) and / or a graphics processing unit (GPU).

[0099] Computing device 300 and computing device 400 may be the same.

[0100] The techniques for extracting environmental features from image data and for training conditional feature extraction models for extracting environmental (especially local) features from image data provide novel approaches to utilizing large-scale text-to-image models (especially conditional feature extraction models) for domain-adaptive visual recognition tasks, such as semantic segmentation and / or object detection.

[0101] Previous work on VPD using trained SD and trained task-specific decoders has shown promising results when tested on the same training domain. Techniques for extracting environmental features from image data and training conditional feature extraction models to extract environmental (especially local) features from image data go one step further, aiming to use models for a variety of unobserved domains by leveraging generalizable representations of conditional feature extraction models (and / or large-scale pre-trained models) and a unified visual-linguistic embedding space. Text prompts (especially visual domains and / or environment classes) are utilized, for example, as domain indicators to adapt the model to various domains. The conditional feature extraction model (as a text-to-image diffusion model) can be conditioned (e.g., naturally) on text. Additionally, the task-specific decoder can be conditioned on the domain embedding to further strengthen this conditioning, as shown schematically in Figure 8.

[0102] For example, SD not only captures rich semantic information, but also may observe many visual domains, e.g., various weather and / or lightning conditions, allowing its representation to be more generalized. Unlike VPD, which potentially overfits to a training set, a generalizable representation (e.g., of SD) can be exploited according to the techniques used for domain adaptation and / or generalization tasks, extracting environmental features from image data and training a conditional feature extraction model for extracting environmental (e.g., local) features from image data. For example, SD can be fine-tuned on a training set while maintaining the ability to adapt to different test sets. For example, a conditional feature extraction model can be fine-tuned on sunny images and still have reasonable performance not only in the trained domain (e.g., sunny conditions) but also on unobserved data, e.g., image data of rainy and / or foggy conditions.

[0103] It should be noted that traditional VPD directly fine-tunes the entire denoising model, as shown in Figure 6, which may destroy prior knowledge.

[0104] Alternatively, one can employ a ControlNet fine-tuning scheme that freezes the original SD and adds trainable branches while maximally preserving prior knowledge, as shown schematically in Figure 7. Furthermore, the techniques for extracting environmental features from image data and for training conditional feature extraction models for extracting environmental (especially local) features from image data leverage the unified visual-text embedding space of the SD, particularly the use of text prompts (e.g., visual domain and / or environmental class) as domain indicators to conveniently condition the conditional feature extraction models to different domains.

[0105] Techniques for extracting environmental features from image data and training conditional feature extraction models for extracting environmental (especially local) features from image data are more robust and generalizable to various (especially unobserved) visual domains, such as different weather and / or lightning conditions and / or new countries. Traditionally, networks tend to overfit to training sets (e.g., sunny images) and are unable to generalize to unobserved domains (e.g., rainy and / or foggy scenes). Because real-world environments are diverse and changing, and conditional feature extraction models can only be trained based on limited data, it is important to design robust conditional feature extraction models that can handle unobserved conditions. This is crucial for real-world applications such as AD (especially autonomous driving) and / or robotics. For example, if a conditional feature extraction model trained on sunny images cannot generalize to nighttime scenes, a person walking on a road at night may be ignored by an autonomously driving vehicle, leading to an accident.

[0106] In contrast, techniques for extracting environmental features from image data can be flexibly adapted to different domains with the assistance of text prompts (e.g., visual domains and / or environment classes). Alternatively or additionally, conditional feature extraction models can be built on large pre-trained models (e.g., SD), which can transfer rich prior knowledge to enable faster convergence of training.

[0107] Techniques for extracting environmental features from image data and for training conditional feature extraction models for extracting environmental (especially local) features from image data can be used to analyze data obtained from sensors that can determine measurements of the environment in the form of sensor signals that can be provided by (especially digital) image data including, for example, video images, radar images, LiDAR images, ultrasound images, motion images, and / or thermal images.

[0108] Techniques for extracting environmental features from image data and for training conditional feature extraction models to extract environmental (especially local) features from image data can be used to classify sensor data, detect the presence of objects in the sensor data, e.g., traffic signs, road surfaces, pedestrians, and / or perform semantic segmentation for object classes that may be present in a semantic segmentation task, e.g., trees and / or sky.

[0109] Alternatively or additionally, techniques for extracting environmental features from image data and training conditional feature extraction models for extracting environmental (especially local) features from image data can be used to identify one or more continuous values ​​(e.g., perform regression analysis) for distance, velocity, acceleration, and / or track items, e.g., objects, within the data. Identification of one or more continuous values ​​can be performed based on low-level features (e.g., image edges and / or pixel attributes).

[0110] Alternatively or additionally, the techniques for extracting environmental features from image data and for training conditional feature extraction models for extracting environmental (especially local) features from image data can be used to detect anomalies in (especially downstream) technical systems (e.g., AD vehicles, robots, domestic devices, and / or access control systems). This can be done by analyzing data (e.g., scalar time series; also described as measurement and control) from sensors and then operating the technical system accordingly. Alternatively or additionally, it is also possible that an anomaly is detected and then the technical system is operated in a fail-safe mode.

[0111] The methods / data for training, i.e., techniques for extracting environmental features from image data and techniques for training conditional feature extraction models to extract environmental (especially local) features from image data, can be considered upstream parts in a machine learning (ML) toolchain. Once trained, the conditional feature extraction models in ML systems can be used downstream, for example, for AD, robot motion planning, the operation of home appliances, and / or the control of access control systems.

[0112] According to one embodiment, the techniques for extracting environmental features from image data and for training a conditional feature extraction model to extract environmental (particularly local) features from image data not only use SD for the domain it was trained on, but also retain its generalization ability so that it can be used in unobserved domains. The unified visual-linguistic embedding space of SD allows the conditional feature extraction model to adapt (e.g., to different domains) using text prompts (particularly visual domains and / or environmental classes).

[0113] As already mentioned above, instead of operating in the image space, SD operates in the latent space of an autoencoder, as shown diagrammatically in Figure 5. First, the encoder E maps a given image x to a spatial latent code z = E(x). Then, the decoder D maps z back to image space. The autoencoder maps a given image, i.e.

number

[0114] In the second stage, a diffusion model is trained in the latent space Z. The diffusion model includes a forward diffusion process and a backward denoising process. The forward pass includes a Markov chain to gradually add Gaussian noise to the clean data. Formally, this can be expressed as, for example,

number

number

[0115] A noisy latent state can be, for example,

number

number

number

[0116] A more mathematical derivation is presented in Jonathan Ho et al. in "Denoising Diffusion Probabilistic Models", arXiv:2006.11239v2 [cs.LG], which is incorporated herein by reference. Conditional information, such as text prompts (particularly visual domain and / or environment class), is inserted via a Cross-Attention mechanism, shown as "CLIPtext" at the bottom of Figure 5.

[0117] During inference, the latent representation z is calculated from the Gaussian distribution. T can be randomly sampled, and then the trained denoising UNet can be successively used to denoise z from t=T to t=1. t Given a denoised latent state z t-1 The final synthesized image can be obtained by feeding the clean latent state z0 through the decoder D.

[0118] VPD, as shown schematically in Figure 6 by the flame symbols, fine-tunes a pre-trained SD UNet for downstream visual recognition tasks, such as semantic segmentation and / or depth estimation. As shown schematically in Figure 6, VPD extracts features and cross-attention maps from the UNet and feeds them into a task-specific decoder. For semantic segmentation tasks, class embeddings can be extracted from a frozen CLIP text encoder (as shown by the snowflake symbols), adapted by a trainable text adaptor (as shown by the flame symbols, e.g., several MLP layers), and then made available to the SD UNet as text prompts (e.g., domain and / or class) (as well as text conditions). Note that VPD focuses only on intra-domain capabilities. For example, the test set is assumed to share the same distribution as the training set, and large models tend to overfit to the specific trained domain.

[0119] In contrast, a technique for extracting environmental features from image data and a technique for training a conditional feature extraction model for extracting environmental (especially local) features from image data allows for the design of a conditional feature extraction model that is more robust to unobserved domains and / or unobserved environments, taking into account the generalization ability of the conditional feature extraction model.

[0120] The ControlNet augments the SD with additional input conditions, e.g., label maps. As shown schematically in Figure 7, the ControlNet clones a trainable copy of the UNet encoder, and the original SD is frozen during fine-tuning (as indicated by the flame and snowflake symbols, respectively). The input conditions are fed as inputs to the ControlNet, and features are fed back to the SD decoder after a zero convolutional layer and trainable encoder, where the zero convolutional layer is effectively a 1x1 convolutional layer with both weights and biases initialized as 0.

[0121] The goal of training the ControlNet plus SDUNet architecture is to insert the condition y from Eq. (2) as

number

[0122] The techniques for extracting environmental features from image data and for training a conditional feature extraction model to extract environmental (particularly local) features from image data utilize the conditional feature extraction model as a large-scale text-to-image generation model (e.g., including SD) for visual recognition tasks (e.g., semantic segmentation and / or object detection) while preserving the generalization ability of the conditional feature extraction model in various (particularly unobserved) domains and / or environments. The key idea is to utilize the unified visual-linguistic latent space of the large-scale model and design the conditional feature extraction model to flexibly adapt based on text prompts (including text conditions) indicating visual domains and / or environmental classes (e.g., visual domains and / or environmental classes).

[0123] As shown at 802 and 804, given data indicative of a class for an environment and / or data indicative of the visual domain of the environment (e.g., including an image), the operating visual domain and / or environment class to which the environment belongs is determined. This can be done manually or automatically. If manual, the operating "visual domain" and / or environment class can be specified directly by a human (e.g., via a UI, particularly a GUI).

[0124] Alternatively or additionally, some datasets, such as the Adversarial Conditions Dataset (ACDC, available at https: / / acdc.vision.ee.ethz.ch / ), already contain meta-information about weather conditions (e.g., rain, fog, snow, and / or nighttime). In downstream (and / or real-world) applications such as AD (especially autonomous driving), humans can dynamically provide such information to the model.

[0125] For automatic identification of the operational visual domain and / or environment class, a pre-trained visual question answering (VQA) model, such as mPlug, can be utilized (e.g., as described in Chenliang Li et al., “mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections,” arXiv:2205.12005v2 [cs.CL], incorporated herein by reference). FIGS. 9A and 9B show an example in which the model asks, “What is the weather like?” based on an image 804 representing the visual domain. The model returns a top response 812 with a likelihood. For example, in FIG. 9A, “sunny” and “cloudy” are the two most likely responses with roughly equal probability, while in FIG. 9B, “cloudy” is the response with a much higher probability.

[0126] Similarly, questions may be asked, for example, regarding lightning conditions and / or countries. The top 1 response may be received as a (e.g., intermediate) visual domain prompt input and / or data indicative of the visual domain (e.g., as in the example of FIG. 9B). Alternatively or additionally, the concept of "visual domain" may be ambiguous (e.g., with approximately equal probability, as illustrated in FIG. 9A), so the top K responses may be captured with probability (e.g., K=2). When the (e.g., intermediate) visual domain prompt input and / or data indicative of the visual domain are provided, a frozen CLIP encoder 806 and a (particularly lightweight) trainable text adapter 808 may be used to obtain a visual domain text prompt (also described as a domain embedding Wi) at 812, e.g., Wi=ψ adp (CLIP text (y i )),i=1,…,K (4) where y i represents a visual domain prompt, e.g., "sunny scene," and CLIP text and ψ adp are the CLIP text encoder and text adaptor, respectively.

[0127] As shown schematically in FIG. 8, the conditional feature extraction model can further be conditioned on an environment class text prompt (also written as class embedding c) 810, which can be obtained from data indicative of classes related to the environment 802 using a separate frozen CLIP encoder 806 and a separate (particularly lightweight) trainable text adapter 808.

[0128] The visual domain text prompts (and / or domain embeddings) 812 can be used to condition a conditional feature extraction model, illustratively comprising a frozen SDUNet 822 and a ControlNet 814 with a trainable copy 816 of a UNet encoder (e.g., of the SD UNet 822) followed by zero convolutions 818. As an alternative option to directly fine-tuning the entire UNet as in traditional VPD, the example of Figure 8 provides a fine-tuning scheme for the ControlNet 814 that keeps the original UNet 822 frozen and adds a trainable copy 816 of the UNet encoder initialized using the SD UNet encoder's pre-trained weights. In this way, prior knowledge of the SD is fully preserved.

[0129] The frozen SDUNet 822 and the trainable copy of the UNet encoder 816 are both configured to receive the latent representations 820.

[0130] Of course, the optional environmental class text prompts 810 (also called text conditions) can be used in a similar manner to the conventional SD of Figure 8, which can be utilized via Cross-Attention. The visual domain text prompts 812 (also called domain embeddings) in the exemplary embodiment of Figure 8 are fed to the trainable UNet encoder 816 of the ControlNet 814 using a Cross-Attention mechanism.

[0131] As shown schematically at 826, features and cross-attention maps can be extracted in a similar manner to VPD. Alternatively or additionally, instead of extracting from a single clean latent state z0, features can be extracted from multiple time steps. Based on equation (1), noisy returns can be obtained directly at different time steps. The feature extraction step can be performed, for example, by: f i,t =SD(W i ,z t ,c),fi =Agg(f i,t ),f=Σ i f i *p i (5) where SD represents stable diffusion, c is the class embedding (and / or environmental class text prompt) 810 for the task, e.g., semantic segmentation. Agg is an aggregation operation, e.g., the sum of the features f at different time steps. i,t Shows the simple average over p i is the current domain embedding W i (and / or visual domain text prompts) 812 probability.

[0132] Additionally, the visual domain text prompts (and / or domain embeddings) 812 are available in the task-specific decoder 828. The domain embeddings (visual domain text prompts) 812 can first be aggregated, as shown at 824: W agg =Agg(W i ,p i ) (6) where Agg is an aggregation operation, e.g., a simple probability average, and / or a learnable mapping. As noted in previous studies, e.g., Seonguk et al. in “Learning to Optimize Domain Specific Normalization for Domain Generalization”, DOI: 10.1007 / 978-3-030-58542-6_5, normalization statistics can have a significant impact on the generalization ability of visual domains (and / or environment classes). Therefore, according to one embodiment, domain embeddings (and / or visual domain text prompts) 812 are used to map these embeddings to normalization parameters, e.g.,

number

[0133] 10A, a first example use case is shown schematically in which a controller 1004 of an AD vehicle 1002-1 receives results of feature extraction (e.g., via step S108) and / or results of a visual recognition task (e.g., via step S110) to perform an AD function. Optionally, the received results may be displayed to the driver and / or passengers, for example on a display 1006 in the center console.

[0134] FIG. 10B illustrates a schematic example of a second use case in which the controller 1004 of the robot 1002-2 receives the results of the feature extraction (e.g., by step S108) and / or the results of the visual recognition task (e.g., by step S110) to plan the movement of the robot 1002.

[0135] 10C shows a schematic diagram of a third example use case in which an access control system 1002-3 includes a first sensor 1010 that acquires image data representative of the environment, an optional second sensor 1012 that acquires audio data representative of the environment, a controller 1004, and an optional display (particularly for the person requesting access) 1006. The controller 1004 receives the results of the feature extraction (e.g., via step S108) and / or the results of the visual recognition task (e.g., via step S110) and determines whether access is granted.

[0136] Cited references [1]R. Rombach et al., “High-Resolution Image Synthesis with Latent Diffusion Models”, arXiv:2112.10752v2 [cs.CV] [2]Wenliang Zhao et al., “Unleashing Text-to-Image Diffusion Models for Visual Perception, arXiv:2303.02153 v1 [cs.CV] [3]P. Esser et al., “Taming Transformers for High-Resolution Image Synthesis”, arXiv:2012.09841v3 [cs.CV] [4]Yong Zheng Ong et al., “VG-GAN: Conditional GAN Framework for Graphical Design Generation”, 2022 IEEE International Conference on Image Processing (ICIP), Bordeaux, France, 2022, pp. 781-785, doi: 10.1109 / ICIP46576.2022.9897935 [5]Dan Zhang et al., “Anomaly-Aware Semantic Segmentation via Style-Aligned OoD Augmentation”, arXiv:2308.09965v1 [cs.CV] [6]Simian Luo et al., “Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference”, arXiv:2310.04378v1 [cs.CV] [7]https: / / www.cityscapes-dataset.com / [8]Jonathan Ho et al., “Denoising Diffusion Probabilistic Models”, arXiv:2006.11239v2 [cs.LG] [9]https: / / acdc.vision.ee.ethz.ch /

[10] Chenliang Li et al., “mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections”, arXiv:2205.12005v2 [cs.CL]

[11] Seonguk Seo et al., “Learning to Optimized Domain Specific Normalization for Domain Generalization” (2020) 10.1007 / 978-3-030-58542-6_5.

Claims

1. A computer-implemented method (100) for extracting environmental features from image data, comprising: The method comprises: - receiving data representative of the visual domain of the environment (S102); - generating (S104) a visual domain text prompt based on the received (S102) data indicative of the visual domain of the environment; - receiving image data representative of said environment (S106); - a step (S108) of extracting features, in particular local features, of the environment from the received (S106) image data, wherein the extraction of features, in particular local features (S108) is performed by a conditional feature extraction model, and further wherein the extraction of features, in particular local features (S108) is conditioned by the visual domain text prompt generated (S104); A method (100) comprising:

2. The method (100) further comprises: - performing (S110) a visual recognition task on the received image data based on the features, in particular local features, extracted (S108); Including, The method (100) of claim 1.

3. The visual recognition task comprises: - depth estimation, - object detection, - Classification, and - Semantic segmentation, at least one of: The method (100) of claim 2.

4. performing the visual recognition task on the image data (S110) is conditioned on the visual domain text prompt generated (S104); The method (100) of claim 2 or 3.

5. The received (S102) data indicative of the visual domain, - environmental sensor data, - location data determined by positioning systems, in particular satellite navigation systems; - manually input data, and - electronically available information data; at least one of: The method (100) of any one of claims 1 to 4.

6. The method (100) further comprises: receiving data indicating a class relating to the environment (S101); generating (S103) an environment class text prompt based on the received (S101) data indicating the class for the environment; Including, Extraction (S108) of features of the environment, in particular local features, from the received (S106) image data is further conditioned by the generated (S103) environment class text prompt. The method (100) of any one of claims 1 to 5.

7. The feature extraction (S108) is further conditioned by the environment class text prompt. The method (100) of claim 6.

8. The step of generating the visual domain text prompts (S104) and / or the step of generating the environmental class text prompts (S103) are performed by an adaptive Contrastive Language-Image Pretraining (CLIP) encoder. The method (100) of any one of claims 1 to 7.

9. the conditional feature extraction model includes a generative image-to-feature model configured to extract features of the environment, in particular local features, from the received (S106) image data of the environment, and a conditioning model configured to encode the visual domain text prompts generated (S104) to condition and / or control the generative image-to-feature model. The method (100) of any one of claims 1 to 8.

10. The generative image-to-feature model includes a diffusion model, in particular a stable diffusion (SD) model, and the SD network includes a UNet architecture having an encoder and, in particular, a skip-connected decoder.

10. The method (100) of claim 9.

11. The conditioning model includes a ControlNet, which includes an encoder and a convolutional layer having a cross-attention mechanism to the generator image-to-feature model, particularly the SD network's encoder. The method (100) according to claim 9 or 10.

12. 1. A computer-implemented method (200) for training a conditional feature extraction model for extracting environmental features, particularly local features, from image data conditioned by visual domain text prompts, comprising: The method comprises: - receiving (S202) a training data set comprising a visual domain of an environment, image data of said environment and visual domain text prompts indicating features of said environment, in particular local features; - training (S204) a conditional feature extraction model based on the received (S202) training data set, wherein training the conditional feature extraction model comprises receiving the image data of the environment as input, receiving the visual domain text prompts as conditions, and receiving features, in particular local features, as ground truth; A computer-implemented method (200) comprising:

13. Use of the method (100) according to any one of claims 1 to 11 or the method (200) according to claim 12, -Autonomous driving AD, - Robot motion planning, - operating domestic devices, and - Control of access control systems Use for at least one of the following:

14. A computing device (300) for extracting environmental features from image data, comprising: a visual domain indication receiving interface (302) adapted to receive data indicative of a visual domain of an environment; a visual domain text prompt generation module (304) configured to generate a visual domain text prompt based on the received data indicative of the visual domain of the environment; an environment image data receiving interface (306) configured to receive image data representative of said environment; a conditional feature extraction model (308) configured to extract features, in particular local features, of the environment from the received image data, wherein the extraction of the features, in particular local features, is conditioned by the generated visual domain text prompt; and A computing device (300) comprising:

15. 1. A computing device (400) for training a conditional feature extraction model for extracting environmental features, in particular local features, from image data conditioned by visual domain text prompts, the device comprising: a training data receiving interface (402) configured to receive a training data set, said training data set comprising a visual domain of an environment, image data of the environment and visual domain text prompts indicating features of the environment, in particular local features; a training module (404) configured to train a conditional feature extraction model based on the received training data set, wherein training the conditional feature extraction model includes receiving the image data of the environment as an input, receiving the visual domain text prompts as conditions, and receiving the features, in particular local features, as ground truth; and A computing device (400) comprising:

Images