Techniques for extracting environmental features from image data

Abstract

The invention relates to techniques for extracting environmental features from image data. A technique for extracting environmental features from image data is provided. A computer-implemented method includes a step of receiving data indicative of a visual domain (804) of an environment and a step of generating a visual domain textual cue (812) based on the received data indicative of the visual domain (804) of the environment. The method further includes the step of receiving image data representing the environment. The method further comprises a step of extracting environmental features, in particular local features (826), from the received image data. The extraction of features, in particular local features, is performed by a conditional feature extraction model (814, 822). The extraction (826) of features, in particular local features, is conditioned by the generated visual domain text cues (812).

Description

Technical Field

[0001] Provided are a technique for extracting environmental features from image data, and a technique for training a conditional feature extraction model for extracting environmental (particularly local) features from image data. These techniques include methods, computing devices, systems, computer program products, and computer-readable storage media. Background Art

[0002] Recent large-scale text-to-image generation models trained on large amounts of text-image pairs, such as Stable Diffusion (SD, as described by R. Rombach et al. in “High-Resolution Image Synthesis with Latent Diffusion Models”, arXiv:2112.10752v2[cs.CV], which is incorporated herein by reference), have demonstrated impressive performance and ability to generate a wide variety of diverse and creative images given text prompts. In particular, SD is a state-of-the-art open-source visual language generation model trained on billions of text-image pairs. SD is a latent diffusion model, which is a special case of a diffusion model trained in some latent space instead of the original image space. More specifically, according to P. Esser et al. in “Taming Transformers for High-Resolution Image Synthesis”, arXiv:2012.09841v3[cs.CV] (also referred to as VQ-GAN for short), which is incorporated herein by reference and as Figure 5 As schematically illustrated in , SD first trains an autoencoder where an image is encoded into a latent space Z and decoded back to reconstruct a given image. In the second stage, a diffusion model is trained in this latent space Z. Since it is trained on a large amount of data, SD has learned rich representations.

[0003] The latest work in W. Zhao et al., "Unleashing Text-to-Image Diffusion Models for Visual Perception (VPD)", arXiv:2303.02153v1 [cs.CV] (which is incorporated herein by reference) builds on SD and leverages the pre-trained knowledge of denoising UNet to provide semantic guidance in various visual perception tasks (e.g., semantic segmentation and depth estimation). W. Zhao et al. explored fine-tuning SD and simultaneously training task-specific decoders for various visual perception tasks (e.g., semantic segmentation, depth estimation), such as Figure 6 Schematically shown in .

[0004] However, SD is fine-tuned on a specific training set by W. Zhao et al. and is only evaluated on the same training domain at inference time. Summary of the Invention

[0005] In the following, techniques for extracting environmental features from image data and for training conditional feature extraction models for extracting environmental (particularly local) features from image data are described with respect to the claimed method and with respect to the claimed computing device. Features, advantages or alternative embodiments herein may be assigned to other claimed objects (e.g., a system comprising a computing device, a computer program or a computer program product), and vice versa. In other words, claims directed to a computing device may be improved with features described or claimed in the context of a method. In this case, the functional features of the method are respectively implemented by structural units of the system, and vice versa.

[0006] Regarding a first methodological aspect, a computer-implemented method for extracting environmental features from image data is provided. The method comprises the step of receiving data indicating a visual domain of an environment. The method further comprises the step of generating a visual domain textual cue based on the received data indicating the visual domain of the environment. The method further comprises the step of receiving image data representing the environment. The method further comprises the step of extracting (particularly local) features of the environment from the received image data. The extraction of the (particularly local) features is performed by a conditional feature extraction model and is conditioned on the generated visual domain textual cue.

[0007] The method can be used to provide a model for image generation that can flexibly adapt based on visual domain textual cues that indicate the visual domain of the environment.

[0008] By computer-implemented technology, it is possible to perform environmental feature extraction from received image data, and preferably perform visual perception tasks based on the extracted features, which is more robust and generalizable to various visual domains (e.g., different weather conditions, lighting conditions and / or new countries). Alternatively or in addition, overfitting of a training data set (e.g., including and / or based on extracted features conditioned on visual domain textual cues, in particular as a reference truth) used to train a model (e.g., a neural network NN and / or artificial intelligence AI) for performing feature extraction and / or visual perception tasks can be avoided. For example, accurate results of feature extraction and / or visual perception tasks (in particular, object detection, depth estimation, classification and / or semantic segmentation) may be critical for safe autonomous driving (AD) and / or safe operation of robots (e.g., in manufacturing environments and / or home automation).

[0009] AD driving functions may include full automation (also designated as autonomous driving, in particular L5), at least partial (e.g. high (L4), conditional (L3) or partial (L2)) automation and / or assisted driving automation (e.g. L1, e.g. including adaptive cruise control ACC).

[0010] Any text prompt may also be labeled as a text prompt and / or a speech prompt. Alternatively or additionally, the text prompt may be provided and / or processed in a text-based and / or acoustic-based format.

[0011] Visual domain textual cues (also: domain indicators) can improve the (particularly visual) domain flexibility of feature extraction and / or visual perception tasks (e.g., by using knowledge about expected under- or overexposure).

[0012] (e.g., pre-trained) models for performing feature extraction and / or visual perception tasks, which are trained on training datasets (in particular, including feature extraction results conditioned by visual domain textual cues and / or visual perception task results based on feature extraction), which can be significantly improved, in particular by enabling the transfer of (in particular, rich) prior knowledge and / or fast training convergence.

[0013] The environment may also be indicated as a (particularly physical, real and / or natural) scene, setting and / or atmosphere. The environment represented by the received image data may comprise the environment of a (e.g. AD) vehicle, a robot, a monitoring system and / or a person. In particular, the environment may represent a traffic scene. Alternatively or in addition, the environment may be real, natural, rural, built, urban and / or spatial. For example, the environment of a (e.g. AD) vehicle may comprise a road layout, the environment of a robot may comprise a manufacturing site, and the environment of a monitoring system and / or a person may comprise a building structure. For example, for AD applications, the environment may comprise an external environment, in particular a street scene. Further exemplarily, for a robot, the environment may comprise a factory site with interior and / or exterior areas.

[0014] The image data representing the environment may be received as digital (and / or analog) data, in particular from a (e.g. video) camera. Alternatively or in addition, the image data representing the environment may be acquired by means of a (e.g. video and / or thermal) camera, radar, lidar, ultrasonic sensor and / or motion sensor.

[0015] The visual domain may also be indicated as visibility conditions, optical conditions, and / or may include an indication of the visual (also: optical) settings of the environment (e.g., taking into account lighting and / or weather conditions). Alternatively or in addition, the visual domain may be generated based on a geographic identifier and / or may be particularly country-specific, such as based on lighting conditions relative to local sunrise and sunset times at the time the image data was acquired, traffic signs and / or street layout of the exterior environment, and / or natural and / or artificial lighting types of the interior environment. Lighting conditions (short: lighting) may alternatively be indicated as illuminance and / or brightness.

[0016] The data indicative of the visual domain may comprise digital data, such as the result of a brightness determination of the imaging data. Alternatively or additionally, the digital data may be converted according to (e.g., manual) input (e.g., selected brightness and / or weather conditions) and / or may be compressed, for example, according to one of a plurality of predetermined visual domains selected (e.g., arranged in a table) (in particular manually and / or automatically).

[0017] The step of generating a visual domain text prompt can be performed at least in part by a contrastive language image pre-training (CLIP) encoder. In some embodiments, a pre-trained visual question answering (VQA) model, such as mPLUG, can first receive data indicating the visual domain of the environment and provide a result, in particular, a probability of a visual domain class in a discrete set of visual domain classes. In one embodiment, the visual domain class with the highest probability can be selected as the visual domain for which the visual domain text prompt is generated. In an alternative embodiment, the visual domain for which the visual domain text prompt is generated can correspond to a weighted combination of K visual domain classes with the K highest probabilities (e.g., K=2 corresponds to the two highest probabilities).

[0018] Conditional feature extraction models can include generative models and / or (especially large-scale) text-to-image generation artificial intelligence (AI).

[0019] The extracted features may locally represent the environment.Alternatively or additionally, the extracted features may include objects, shapes and / or colors.

[0020] Since the feature extraction process is conditioned on textual cues indicating the visual domain, for example, location and / or country information may be relevant to the extracted features. For example, some local traffic signs may vary depending on the location, and / or left-hand side or right-hand side driving may be used as prior knowledge for model predictions. Feature extraction can, for example, be additionally conditioned on context class textual cues, where the context class relates to, for example, country (e.g., for typical road layouts, and / or the positioning and / or appearance of traffic signs and / or traffic lights), rural environments, and / or urban environments (e.g., taking into account typical planting and / or architecture).

[0021] The (particularly local) extracted features can be supplemented by a cross-attention map. The cross-attention map can associate different patches of (particularly local) extracted features. Alternatively or in addition, the cross-attention map can promote consistency across environments included in the image data, especially as a result of subsequently performed visual perception tasks.

[0022] The extracted environment features can be provided, optionally supplemented with cross-attention maps for output.

[0023] The method may further comprise the step of performing a visual perception task on the received image data representing the environment based on the extracted (particularly local) features. The result of the performed visual perception task may comprise a (e.g., weighted) average of the features (particularly at different time steps). Alternatively or additionally, the result of the performed visual perception task may be provided and / or output.

[0024] Visual perception tasks can be performed by task-specific decoders.

[0025] By performing visual perception tasks, the environment captured in the image data can be analyzed in a sufficiently accurate manner to enable, for example, AD, the use of autonomous robots, and / or access control.

[0026] Visual perception tasks can include depth estimation, object detection, classification, and / or semantic segmentation. Therefore, the method can be used for image classification. Image classification can include object detection and segmentation, particularly semantic segmentation. Classification can be based on low-level features of the image data, such as pixels, edges, and / or corners.

[0027] By means of depth estimation, distance estimation (e.g., distance to obstacles for (in particular AD) vehicles and / or (in particular autonomous) robots) can be performed. Alternatively or additionally, by object detection and / or by classification, it is possible to detect paths, road surfaces, traffic signs, pedestrians, vehicles, (e.g., further) obstacles (e.g., trees), the sky and / or the destination. Further alternatively or additionally, performing classification allows, for example, traffic signs and / or obstacles to be recognized and paths (and / or trajectories) to be planned accordingly. Further alternatively or additionally, by means of semantic segmentation, the (e.g., complete) semantic content of the environment can be provided and the accuracy of the path (and / or trajectory) planning of (e.g., AD) vehicles and / or (e.g., autonomous) robots can be improved, in particular depending on the real-time situation of the environment.

[0028] Alternatively or additionally, the visual perception task may comprise, for example, performing a regression analysis to determine (in particular continuous) values (such as distance (e.g. based on depth estimation), speed (also known as: velocity), acceleration) and / or tracking objects and / or persons, in particular by taking into account edge and / or pixel properties of image data, and / or by analyzing (in particular image) data (e.g. scalar time series) for performing corresponding measurements (e.g. speed and / or acceleration) and / or for tracking.

[0029] By performing visual perception tasks, anomaly detection can be provided and, optionally, control of, for example, a (particularly AD) vehicle and / or a (particularly autonomous) robot in a fail-safe mode after anomaly detection.

[0030] For example, in one embodiment applied to AD, anomalies may occur on the road, such as described in Dan Zhang et al., "Anomaly-Aware Semantic Segmentation via Style-Aligned OoD Augmentation," arXiv:2308.09965v1 [cs.CV], which is incorporated herein by reference. For example, in the event of an accident, a text prompt (e.g., an environmental context) may indicate "potential debris on the road."

[0031] Performing visual perception tasks on image data can be conditioned on generated visual domain textual cues.

[0032] According to some embodiments, (particularly local) feature extraction may be performed for multiple time steps, particularly in connection with generating latent image data with varying noise levels (short for noisy latents). Visual perception tasks may be performed based on the combined (and / or aggregated and / or weighted) extracted features from multiple time steps.

[0033] Alternatively or additionally, conditioning the generated visual-domain textual cues may include conditioning the combined (and / or aggregated and / or weighted) visual-domain textual cues associated with a plurality of time steps, particularly in relation to generating the latent image data.

[0034] The received data indicative of the visual field may comprise environmental sensor data, positioning data determined by means of a positioning system, in particular by means of a satellite navigation system, manually entered data and / or electronically available information data.

[0035] Environmental sensor data can be acquired with the aid of environmental sensors. Environmental sensors can include optical sensors, acoustic sensors, rain sensors, humidity sensors, thermal sensors, temperature sensors, wind sensors, radar sensors, LiDAR sensors, ultrasonic sensors and / or motion sensors. Alternatively or in addition, environmental sensors can be mounted on (e.g., AD) vehicles, (e.g., autonomous) robots, and / or can be stationary in the environment. With the aid of environmental sensors, information about the visual domain, such as weather conditions (e.g., sunny, rainy and / or foggy, in particular using humidity sensors) and / or lighting conditions (e.g., daytime, nighttime, dawn and / or dusk) can be advantageously determined locally (and / or in) the environment in real time.

[0036] The positioning system, in particular a satellite navigation system, may comprise a global navigation satellite system (GNSS), such as the Global Positioning System (GPS) and / or Galileo. Alternatively or in addition, the positioning system may comprise a regional system (e.g., in particular using radio signals), a local positioning system (LPS, for example, beacons using radio signals and / or optical signals, Wi-Fi and / or Li-Fi) and / or an indoor system (e.g., using ultrasound and / or multilateration, such as Active Bat). By means of the positioning system, information about the visual domain, such as the type of terrain, building development and / or fixed installations, may advantageously be determined for the environment (and thus, for example, information about lighting settings, such as dimming under bridges and / or in tunnels).

[0037] The manual input data may be received from a user (also referred to as: a human operator, e.g., in particular a driver of an AD vehicle and / or an operator of a robot) by means of a user interface (UI), in particular a graphical user interface (GUI), and / or by means of a microphone (and / or an acoustic interface). The manual input data may comprise data selected from a drop-down menu and / or any form of discrete selection (e.g., for lighting conditions, sunny, foggy, rainy, snowy and / or dark). Alternatively or in addition, the manual input data may comprise a (e.g., natural language) text indicating the visual domain. The user may provide information indicating the visual domain by means of the manual input data, in particular based on his / her perception and / or based on reading information from one or more sensors, without requiring a (e.g., wireless and / or wired) data connection of the sensors.

[0038] The electronically available information data may be received via a (e.g., wireless and / or wired) interface (in particular via the Internet or a cloud-based data storage) and / or may include data retrieved from the Internet, such as a local weather forecast. Thus, a wide range of, in particular already existing, electronically available information may be advantageously utilized, for example for generating visual domain textual prompts.

[0039] The data indicative of the visual domain may include any combination of data types. For example, positioning data may be used to determine appropriate electronically available information data, including a local weather forecast for the environment.

[0040] The method may include the step of receiving data indicating a class related to the environment. The method may further include the step of generating a textual prompt of the environment class based on the received data indicating a class related to the environment. Extracting environmental (particularly local) features from the received image data may further be conditioned on the generated textual prompt of the environment class.

[0041] Generating the context-class textual prompt may be performed, at least in part, by a CLIP encoder.

[0042] While conventionally, a CLIP encoder is frozen (and / or unchanging), in particular after a training phase, in contrast, and according to the computer-implemented techniques presented herein, generating context-class textual cues and / or visual-domain textual cues can at least partially change over time (e.g., after first processing the corresponding received data by the CLIP encoder) and / or can be performed by a trainable text adapter (e.g., comprising some multi-layer perceptron (MLP) layers).

[0043] By further conditioning the extraction of environmental (especially local) features on the generated environmental text cues, visual perception tasks such as classification tasks and / or semantic segmentation tasks can be facilitated.

[0044] The feature extraction can be further conditioned on the contextual textual cues. By conditioning on the contextual textual cues, classification and / or semantic segmentation as visual perception tasks can be facilitated and / or improved.

[0045] The step of generating a visual domain textual prompt and / or the step of generating an environmental class textual prompt can be performed by an adaptive contrastive language image pre-training (CLIP) encoder. By using an adaptive CLIP encoder to generate (particularly visual domain and / or environmental class) textual prompts, the generalizability of the computer-implemented technique can be achieved and / or improved, particularly considering previously untrained (and / or unseen) visual domains and / or previously untrained (and / or unseen) environmental classes.

[0046] The conditional feature extraction model may include a generative image-to-feature model and a conditioning model, wherein the generative image-to-feature model is configured to extract environmental (particularly local) features from the received environmental image data, and the conditioning model is configured to encode the generated visual domain text prompts for conditioning and / or controlling the generative image-to-feature model.

[0047] Generative image-to-feature models can be trained (and / or can operate) on a joint vision-language space. Generative image-to-feature models may include visual transformers (ViTs, e.g., as described in P. Esser et al., “Taming Transformers for High-Resolution Image Synthesis”, arXiv:2012.09841v3 [cs.CV]), (particularly vector graphics, VG) generative adversarial networks (GANs, particularly VG-GANs, e.g., as described in YongZheng Ong et al., “VG-GAN: Conditional GAN Framework for Graphical Design Generation”, 2022 IEEE International Conference on Image Processing (ICIP), Bordeaux, France, 2022, pp. 781-785, DOI: 10.1109 / ICIP46576.2022.9897935, which is incorporated herein by reference), diffusion models for visual perception (VPDs, e.g., as described in Wenliang Zhao et al., “Unleashing Text-to-Image Diffusion Models for Visual Perception, described in arXiv:2303.02153v1 [cs.CV]) and / or Stable Diffusion (SD), in particular SD Denoising UNet.

[0048] When executing this method, the generated image-to-feature model can be frozen and / or does not require further adaptation, in the sense that the weights of the generated image-to-feature model remain unchanged during the fine-tuning phase and / or the inference phase.

[0049] Optional context-like text cues can (e.g., directly) condition the image-to-feature model (e.g., by feeding the context-like text cues to the encoding and / or decoding layers and / or to optional intermediate layers).

[0050] The image-to-feature model may include a diffusion model, particularly SD. The SD network may include a U-Net architecture with an encoder and a decoder (particularly with skip connections).

[0051] Diffusion models (also known as diffusion probability models and / or score-based generative models) may include machine learning (ML) models and in particular generative models, which are configured to perform image generation, image denoising, image restoration and / or super-resolution with the help of a diffusion process, in particular by using a forward process (e.g., adding noise, in particular Gaussian noise, to the image), an inverse process (e.g., predicting noise in the image, in particular Gaussian noise, and compensating and / or subtracting accordingly) and a sampling process.

[0052] SD can include a latent diffusion model. SD can involve or include a neural network (NN) architecture (model) for text-to-image (especially diffusion) generation. SD aims to learn the diffusion process that generates the probability distribution of a given (image) dataset.

[0053] The NN architecture of SD may include a convolutional NN (CNN)-based architecture, particularly a U-Net with an encoder and a decoder, which has skip connections (e.g., between the encoder and decoder layers) and / or (e.g., cross) attention mechanisms (e.g., within the encoder and / or decoder layers). The NN architecture of SD may further include a diffusion process by which the input image is converted into a latent space image. Through the U-Net, the latent space image can be converted back into an image in a non-latent (and / or image) space.

[0054] Denoising UNets can utilize latent diffusion and / or can be trained in latent space rather than original image space. For example, SD can be specifically trained in latent space. Alternatively or in addition, a general diffusion model can be trained in original (e.g., image) space.

[0055] Training may comprise two phases, wherein the first phase trains an autoencoder to encode and / or transform an image in a latent space, and subsequently decode and / or reconstruct the image. In the second phase, a diffusion model may be trained in the latent space, wherein forward diffusion gradually adds (e.g., Gaussian) noise (particularly according to a Markov chain), and backward denoising (and / or UNet) predicts the (e.g., Gaussian) noise.

[0056] SD can advantageously produce high-quality (e.g., high-resolution and / or photorealistic) synthetic images based on textual cues (particularly visual domain and / or context-class) at low computational cost.

[0057] Computer-implemented techniques generalize conventional generative image-to-feature models, in particular SD and / or VPD, by preserving generalizability and / or facilitating adaptation to different visual domains and optionally to different environment classes through corresponding textual cues.

[0058] Compared to a general "diffusion model", SD can train a diffusion model in a latent space (in particular a pre-trained and / or subsequently frozen autoencoder, for example) instead of training in image or pixel space. By training in the latent space, the computational cost can be reduced because the latent space is much smaller than the regular pixel space, for example eight (8) times smaller.

[0059] For generating image-to-feature models (and / or conditional feature extraction models), there are alternatives to SD, such as described by Simian Luo et al. in "Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference", arXiv:2310.04378v1 [cs.CV], which is incorporated herein by reference (and referred to as LCM for short). In general, techniques for extracting environmental features from image data are not limited to SD, but can be performed, for example, using LCM. The main advantage of SD can be seen from its open source availability.

[0060] The conditioning model may include a ControlNet. The ControlNet may include an encoder and convolutional layers with a cross-attention mechanism for generating an image-to-feature model, particularly the encoder of the SD network.

[0061] Generative image-to-feature models, and in particular SD networks, can be conditioned by the visual domain with the aid of conditioning models as additional trainable branches, in particular by using ControlNet. For the basics of conditioning SD networks using ControlNet, please refer to L. Zhang et al., “Adding Conditional Control to Text-to-Image Diffusion Models” (2023), available at Computer Vision Foundations and IEEE Xplore, which is incorporated herein by reference.

[0062] SD may include a frozen UNet (e.g., after initial training), particularly in order to retain trained (and / or prior) knowledge during fine-tuning. ControlNet may include a trainable copy of SD's UNet encoder for conditioning on visual domain textual cues. ControlNet may further include one or more zero convolutional layers through which visual domain textual cues are fed into SD's decoder. A zero convolutional layer may include a (e.g., 1×1) convolutional layer with both weights and biases initialized to zero.

[0063] A connection between SD Networks and ControlNet or a combination of SD Networks and ControlNet can provide a particularly computationally efficient conditioning, which saves time and (eg, Graphics Processing Unit GPU) memory.

[0064] The combination of SD-Net and ControlNet with visual domain textual cues and optional context-sensitive textual cues can provide a particularly rich and generalizable framework for feature extraction and / or visual perception task performance.

[0065] Due to the expensive annotation process (e.g., for segmentation tasks), conventional models for perception tasks are trained on specific datasets that may include limited data and / or scenes. In contrast, techniques for extracting environmental features from image data (e.g., according to the first method aspect) leverage the rich prior knowledge of a pre-trained large-scale text-to-image model that has seen a wide variety of scenes and is equipped with a joint visual-language latent space. Techniques for extracting environmental features from image data particularly leverage the control potential of language and / or text cues and the powerful pre-training knowledge to improve the generalization and adaptability of downstream task (e.g., AD, robotic motion planning, and / or access control) models across different operational domains during deployment. Alternatively or additionally, only a lightweight decoder needs to be trained for a specific task (e.g., semantic segmentation and / or object detection).

[0066] Regarding the second method aspect, a computer-implemented method for training a conditional feature extraction model is provided, the model being used to extract environmental (particularly local) features from image data conditioned on a visual domain textual cue. The method comprises the step of receiving a training dataset. The training dataset comprises a visual domain textual cue indicating a visual domain of the environment, image data of the environment, and (particularly local) features of the environment. The method further comprises the step of training the conditional feature extraction model based on the received training dataset. Training the conditional feature extraction model comprises receiving the image data of the environment as input, receiving the visual domain textual cue as a condition, and receiving the (particularly local) features as a ground truth.

[0067] The training dataset may comprise synthetic data (e.g., synthetic image data generated based on (particularly local) features and visual domain textual cues). Alternatively or in addition, the training dataset may comprise real data, particularly real image data (e.g., with (particularly local) features and / or visual domain cues as annotations).

[0068] By training conditional feature extraction models on multiple training datasets (particularly including synthetic training datasets), feature extraction and / or performance of visual perception tasks can be rendered adaptable (and / or generalizable) to (particularly previously untrained) visual domains and / or environment classes, and / or adaptable (and / or generalizable) to visual domains and / or environment classes for which real training datasets are scarce (e.g., due to the scarcity of real image data).

[0069] SD, for example, can be pre-trained on a large-scale dataset, such as LAION-5B. Conditional feature extraction models can be trained on a small, labeled real dataset for a specific task, such as semantic segmentation. For AD use cases, for example, the Cityscapes dataset (which includes 4,000 datasets with high-quality annotations and 20,000 datasets with coarse annotations, https: / / www.cityscapes-dataset.com / ) can be used. Alternatively or in addition, synthetic data can also be used, especially if annotations (such as segmentation label maps) are also generated along with the images.

[0070] The method according to the first method aspect may be used for autonomous driving (AD), planning the movement of a robot, operating a household appliance and / or controlling an access control system.

[0071] By (particularly local) feature extraction and / or by performing visual perception tasks based on image data and visual domain textual cues (and optionally context-related textual cues), sufficiently accurate data about the environment can be provided to facilitate AD and / or path planning (and / or trajectory planning) for (particularly AD) vehicles and / or (particularly autonomous) robots and / or household appliances (e.g. autonomous mobile vacuum cleaners and / or human housekeepers).

[0072] Regarding a first device aspect, a computing device for extracting environmental features from image data is provided. The computing device includes a visual domain indication receiving interface configured to receive data indicating a visual domain of an environment. The device further includes a visual domain text prompt generation module configured to generate a visual domain text prompt based on the received data indicating the visual domain of the environment. The computing device further includes an environmental image data receiving interface configured to receive image data representing the environment. The computing device further includes a conditional feature extraction model configured to extract environmental features, particularly local features, from the received image data. The extraction of the (particularly local) features is conditional on the generated visual domain text prompt. Optionally, the computing device according to the first device aspect may include a visual perception task execution module configured to perform a visual perception task on the environment based on the extracted (particularly local) features. Further optionally, the computing device according to the first device aspect may include an environmental class receiving interface configured to receive data indicating a class related to the environment. The computing device further includes an environmental class text prompt generation module configured to generate an environmental class text prompt based on the received data indicating a class related to the environment. Extraction of environmental (particularly local) features from the received image data may be further conditioned on the generated environmental-related textual cues.

[0073] The computing device according to the first apparatus aspect may be configured to perform any one of the steps and / or include any one of the features disclosed in the context of the first method aspect.

[0074] With respect to the second device aspect, a computing device for training a conditional feature extraction model is provided, which is used to extract environmental (particularly local) features from image data conditioned on a visual domain textual prompt. The computing device includes a training data receiving interface, which is configured to receive a training data set. The training data set includes a visual domain textual prompt indicating a visual domain of the environment, image data of the environment, and environmental (particularly local) features. The computing device further includes a training module, which is configured to train the conditional feature extraction model based on the received training data set. Training the conditional feature extraction model includes receiving the image data of the environment as input, receiving the visual domain textual prompt as a condition, and receiving the (particularly local) features as a reference truth.

[0075] The computing device according to the second apparatus aspect may be configured to perform any one of the steps and / or include any one of the features disclosed in the context of the second method aspect.

[0076] The computing device according to the first method aspect and the second method aspect may be the same.

[0077] Regarding a system aspect, a system for extracting environmental features from image data is provided. The system includes a computing device according to the first device aspect and at least one sensor and / or at least one interface for receiving data indicative of a visual domain and image data representative of an environment. The data indicative of the visual domain and / or the image data representative of the environment can be received wirelessly (e.g., from a sensor external to the AD vehicle and / or a network server). Alternatively or in addition, the data indicative of the visual domain and / or the image data representative of the environment can be received via a wired connection (e.g., for a fixedly installed access control system).

[0078] In another aspect, a computer program product is provided. The computer program product includes program elements that, when loaded into a memory of a computing device, direct the computing device to perform the steps of a method for extracting environmental features from image data and / or for training a conditional feature extraction model that extracts environmental (particularly local) features from image data conditioned on a visual domain textual cue, according to the first method aspect and / or the second method aspect.

[0079] With respect to still another aspect, a computer-readable medium is provided having stored thereon program elements that can be read and executed by a computing device so that when the program elements are executed by the computing device, the steps of the method according to the first method aspect and / or the second method aspect are performed, the method being used to extract environmental features from image data and / or for training a conditional feature extraction model that extracts environmental (particularly local) features from image data conditioned on visual domain textual cues. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 is a flow chart of a method for extracting environmental features from image data.

[0081] Figure 2 is a flowchart of a method for training a conditional feature extraction model for extracting contextual (particularly local) features from image data conditioned on visual domain textual cues.

[0082] Figure 3 is a schematic illustration of the architecture of a computing device for extracting environmental features from image data.

[0083] Figure 4 is a schematic illustration of the architecture of a computing device for training a conditional feature extraction model for extracting contextual (particularly local) features from image data conditioned on visual domain textual cues.

[0084] Figure 5 A conventional SD architecture is schematically illustrated.

[0085] Figure 6A conventional VPD architecture is schematically illustrated.

[0086] Figure 7 The conventional ControlNet architecture is schematically illustrated.

[0087] Figure 8 Techniques for extracting environmental features from image data are schematically illustrated.

[0088] Figure 9A and Figure 9B Data indicative of a visual domain of an environment and a probabilistic analysis for generating, in particular automatically generating, a predetermined set of visual domain textual cues are exemplarily illustrated.

[0089] Figure 10A 、 Figure 10B and Figure 10C Schematically illustrate use cases for techniques for extracting environmental features from image data, in particular for autonomous driving, robotic motion planning, and access control, respectively. DETAILED DESCRIPTION

[0090] Figure 1 A computer-implemented method 100 for extracting environmental features from image data is schematically illustrated.

[0091] Method 100 includes a step (S102) of receiving data indicating a visual domain of an environment. Method 100 further includes a step (S104) of generating a visual domain textual prompt based on the received (S102) data indicating the visual domain of the environment. In step (S106), image data representing the environment is received. The method further includes a step (S108) of extracting (particularly local) features of the environment from the received (S106) image data. Extraction (particularly local) features (S108) is performed by a conditional feature extraction model. Extraction (particularly local) features (S108) is conditional on the generated (S104) visual domain textual prompt.

[0092] Optionally, in step S110 , a visual perception task of the environment is performed based on the extracted (especially local) features in S108 .

[0093] Further optionally, the method 100 includes a step S101 of receiving data indicating an environmental class related to the environment and a step S103 of generating an environmental class text prompt based on the received S101 data indicating the class related to the environment. Extracting S108 environmental (particularly local) features from the received S106 image data can be further conditioned on the generated S103 environmental class text prompt.

[0094] Figure 2A computer-implemented method 200 for training a conditional feature extraction model for extracting contextual (particularly local) features from image data conditioned on visual-domain textual cues is schematically illustrated.

[0095] Method 200 includes a step S202 of receiving a training dataset. The training dataset includes a visual domain textual cue indicating a visual domain of an environment, image data of the environment, and (particularly, local) features of the environment. Method 200 further includes a step S204 of training a conditional feature extraction model based on the received training dataset S202. Training S204 the conditional feature extraction model includes receiving the image data of the environment as input, receiving the visual domain textual cue as a condition, and receiving (particularly, local) features as ground truth.

[0096] Figure 3 The architecture of a computing device 300 for extracting environmental features from image data is schematically illustrated.

[0097] The computing device 300 includes a visual domain indication receiving interface 302 configured to receive data indicating the visual domain of an environment. The computing device 300 further includes a visual domain text prompt generation module 304 configured to generate a visual domain text prompt based on the received data indicating the visual domain of the environment. The computing device 300 further includes an environmental image data receiving interface 306 configured to receive image data representing the environment. The computing device 300 further includes a conditional feature extraction model 308 configured to extract environmental (particularly local) features from the received image data. The extraction of the (particularly local) features is conditional on the generated visual domain text.

[0098] Optionally, the computing device 300 may include a visual perception task execution module 310 configured to perform a visual perception task on the environment based on the extracted (especially local) features.

[0099] Further optionally, the computing device 300 may include an environment class receiving interface 301 configured to receive data indicating an environment-related class. The computing device may further include an environment class text prompt generating module 303 configured to generate an environment class text prompt based on the received data indicating an environment-related class. Extracting environment (particularly local) features from the received image data may be further conditioned on the generated environment class text prompt.

[0100] Any of the visual domain indication receiving interface 302, the optional environment class receiving interface 301, and the environment image data receiving interface 306 can be embodied by the input-output interface 312. Alternatively or additionally, any of the visual domain text prompt generation module 304, the optional environment class text prompt generation model 303, the conditional feature extraction model 308, and the optional visual perception task execution module 310 can be embodied by the processing unit 314. Further alternatively or additionally, the computing device 300 can include at least one memory 316.

[0101] Computing device 300 may be configured to perform any of the steps and / or include any of the features disclosed in the context of method 100 .

[0102] Figure 4 The architecture of a computing device 400 for training a conditional feature extraction model for extracting contextual (particularly local) features from image data conditioned on visual domain textual cues is schematically illustrated.

[0103] The computing device 400 includes a training data receiving interface 402 configured to receive a training data set. The training data set includes a visual domain textual prompt indicating a visual domain of an environment, image data of the environment, and (particularly local) features of the environment. The computing device 400 further includes a training module 404 configured to train a conditional feature extraction model based on the received training data set. Training the conditional feature extraction model includes receiving the image data of the environment as input, receiving the visual domain textual prompt as a condition, and receiving (particularly local) features as a ground truth.

[0104] The training data receiving interface 402 may be embodied by an input-output interface 406. Alternatively or additionally, the training module 404 may be embodied by a processing unit. Further alternatively or additionally, the computing device 400 may include at least one memory 410.

[0105] Computing device 400 may be configured to perform any of the steps and / or include any of the features disclosed in the context of method 200 .

[0106] Any of the processing units 314 , 408 may include a central processing unit (CPU) and / or a graphics processing unit (GPU).

[0107] Computing device 300 and computing device 400 may be the same.

[0108] Techniques for extracting context features from image data and for training conditional feature extraction models The techniques for extracting context (especially local) features from image data provide a novel way to leverage large-scale text-to-image models (especially conditional feature extraction models) for domain adaptive visual perception tasks (e.g., semantic segmentation and / or object detection).

[0109] Previous work on VPD has utilized a fine-tuned SD and a trained task-specific decoder, and has shown promising results when tested on the same training domain. The technique for extracting context features from image data and for training a conditional feature extraction model for extracting context (particularly local) features from image data takes this a step further and aims to apply the model to a variety of unseen domains using a conditional feature extraction model (and / or a large-scale pre-trained model) and a generalizable representation of a joint visual-linguistic embedding space. Textual cues (particularly visual domain and / or context class) are utilized, for example as domain indicators, to adapt the model to different domains. The conditional feature extraction model (as a text-to-image diffusion model) can (e.g. naturally) be conditioned on text. Additionally, the task-specific decoder can also be conditioned on the domain embedding to further enhance the conditioning, e.g. Figure 8 Schematically illustrated.

[0110] For example, SD not only captures rich semantic information, but further may have seen many visual domains, such as different weather and / or lighting conditions, making its representation more generalizable. Unlike VPD, which may overfit on the training set, generalizable representations (e.g., generalizable representations of SD) can be utilized based on techniques for extracting environmental features from image data and for training conditional feature extraction models for extracting environmental (especially local) features from image data, and used for domain adaptation and / or generalizable tasks. For example, SD can be fine-tuned on the training set while retaining its ability to adapt to different test sets. For example, a conditional feature extraction model can be fine-tuned on sunny images, and the conditional feature extraction model will have reasonable performance not only on the trained domain (e.g., sunny conditions), but also on unseen data (e.g., image data of rainy and / or foggy days).

[0111] Note that conventional VPD directly fine-tunes the entire denoising model, which may destroy prior knowledge, such as Figure 6 As shown.

[0112] As an alternative, one can adopt the fine-tuning approach of ControlNet, which freezes the original SD while adding trainable branches to maximize the retention of prior knowledge, such as Figure 7Additionally, techniques for extracting environmental features from image data and for training a conditional feature extraction model for extracting environmental (particularly local) features from image data exploit the use of textual cues (e.g., visual domain and / or environmental class), particularly as domain indicators, to leverage the joint visual and textual embedding space of SD to conveniently adapt the conditional feature extraction model to different domains.

[0113] The technology for extracting environmental features from image data and for training conditional feature extraction models for extracting environmental (particularly local) features from image data is more robust and can be generalized to various (particularly unseen) visual domains, such as different weather and / or lighting conditions, and / or new countries. Conventionally, the network tends to overfit to the training set (for example, images of sunny days) and cannot be generalized to unseen domains (for example, scenes of rainy and / or foggy days). Since the environment in the real world is diverse and constantly changing, and the conditional feature extraction model can only be trained on limited data, it is important to design a robust conditional feature extraction model that can cope with unseen situations. This is crucial for real-world applications such as AD (particularly autonomous driving) and / or robots. For example, if the conditional feature extraction model trained on sunny images cannot be generalized to night scenes, people walking on the road at night may be ignored by autonomous cars and cause accidents.

[0114] In contrast, techniques for extracting environmental features from image data can flexibly adapt to different domains with the help of textual cues (e.g., visual domain and / or environmental class). Alternatively or additionally, conditional feature extraction models are built on top of large-scale pre-trained models (such as SD), which can transfer rich prior knowledge and enable faster training convergence.

[0115] Techniques for extracting environmental features from image data and for training conditional feature extraction models for extracting (particularly local) environmental features from image data can be used to analyze data obtained from sensors. The sensors can determine measurements of the environment in the form of sensor signals, which can be given by (particularly digital) image data, including, for example, video, radar, LiDAR, ultrasound, motion, and / or thermal images.

[0116] Techniques for extracting environmental features from image data and for training conditional feature extraction models for extracting environmental (particularly local) features from image data can be used to classify sensor data, detect the presence of objects in sensor data, and / or perform semantic segmentation on sensor data, for example with respect to traffic signs, road surfaces, pedestrian vehicles and / or object classes that may appear in semantic segmentation tasks, such as trees and / or sky.

[0117] Alternatively or in addition, techniques for extracting environmental features from image data and for training conditional feature extraction models for extracting environmental (particularly local) features from image data can be used to determine a continuous value or multiple continuous values (e.g., perform regression analysis), for example, regarding distance, speed, acceleration, and / or items in tracking data, such as objects. The determination of one or more continuous values can be implemented based on low-level features (e.g., edges and / or pixel attributes of the image).

[0118] Alternatively or additionally, techniques for extracting environmental features from image data and for training conditional feature extraction models for extracting environmental (particularly local) features from image data can be used to detect anomalies in (particularly downstream) technical systems (e.g., AD vehicles, robots, home appliances, and / or access control systems). This can be achieved by analyzing data (e.g., scalar time series, also denoted as measurement and control), particularly data from sensors, and then operating the technical system accordingly. Alternatively or additionally, anomalies can be detected, and the technical system can then be operated in a fail-safe mode.

[0119] Training methods / data: Techniques for extracting environmental features from image data and for training conditional feature extraction models. Techniques for extracting environmental (especially local) features from image data can be considered upstream in the machine learning (ML) tool chain. After training, the conditional feature extraction model of the ML system can be put into downstream use, such as for AD, robotic motion planning, home appliance operation, and / or access control system control.

[0120] According to one embodiment, a technique for extracting environmental features from image data and for training a conditional feature extraction model for extracting environmental (particularly local) features from image data not only employs SD on the training domain but also retains its generalization capability, allowing it to be used in unseen domains. Due to SD's joint visual-linguistic embedding space, the conditional feature extraction model can be adapted (e.g., to different domains) using textual cues (particularly visual domain and / or environmental class).

[0121] As already mentioned above, SD does not operate in the image space, but in the latent space of the autoencoder, as Figure 5 Schematically illustrated. First, the encoder ε maps the given image x into a spatial latent code z = ε(x). Then, the decoder D maps z back to the image space.

[0122] The autoencoder is trained to reconstruct a given image, i.e., D(ε(x))≈x.

[0123] In the second stage, a diffusion model is trained in the latent space Z. The diffusion model consists of a forward diffusion process and a backward denoising process. The forward pass consists of a Markov chain to gradually add Gaussian noise to the clean data. Formally, it can be written, for example, in is a fixed variance table.

[0124] The noise potential can be calculated in closed form, e.g.,

[0125]

[0126] where z0 = ε(x), and The denoising process can be parameterized by another Gaussian distribution p θ (z t-1 |z t ):=N(z t-1 ;μ θ (z t , t), σ θ (z t , t)). Basically, μ θ (z t , t) can be expressed as z modeled by UNet t and prediction noise ∈ θ (z t , t). The parameters of the UNet can be learned by minimizing the L2 norm of the noise prediction at sampling time step t:

[0127]

[0128] A more mathematical derivation is provided in Jonathan Ho et al., “Denoising Diffusion Probabilistic Models”, arXiv:2006.11239v2 [cs.LG], which is incorporated herein by reference. Conditional information, such as textual cues (particularly visual domain and / or environment class), is used to Figure 5 The bottom of the 文本 ” is inserted into the cross-attention mechanism.

[0129] At inference time, the latent representation z can be randomly sampled from a Gaussian distribution T , and the trained denoising UNet can then be employed sequentially to obtain a given Z from t=T to t=1 t The denoising potential Z t-1 The final composite image can be obtained by feeding the clean latent volume Z0 through the decoder D.

[0130] VPD, such as Figure 6As schematically indicated by flame symbols in , the pre-trained SD UNet is fine-tuned for downstream visual perception tasks such as semantic segmentation and / or depth estimation. Figure 6 Schematically illustrated, VPD extracts features and cross-attention maps from UNet and feeds them to a task-specific decoder. For semantic segmentation tasks, class embeddings can be extracted from a frozen (as indicated by the snowflake symbol) CLIP text encoder and adapted through a trainable text adapter (as indicated by the flame symbol, e.g., several MLP layers), which can then be used as (e.g., domain and / or class) text cues (also known as: text conditions) to the SD UNet. However, VPD only focuses on in-domain capabilities, e.g., assuming that the test set shares the same distribution as the training set, and large models tend to overfit to a specific training domain.

[0131] In contrast, techniques for extracting environmental features from image data and for training conditional feature extraction models for extracting environmental (particularly local) features from image data take into account the generalizability of the conditional feature extraction models and design more robust conditional feature extraction models for unseen domains and / or unseen environments.

[0132] ControlNet enhances SD with additional input conditions such as tag mapping. Figure 7 Schematically illustrated, ControlNet clones a trainable copy of the UNet encoder and freezes the original SD during fine-tuning (as indicated by the flame and snowflake symbols, respectively). The input conditions are fed as input to ControlNet, and after a zero convolution layer and a trainable encoder, the features are inserted back into the decoder of SD, where the zero convolution layer is basically a 1×1 convolution layer with weights and bias initialized to zero.

[0133] By inserting the condition y in Equation (2), the training objective of the ControlNet plus SD UNet architecture is adopted:

[0134]

[0135] Techniques for extracting context features from image data and for training conditional feature extraction models for extracting context (particularly local) features from image data utilize conditional feature extraction models, particularly as large-scale text-to-image generation models (e.g., including SD) for visual perception tasks (e.g., semantic segmentation and / or object detection), while preserving the generalizability of the conditional feature extraction models across various (particularly unseen) domains and / or environments. The key concept is to leverage the joint visual-linguistic latent space of the large-scale model and design the conditional feature extraction model to flexibly adapt based on textual cues (also known as: textual conditioning) that indicate visual domain and / or context class (particularly visual domain and / or context class).

[0136] As indicated by reference numerals 802 and 804, given data (e.g., including an image) indicating a class associated with an environment and / or indicating a visual domain of the environment, the operational visual domain and / or environmental class to which it belongs is determined. This can be done manually or automatically. In the manual case, the operational "visual domain" and / or environmental class can be directly specified by a human (e.g., via a UI, particularly a GUI).

[0137] Alternatively or additionally, some datasets, such as the Adverse Conditions Dataset (ACDC, available at https: / / acdc.vision.ee.ethz.ch / ), already include metadata about weather conditions (e.g., rain, fog, snow, and / or night). In downstream (and / or real-world) applications, such as AD (especially autonomous driving), humans can dynamically provide such information to the model.

[0138] To automatically identify the operational visual domain and / or environment class, a pre-trained visual question answering (VQA) model such as mPlug can be utilized (e.g., as described in Chenliang Li et al., “mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections,” arXiv:2205.12005v2 [cs.CL], incorporated herein by reference). For example, Figure 9A and Figure 9B , where the model asks “What are the weather conditions?” based on image 804 as data indicating the visual domain. The model returns the top answer 812 with likelihood. For example, Figure 9A In , "sunny" and "cloudy" are the two most likely answers with almost equal probability, while in Figure 9B Among them, "cloudy" is by far the most likely answer.

[0139] Similarly, questions may be asked, for example, about lighting conditions and / or country. The top 1 answer may serve as a (e.g., intermediate) visual domain hint input and / or data indicative of the visual domain (e.g., for Figure 9B Alternatively or additionally, since the concept of “visual domain” may be vague (e.g., Figure 9A , with almost equal probability), so there is a probability that the top K answers (e.g., K=2) may be taken. With (e.g., intermediate) visual domain hint input and / or data indicating the visual domain, a frozen CLIP encoder 806 and a (particularly lightweight) trainable text adapter 808 may be employed to obtain a visual domain text hint at a reference marker 812 (also denoted as a domain embedding W i ),For example:

[0140] W i =ψ adp (CLIP text (y i )), i=1,...,K (4)

[0141] where y i Represents visual domain cues, such as "sunny scene", and CLIP 文本 and ψ adp They are CLIP text encoder and text adapter respectively.

[0142] like Figure 8 As schematically illustrated in , the conditional feature extraction model can be further conditioned on an environment class textual cue (also called class embedding c) at reference marker 810, which can be obtained from data indicating a class associated with the environment 802 using a further frozen CLIP encoder 806 and a further (particularly lightweight) trainable textual adapter 808.

[0143] Visual domain textual cues (and / or domain embeddings) 812 can be used to condition a conditional feature extraction model that includes an exemplary frozen SD UNet 822 and a ControlNet 814 having a trainable copy 816 of a UNet encoder (e.g., SD UNet 824) followed by zero convolutions 818. As an alternative to directly fine-tuning the entire UNet as in conventional VPD, Figure 8 In the example, the fine-tuning of ControlNet 814 keeps the original UNet 822 frozen and adds a trainable copy of the UNet encoder 816, which is initialized with the pre-trained weights of the SD UNet encoder. In this way, the prior knowledge of SD is fully preserved.

[0144] Both the frozen SD UNet 822 and the trainable copy of the UNet encoder 816 are configured to receive the latent quantity representation 820.

[0145] Naturally, the optional environmental text prompt 810 (also called: text condition) can be used with Figure 8 The same way as the conventional SD in is used, which can be used via crisscross attention. Figure 8 The visual domain textual cues 812 (also known as: domain embeddings) in an exemplary embodiment of are fed into the trainable UNet encoder 816 of ControlNet 814.

[0146] Features and cross-attention maps can be similarly extracted to the VPD, as schematically illustrated by reference numeral 826. Alternatively or additionally, features can be extracted from multiple time steps rather than from a single clean potential z0. Based on equation (1), the noise potential can be obtained directly at different time steps. The feature extraction step can be formulated, for example, as:

[0147] f i，t =SD(W i , z t ,c),f i =Agg(f i，t ), f=∑ i f i *p i (5)

[0148] Where SD stands for stable diffusion, and c is the class embedding (and / or context class text hint) 810 for the task (e.g., semantic segmentation). Agg indicates the aggregation operation, e.g., the features f at different time steps i，t The simple mean of p i is the current domain embedding W i (and / or visual domain textual prompts) 812 probability.

[0149] Further, the visual domain textual hints (and / or domain embeddings) 812 may be used in the task specific decoder 828. The domain embeddings (visual domain textual hints) 812 may first be aggregated as indicated by reference numeral 824,

[0150] W agg =Agg(W i , p i ), (6)

[0151] Where Agg is an aggregation operation, such as a simple probabilistic average and / or a learnable mapping. As described in previous work, such as Seonguk et al., "Learning to Optimize Domain Specific Normalization for Domain Generalization," DOI: 10.1007 / 978-3-030-58542-65, which is incorporated herein by reference, normalization statistics can have a significant impact on visual domain (and / or environment class) generalization capabilities. Therefore, according to one embodiment, domain embeddings (and / or visual domain textual hints) 812 are used to map these embeddings to normalization parameters, such as:

[0152]

[0153] where μ fdec and γ fdec is the decoder f dec The mean and standard deviation of the features within the task-specific decoder 828. agg ) and γ(W agg ) are the learned affine transformation parameters, and the aggregated domain is embedded in W agg Finally, the task-specific decoder 828 will output predictions for a specific task (e.g., semantic segmentation and / or object classification).

[0154] Figure 10A An example of a first use case is schematically illustrated, wherein a controller 1004 of an AD vehicle 1002-1 receives results of feature extraction (e.g., according to step S108) and / or visual perception tasks (e.g., according to step S110) for performing AD functions. Optionally, the received results may also be displayed to the driver and / or passengers, for example, on a display 1006 on a center console.

[0155] Figure 10B An example of a second use case is schematically illustrated, where the controller 1004 of the robot 1002 - 2 receives results of feature extraction (e.g., according to step S108 ) and / or visual perception tasks (e.g., according to step S110 ) for planning the movement of the robot 1002 .

[0156] Figure 10CAn example of a third use case is schematically illustrated, wherein an access control system 1002-3 has a first sensor 1010 for acquiring image data representing the environment, an optional second sensor 1012 for acquiring audio data representing the environment, a controller 1004, and an optional display 1006 (particularly for a person requesting access). The controller 1004 receives the results of the feature extraction (e.g., according to step S108) and / or the visual perception task (e.g., according to step S110) to determine whether to grant access.

[0157] Cited prior art

[0158] [1] R.Rombach et al., "High-Resolution Image Synthesis with LatentDiffusion Models", arXiv: 2112.10752v2[cs.CV]

[0159] [2]Wenliang Zhao et al., "Unleashing Text-to-Image Diffusion Models forVisual Perception, arXiv: 2303.02153v1[cs.CV])

[0160] [3] P.Esser et al., "Taming Transformers for High-Resolution ImageSynthesis", arXiv: 2012.09841v3[cs.CV]

[0161] [4] Yong Zheng Ong et al., “VG-GAN: Conditional GAN Framework for Graphical Design Generation”, 2022 IEEE International Conference on Image Processing (ICIP), Bordeaux, France, 2022, pp. 781-785, doi:10.1109 / ICIP46576.2022.9897935

[0162] [5]Dan Zhang et al., ""Anomaly-Aware Semantic Segmentation via Style-Aligned OoD Augmentation", arXiv:2308.09965v1[cs.CV]

[0163] [6]Simian Luo et al., "Latent Consistency Models: Synthesizing High-Resolution Images with Few-Step Inference", arXiv:2310.04378v1 [cs.CV]

[0164] [7]https: / / www.cityscapes-dataset.com /

[0165] [8]Jonathan Ho et al., "Denoising Diffusion Probabilistic Models", arXiv:2006.11239v2 [cs.LG]

[0166] [9]https: / / acdc.vision.ee.ethz.ch /

[0167]

[10] Chenliang Li et al., "mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections", arXiv:2205.12005v2 [cs.CL]

[0168]

[11] Seonguk Seo et al., "Learning to Optimized Domain Specific Normalization for Domain Generalization" (2020) 10.1007 / 978-3-030-58542-6_5

Claims

1. A computer-implemented method (100) for extracting environmental features from image data, comprising the following method steps: - receiving (S102) data indicative of a visual domain of an environment; - generating (S104) a visual domain text prompt based on the received (S102) data indicating the visual domain of the environment; - receiving (S106) image data representing the environment; and - extracting (S108) environmental features, in particular local features, from the received (S106) image data, wherein The extraction of features, in particular local features (S108) is performed by a conditional feature extraction model, and wherein the extraction of features, in particular local features (S108) is conditioned on the generated (S104) visual domain textual cues.

2. The method (100) according to claim 1, further comprising the following method steps: - performing (S110) a visual perception task on the received image data based on the extracted (S108) features, in particular local features.

3. The method (100) of the immediately preceding claim, wherein the visual perception task comprises at least one of the following: - Depth estimation; - Object detection; - classification; and -Semantic segmentation.

4. The method (100) according to claim 2 or claim 3, wherein performing (S110) the visual perception task on the image data is conditioned on the generated (S104) visual domain textual cue.

5. The method (100) according to any one of the preceding claims, wherein the received (S102) data indicative of the visual domain comprises at least one of the following: -Environmental sensor data; - positioning data determined with the aid of a positioning system, in particular a satellite navigation system; - Manual data entry; and - Electronically available information data.

6. The method (100) according to any one of the preceding claims, further comprising the steps of: - receiving (S101) data indicating a class associated with an environment; and - generating (S103) an environment class text prompt based on the received (S101) data indicating a class related to the environment, in, Environmental features, particularly local features, are extracted (S108) from the received (S106) image data, further conditioned on the generated (S103) environmental text prompt.

7. The method (100) of the immediately preceding claim, wherein the extraction of features (S108) is further conditioned on the context-related textual prompt.

8. A method (100) according to any one of the preceding claims, wherein the step of generating (S104) the visual domain text prompt and / or the step of generating (S103) the environmental class text prompt is performed by an adaptive contrastive language image pre-trained CLIP encoder.

9. A method (100) according to any one of the preceding claims, wherein the conditional feature extraction model comprises a generated image-to-feature model configured for extracting environmental features, in particular local features, from the received (S106) environmental image data, and a conditioning model configured for encoding the generated (S104) visual domain textual cues to condition and / or control the generated image-to-feature model.

10. The method (100) of the immediately preceding claim, wherein the generative image-to-feature model comprises a diffusion model, in particular stabilized diffusion (SD), wherein the SD network comprises a U-Net architecture with an encoder and a decoder, in particular with skip connections.

11. The method (100) of claim 9 or claim 10, wherein the conditioning model comprises a ControlNet, wherein the ControlNet comprises an encoder and convolutional layers having a cross-attention mechanism for the generated image-to-feature model, in particular for the encoder of the SD network.

12. A computer-implemented method (200) for training a conditional feature extraction model for extracting environmental features, particularly local features, from image data conditioned on visual domain textual cues, the method (200) comprising the following method steps: - receiving (S202) a training dataset, wherein the training dataset comprises a visual domain textual cue indicating a visual domain of an environment, image data of the environment and features of the environment, in particular local features; and -Training (S204) a conditional feature extraction model based on the received (S202) training dataset, wherein training (S204) the conditional feature extraction model includes receiving image data of the environment as input, receiving the visual domain text prompt as a condition, and receiving the features, especially local features, as a reference truth.

13. Use of the method (100; 200) according to any one of claims 1 to 11 or claim 12 for at least one of the following: -Autonomous driving, AD; - Plan the robot's movements; -operate household appliances; and -Control access control system.

14. A computing device (300) for extracting environmental features from image data, comprising: - a visual domain indication receiving interface (302) configured for receiving data indicative of a visual domain of an environment; - a visual domain textual prompt generation module (304) configured for generating a visual domain textual prompt based on the received data indicating the visual domain of the environment; - an environment image data receiving interface (306) configured to receive image data representing the environment; as well as - a conditional feature extraction model (308) configured for extracting environmental features, in particular local features, from the received image data, wherein the extraction of the features, in particular local features, is conditioned on the generated visual domain textual cues.

15. A computing device (400) for training a conditional feature extraction model, wherein the model is used to extract environmental features, particularly local features, from image data conditioned on visual domain textual cues, the computing device (400) comprising: - a training data receiving interface (402) configured for receiving a training data set, wherein the training data set comprises a visual domain textual cue indicating a visual domain of an environment, image data of the environment and features of the environment, in particular local features; and - A training module (404) configured to train a conditional feature extraction model based on the received training dataset, wherein training the conditional feature extraction model comprises receiving image data of the environment as input, receiving the visual domain textual prompt as a condition, and receiving the features, in particular local features, as ground truth.

Images