Unsupervised prompt learning for data preselection using vision-language models

Abstract

A method for performing data preselection for an object detection system involves receiving an initial dataset containing unlabeled data corresponding to one or more images, and supplying the initial dataset and a plurality of learnable prompt vectors to a pretraining model. The learnable prompt vectors contain text inputs. The method further involves generating, using the pretraining model, an unsupervised learning prompt based on the initial dataset and the plurality of learnable prompt vectors. The unsupervised learning prompt corresponds to a multimodal feature of the one or more images in the initial dataset.The procedure further involves extracting features from one of the first datasets and a second dataset based on the unsupervised learning prompt, selecting and labeling a subset of instances of the extracted features, and generating and outputting a labeled dataset based on the labeled subset of instances.

Description

Technical area

[0001] The present disclosure relates to data pre-selection for object detection systems and methods. background

[0002] Object detection techniques, such as visual object detection techniques, can use various detection models to detect or predict, classify, and label objects and regions in captured images. For example, one or more machine learning models for object detection and labeling are trained using subsets of data samples. Brief description

[0003] A method for performing data pre-selection for an object detection system includes receiving a first data set including unlabeled data corresponding to one or more images, providing the first data set and a plurality of learnable prompt vectors to a pre-training model. The learnable prompt vectors include text inputs. The method further includes generating, using the pre-training model, an unsupervised learning prompt based on the first data set and the plurality of learnable prompt vectors. The unsupervised learning prompt corresponds to a multimodal feature of the one or more images of the first data set.The method further includes extracting features from one of the first dataset and a second dataset based on the unsupervised learning prompt, selecting and labeling a subset of instances of the extracted features, and generating and outputting a labeled dataset based on the labeled subset of instances.

[0004] For other features, the pre-training model is a Bootstrapping Language Image Pre-Training (BLIP-2) model. The extracted features include clusters of unlabeled image data. The selected subset of instances includes one or more of the clusters of unlabeled data. The method further includes selecting a representative image for labeling from each of the clusters of unlabeled image data. Selecting the representative image includes selecting the representative image based on a medoid of a corresponding one of the clusters of unlabeled image data. Generating the unsupervised learning prompt includes computing the instance-level contrastive loss and the cluster-level contrastive loss.

[0005] A computing device configured to perform data pre-qualification for an object detection system includes a processing device configured to execute instructions stored in a memory to receive a first data set including unlabeled data corresponding to one or more images, and provide the first data set and a plurality of learnable prompt vectors to a pre-training model. The learnable prompt vectors include text inputs.The processing device is further configured to execute instructions for generating, using the pre-training model, an unsupervised learning prompt based on the first data set and the plurality of learnable prompt vectors, wherein the unsupervised learning prompt corresponds to a multimodal feature of the one or more images of the first data set, extracting features from one of the first data set and a second data set based on the unsupervised learning prompt, selecting and labeling a subset of instances of the extracted features, and generating and outputting a labeled data set based on the labeled subset of instances.

[0006] For other features, the pre-training model is a Bootstrapping Language Image Pre-Training (BLIP-2) model. The extracted features include clusters of unlabeled image data. The selected subset of instances includes one or more of the clusters of unlabeled data. The processing device is configured to execute instructions for selecting a representative image for labeling from each of the clusters of unlabeled image data. The processing device is configured to execute instructions for selecting the representative image based on a medoid of a corresponding one of the clusters of unlabeled image data. The processing device is configured to execute instructions for calculating the instance-level contrastive loss and the cluster-level contrastive loss.

[0007] A computer-controlled machine includes at least one sensor configured to generate an input image, a control system configured to perform data pre-selection for an object detection system, wherein the control system is configured to receive a first data set including unlabeled data corresponding to one or more images, provide the first data set and a plurality of learnable prompt vectors including text inputs to a pre-training model, generate, using the pre-training model and based on the first data set and the plurality of learnable prompt vectors, an unsupervised learning prompt corresponding to a multimodal feature of the one or more images of the first data set, extract features from one of the first data set and a second data set based on the unsupervised learning prompt,Selecting and labeling a subset of instances of the extracted features and generating and outputting a labeled dataset based on the labeled subset of instances, and an actuator configured to control operation of the computer-controlled machine based on the labeled dataset.

[0008] For other features, the pre-training model is a Bootstrapping Language Image Pre-Training (BLIP-2) model. The extracted features include clusters of unlabeled image data, and the selected subset of instances includes one or more of the clusters of unlabeled data. The control system is configured to select a representative image for labeling from each cluster of unlabeled image data. The control system is configured to select the representative image based on a medoid of a corresponding one of the clusters of unlabeled image data. The control system is configured to compute the instance-level contrastive loss and the cluster-level contrastive loss. Short description of the drawings Fig. 1 generally illustrates a system for training a neural network according to the principles of the present disclosure. Fig. 2 generally illustrates a computer-implemented method for training and utilizing a neural network in accordance with the principles of the present disclosure. Fig. 3A generally illustrates an audio data labeling system according to the principles of the present disclosure. Fig. 3B generally illustrates a portion of a data acquisition system according to the principles of the present disclosure. Fig. 3C generally illustrates an alternative audio data labeling system according to the principles of the present disclosure. Fig. 4A generally illustrates an exemplary data pre-selection system in accordance with the present disclosure. Fig. 4B generally illustrates data pre-selection according to the systems and methods of the present disclosure. Fig. 4C generally illustrates results of unsupervised prompt learning for data pre-selection (UP-DP) systems and methods of the present disclosure for downstream tasks. Fig. 4D illustrates steps of an exemplary data pre-selection method according to the principles of the present disclosure. Fig. 5 shows a schematic diagram of an interaction between a computer-controlled machine and a control system according to the principles of the present disclosure. Fig. Figure 6 shows a schematic diagram of the control system of Fig. 5, configured to control a vehicle, which may be a partially autonomous vehicle, a fully autonomous vehicle, a partially autonomous robot, or a fully autonomous robot, in accordance with the principles of the present disclosure. Fig. Figure 7 shows a schematic diagram of the control system of Fig. 5, designed to control a manufacturing machine, such as a punch-cutting device, a cutting device or a gun drill, of a manufacturing system, such as part of a production line. Fig. Figure 8 shows a schematic diagram of the control system of Fig. 5, designed to control a power tool, such as a drill or a cordless drill, having an at least partially autonomous mode. Fig. Figure 9 shows a schematic diagram of the control system of Fig. 5, designed to control an automated personal assistant. Fig. Figure 10 shows a schematic diagram of the control system of Fig. 5, designed to control a monitoring system, such as an access control system or a surveillance system. Fig. Figure 11 shows a schematic diagram of the control system of Fig. 5, designed to control an imaging system, for example an MRI device, an X-ray imaging device or an ultrasound device. Detailed description

[0009] Embodiments of the present disclosure are described herein. It should be understood, however, that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The figures are not necessarily to scale; some features may be exaggerated or minimized to show details of particular components. Therefore, specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching those skilled in the art to variously employ the embodiments.Those of ordinary skill in the art will recognize that various features illustrated and described with reference to any of the figures may be combined with features illustrated in one or more other figures to produce embodiments not explicitly illustrated or described. The combinations of illustrated features provide representative embodiments for a typical application. However, various combinations and modifications of the features consistent with the teachings thereof may be desirable for particular applications or implementations.

[0010] "A" and "the," as used herein, refers to both singular and plural references unless the context clearly indicates otherwise. For example, "a processor" programmed to perform various functions refers to one processor programmed to perform each and all of the functions, or to more than one processor programmed together to perform each of the various functions.

[0011] Vision language machine learning models for object detection systems and methods are trained using subsets of data samples. Data-efficient machine learning, which aims to identify the best subsets of data samples (either labeling them or not) to achieve optimal model performance, has emerged as a crucial research area in the era of data-hungry deep learning.

[0012] Semi-supervised learning (SSL) and active learning (AL) are two exemplary approaches for improving data-efficient learning. SSL aims to solve the problem of label shortages by leveraging a limited set of labeled data in conjunction with a larger pool of unlabeled data to improve model performance. In contrast, AL approaches start with an initial set of labeled data and select the most informative data point for labeling to maximize model performance within a limited label budget. However, existing AL methods initially require a labeled set and require training a series of models.

[0013] Accordingly, the task of data pre-selection presents several challenges that SSL and AL cannot adequately address. In data pre-selection, the initial data is unlabeled, and specific downstream tasks are unknown. For example, in object detection, the categories of objects to be detected are unknown. However, SSL and AL methods require both initial labeled data and knowledge of specific learning tasks. Therefore, a method for selecting diverse and representative instances is needed to cover the entire dataset with a minimal number of required labels.

[0014] In some examples, basic vision language (VL) models, such as contrastive language image pre-training (CLIP) models and bootstrapping language image pre-training (BLIP-2) models, can solve some problems associated with SSL and AL methods. These VL models acquire comprehensive semantic knowledge by learning from a vast collection of image-text pairs. Consequently, VL models can potentially extract meaningful features from diverse input modalities. While VL models can be adapted for various tasks such as few-shot classification, semi-supervised learning, and selective labeling, such models may focus on unimodal features (e.g., vision) and inadvertently overlook possible contributions from multimodal features that encompass both vision and language.

[0015] Designing a suitable prompt for VL models can be challenging and time-consuming, often requiring trial and error. Furthermore, various automatic prompt learning approaches typically rely on a small set of labeled or pseudo-labeled data, which is impractical for data pre-selection when the downstream tasks are undefined.

[0016] Data pre-selection systems and methods according to the present disclosure incorporate unsupervised prompt learning in a vision language model for data pre-selection. In particular, systems and methods described herein implement prompt learning techniques designed to select instances for labeling from an unlabeled dataset via a single pass, thereby optimizing performance for undefined downstream tasks within a limited annotation budget. Previous approaches to data pre-selection were based exclusively on visual features extracted from baseline models (e.g., CLIP and BLIP-2 models as described above), but largely ignored the importance of textual features. Pure, unsupervised prompt learning according to the present disclosure aims to enhance the multimodal features extracted from VL models for data pre-selection.

[0017] As described herein, the prompt can be designed so that the multimodal features extracted by VL models provide a more effective representation for data pre-screening. The shared feature space of vision and text can provide a better representation for data pre-screening. Unsupervised Prompt Learning for Data Pre-screening ("UP-DP") according to the present disclosure implements an unsupervised prompt learning approach in which vision language pre-training models (e.g., BLIP-2 models) are adapted for data pre-screening. Specifically, with fixed or frozen BLIP-2 parameters, text prompts are trained to extract the shared features with improved representation, thus ensuring a diverse cluster structure that covers the entire dataset.The prompts learned from an example dataset show significant generalizability and can be directly applied to improve the feature extraction of BLIP-2 models from other datasets.

[0018] Fig. 1 shows an exemplary system 100 for training a neural network (e.g., an ML model). The system 100 may be configured (and / or include circuitry configured) to implement the systems and methods of the present disclosure described in more detail below. The system 100 may include an input interface for accessing training data 102 for the neural network. For example, as shown in Fig. 1, the input interface may consist of a data storage interface 104 that can access the training data 102 from a data storage 106. For example, the data storage interface 104 may be a memory or persistent storage interface, e.g., a hard disk or SSD interface, but may also be a personal, local, or wide area network interface, such as a Bluetooth, Zigbee, or Wi-Fi interface, or an Ethernet or fiber optic interface. The data storage 106 may be an internal data storage of the system 100, such as a hard disk or SSD, but may also be an external data storage, e.g., a network-accessible data storage.

[0019] In some embodiments, the data store 106 may further include a data representation 108 of an untrained version of the neural network, which the system 100 may access from the data store 106. However, it should be understood that the training data 102 and the data representation 108 of the untrained neural network may also be accessed from another data store, e.g., via another subsystem of the data store interface 104. The nature of each subsystem may be as described above for the data store interface 104.

[0020] In some embodiments, the data representation 108 of the untrained neural network may be generated internally by the system 100 based on design parameters for the neural network and may therefore not be explicitly stored in the data store 106. The system 100 may further include a processor subsystem 110 that may be configured to provide an iterative function as a replacement for a stack of layers of the neural network to be trained during operation of the system 100. Here, the respective layers of the layer stack being replaced may have mutually shared weights and receive as input an output of a previous layer or, for a first layer of the layer stack, an initial activation and a portion of the input of the layer stack.

[0021] The processor subsystem 110 may be further configured to iteratively train the neural network using the training data 102. Here, an iteration of the training by the processor subsystem 110 may include a forward propagation portion and a backward propagation portion. The processor subsystem 110 may be configured to perform the forward propagation portion by, among other operations that may be performed defining the forward propagation portion, determining an equilibrium point of the iterative function at which the iterative function converges to a fixed point, wherein determining the equilibrium point comprises using a numerical root-finding algorithm to find a root solution for the iterative function minus its input, and providing the equilibrium point as a substitute for an output of the layer stack in the neural network.

[0022] The system 100 may further include an output interface for outputting a data representation 112 of the trained neural network. This data may also be referred to as trained model data 112. For example, as also shown in Fig. 1, the output interface may consist of the data storage interface 104, wherein the interface in these embodiments is an input / output ("IO") interface via which the trained model data 112 may be stored in the data storage 106. For example, the data representation 108 defining the "untrained" neural network may be at least partially replaced by the data representation 112 of the trained neural network during or after training, in the sense that the parameters of the neural network, such as weights, hyperparameters, and other types of neural network parameters, may be adjusted to reflect the training on the training data 102. This is also illustrated in Fig. 1 by reference numerals 108, 112, which refer to the same data set in the data store 106. In some embodiments, the data representation 112 may be stored separately from the data representation 108 that defines the "untrained" neural network. In some embodiments, the output interface may be separate from the data store interface 104, but may generally be of a type as described above for the data store interface 104.

[0023] Fig. 2 shows a data annotation / enhancement system 200 configured (and / or including circuitry configured) to implement a system for annotating and / or enhancing data. The data annotation system 200 may include at least one computing system 202 configured to implement all or portions of the systems and methods of the present disclosure, which are explained in more detail below. The computing system 202 may include at least one processor 204 operatively coupled to a memory device 208. The processor 204 may include one or more integrated circuits implementing the functionality of a central processing unit (CPU) 206. The CPU 206 may be a commercially available processing unit implementing an instruction set, such as one of the x86, ARM, Power, or MIPS instruction set families.Various components of the system 200 may be implemented with the same or different circuit arrangements.

[0024] During operation, CPU 206 may execute stored program instructions retrieved from memory unit 208. The stored program instructions may include software that controls the operation of CPU 206 to perform the operation described herein. In some embodiments, processor 204 may be a system on a chip (SoC) that integrates the functionality of CPU 206, memory unit 208, a network interface, and input / output interfaces into a single integrated device. Computing system 202 may implement an operating system to manage various aspects of operation.

[0025] The storage unit 208 may include volatile memory and non-volatile memory for storing instructions and data. The non-volatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when the computing system 202 is disabled or disconnected from power. The volatile memory may include static or dynamic random access memory (RAM) that stores program instructions and data. For example, the storage unit 208 may include a machine learning model 210 (e.g., in Fig. 2 as ML model 210) or algorithm, a training dataset 212 for the machine learning model 210, raw source dataset 216, etc.

[0026] Computing system 202 may include a network interface device 222 configured to provide communication with external systems and devices. For example, network interface device 222 may include a wired and / or wireless Ethernet interface according to the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards. Network interface device 222 may include a cellular communication interface for communicating with a cellular network (e.g., 3G, 4G, 5G). Network interface device 222 may be configured to provide a communication interface with an external network 224 or a cloud.

[0027] The external network 224 may be referred to as the World Wide Web or the Internet. The external network 224 may establish a standard communication protocol between computing devices. The external network 224 may enable information and data to be easily exchanged between computing devices and networks. One or more servers 230 may be in communication with the external network 224.

[0028] Computing system 202 may include an input / output (I / O) interface 220, which may be configured to provide digital and / or analog inputs and outputs. I / O interface 220 may include additional serial interfaces for communicating with external devices (e.g., the Universal Serial Bus (USB) interface).

[0029] Computing system 202 may include a human-machine interface (HMI) device 218, which may include any device that enables system 200 to receive control input. Examples of input devices may include human interface inputs such as keyboards, mice, touch screens, voice input devices, and other similar devices. Computing system 202 may include a display device 232. Computing system 202 may include hardware and software for outputting graphic and text information to display device 232. Display device 232 may include an electronic display screen, a projector, a printer, or other suitable device for displaying information to a user. Computing system 202 may be further configured to enable interaction with remote HMI and remote display devices via network interface device 222.

[0030] System 200 may be implemented using one or more computing systems. Although the example illustrates a single computing system 202 implementing all of the described features, it is contemplated that various features and functions may be separated and implemented by multiple, communicating computing units. The particular system architecture selected may depend on a variety of factors.

[0031] The system 200 may implement a machine learning model 210 configured to analyze the raw source data set 216. For example, the CPU 206 and / or other circuitry may implement the machine learning model 210. The raw source data set 216 may include raw or unprocessed sensor data, which may represent an input data set for a machine learning system. The raw source data set 216 may include video, video segments, images, audio, text-based information, and raw or partially processed sensor data (e.g., radar map of objects). In some embodiments, the machine learning model 210 may be a deep learning or neural network algorithm configured to perform a predetermined function. For example, the neural network algorithm may be configured to identify events or objects in video segments based on audio data.

[0032] Computer system 200 may store the training dataset 212 for the machine learning model 210. The training dataset 212 may represent a set of previously created data for training the machine learning model 210. The training dataset 212 may be used by the machine learning model 210 to learn weighting factors associated with a neural network algorithm. The training dataset 212 may include a set of source data having corresponding outcomes or results that the machine learning model 210 attempts to duplicate through the learning process.

[0033] The machine learning model 210 may operate in a learning mode using the training dataset 212 as input. The machine learning model 210 may be executed using the data from the training dataset 212 over a series of iterations. During each iteration, the machine learning model 210 may update internal weighting factors based on the results obtained. For example, the machine learning model 210 may compare the output results (e.g., annotations) with those contained in the training dataset 212. Because the training dataset 212 includes the expected results, the machine learning model 210 may determine when performance is acceptable. After the machine learning model 210 has reached a predetermined level of performance (e.g.,100% agreement with the results associated with the training dataset 212), the machine learning model 210 may be executed using data not included in the training dataset 212. The trained machine learning model 210 may be applied to new datasets to generate annotated data.

[0034] The machine learning model 210 may be configured to identify a particular feature in the raw source data set 216. The raw source data set 216 may include a plurality of instances or input data sets for which annotation results are desired (e.g., a video stream or a video segment including audio data). For example, the machine learning model 210 may be configured to identify objects or events in a video segment based on audio data and annotate the events. The machine learning model 210 may be programmed to process the raw source data 216 to identify the presence of the particular features. The machine learning model 210 may be configured to identify a feature in the raw source data 216 as a predetermined feature. The raw source data 216 may originate from a variety of sources.For example, the raw source data 216 may be actual input data acquired by a machine learning system. The raw source data 216 may be machine-generated for testing the system. For example, the raw source data 216 may include raw video and / or audio data from a camera, audio data from a microphone, etc.

[0035] In one example, the machine learning model 210 may process the raw source data 216 and output video and / or audio data including one or more indications of an identified event. The machine learning model 210 may generate a confidence level or confidence factor for each generated output. For example, a confidence value that exceeds a predetermined high confidence threshold may indicate that the machine learning model 210 is certain that the identified event (or feature) corresponds to the particular event. A confidence value that is less than a low confidence threshold may indicate that the machine learning model 210 has some uncertainty regarding the presence of the particular feature.

[0036] As is generally the case in Fig. 3A and Fig. 3B, a system 300 may include an image (e.g., video) capture device 302, an audio capture array 304, and the computing system 202. The system may receive video stream data associated with a data capture environment from the image capture device 302. The system 202 may be configured to perform video object detection to identify one or more objects in corresponding frames of the video stream data. The system 202 may receive audio stream data corresponding to at least a portion of the video stream data from the audio capture array 304. The audio capture array 304 may include one or more microphones 306 or other suitable audio capture devices. The systems and methods described herein may be configured to perform video object detection using outputs from at least a first machine learning model (e.g.,the machine learning model 210 or another suitable machine learning model configured to provide an output including one or more object or event detection predictions) to label at least some objects of the video stream data and / or audio stream data.

[0037] The system 202 may calculate (e.g., using at least one probabilistic function or other suitable technique or function) at least one offset value for at least a portion of the audio stream data corresponding to at least one labeled object of the video stream data based on at least one data acquisition characteristic. The system 202 may synchronize at least a portion of the video stream data with the portion of the audio stream data corresponding to the at least one labeled object of the video stream data using at least the at least one offset value.The at least one data acquisition characteristic may include one or more characteristics of the at least one image acquisition device, one or more characteristics of the at least one audio acquisition array, one or more characteristics corresponding to a position of the at least one image acquisition device relative to the at least one audio acquisition device, one or more characteristics corresponding to a movement of an object in the video stream data, one or more other suitable data acquisition characteristics, or a combination thereof.

[0038] The system 202 may label at least the portion of the audio stream data corresponding to the at least one labeled object of the video stream data using one or more labels of the labeled objects of the video stream data and the at least one offset value. Each respective label may include an event type, an event start indicator, and an event end indicator. The system 202 may generate training data using at least some of the labeled portion of the audio stream data. The system 202 may train a second machine learning model using the training data. The system 202 may detect one or more tones associated with audio data provided as input to the second machine learning model using the second machine learning model.The second machine learning model may include any suitable machine learning model and be configured to perform any suitable function, such as those described herein with respect to . Fig. 4-11 described.

[0039] In some embodiments, as generally in Fig. 3C, the computing system 202 may be configured to label audio data based on sensor data received from one or more sensors, such as those described herein or any other suitable sensors or combination of sensors. The system 202 may receive audio stream data associated with a data collection environment from the audio acquisition array 354 or any suitable audio acquisition device, e.g., one or more of the microphones 306 or other suitable audio acquisition device. It is understood that the audio acquisition array 354 may have similar features to the audio acquisition array 304 and may include any suitable number of audio acquisition devices. The system 202 may receive sensor data associated with the data collection environment from at least one sensor (such as the sensor 352) that is asynchronous with the audio acquisition array 354.The sensor 354 may include at least one of an induction coil, a radar sensor, a LiDAR sensor, a sonar sensor, an image capture device, any other suitable sensor, or a combination thereof. The audio capture array 354 may be located remote from the sensor 354, proximate to the sensor 354, or in any suitable relationship to the sensor 354.

[0040] The system 202 may identify at least some events in the sensor data using outputs from at least a first machine learning model, such as the machine learning model 210 or another suitable machine learning model. The machine learning model 210 may be configured to provide an output including one or more event detection predictions based on the sensor data. The system 202 may synchronize at least a portion of the sensor data associated with the portion of the audio stream data corresponding to the at least one event of the sensor data. The system 202 may label at least the portion of audio stream data corresponding to the at least one event of the sensor data using one or more labels extracted for respective events of the sensor data value. Each respective label may include an event type, an event start indicator, and an event end indicator.System 202 may generate training data using at least some of the labeled portion of the audio stream data. System 202 may train a second machine learning model using the training data. System 202 may detect, using the second machine learning model, one or more tones associated with audio data provided as input to the second machine learning model. The second machine learning model may include any suitable machine learning model and be configured to perform any suitable function, such as those described herein with respect to

[0034] FIG. Fig. 4-11 described.

[0041] Any of the systems described in more detail above and / or below may implement the data pre-selection systems and methods of the present disclosure, including unsupervised prompt learning techniques in a vision language model for data pre-selection.

[0042] Fig. 4A illustrates an example data pre-selection system 400 according to the present disclosure. Example unimodal and multimodal feature representations obtained from an example VL pre-training model (e.g., a BLIP-2 model) for various ground truth labels are shown as U-map visualizations at 402 (e.g., a unimodal feature representation using only one vision / image feature) and 404 (e.g., a multimodal feature representation using a learned prompt). For comparison, results using a random prompt are shown at 406. In particular, when class information of downstream tasks is unknown, the unimodal features are shuffled, as shown at 402, making identification of specific classes difficult.In contrast, when paired with a suitable prompt, the multimodal features exhibit a more scattered and distinct distribution, as shown at 404, allowing for better discrimination between classes. Accordingly, as shown at 404, a VL model with a suitable prompt can provide visually superior multimodal features compared to unimodal features, as shown at 402, which can serve as an effective starting point for data pre-selection. This improved discrimination is crucial for users to identify the most representative and diverse instances during the data pre-selection process.

[0043] The goal of data pre-selection is to select instances for labeling from an unlabeled dataset 408 (e.g., an OxfordPets training dataset) over a single pass to maximize model performance for unknown downstream vision tasks (i.e., without knowledge of prediction categories) while maintaining a limited annotation budget. This task is motivated by two practical needs: (i) in the data acquisition phase, it is desirable to collect a minimal amount of data to support potentially diverse downstream tasks (e.g., classification, action recognition, detection for videos or images, etc.) and training paradigms (e.g., active learning, semi-supervised learning, supervised learning, etc.); and (ii) in the annotation phase, the goal is to achieve a desired level of performance with fewer human annotations over the pre-selected data.

[0044] As in Fig. As shown in Figure 4A, the unlabeled data set 408 is provided to a data set pre-selection module or circuitry 410 configured to perform data pre-selection for a data efficiency learning task, as described in more detail below. The data efficiency learning task according to the present disclosure includes multimodal features using a learned prompt and differs from semi-supervised learning tasks and active learning tasks, and does not require seed labels and information about downstream tasks.

[0045] The data pre-selection according to the systems and methods of the present disclosure is described in Fig. 4B with continued reference to Fig. 4A. The systems and methods are configured to formally define a data pre-selection task, provide an unsupervised prompt learning framework that assists a pre-training model (e.g., BLIP-2 model) in extracting improved multimodal features for data pre-selection, and use the adapted BLIP-2 model to select instances for downstream tasks. The data pre-selection workflow, an unsupervised prompt learning training process, and generalization of the learned prompts across different datasets are shown at 412, 414, and 416, respectively.

[0046] For the data pre-selection, a very large dataset D consisting of d instances and an annotation budget of l is assumed for illustrative purposes. The task is to select l (l « D) instances for labeling such that undefined downstream tasks trained on such a partially labeled dataset achieve the best performance. Formally, D=(xi,yi)i=1dd Pairs of images x i and a corresponding class label y i (y i = Ø if the label is unknown). D L Denotes a subset of size l of D with known class labels. The goal is to L ⊂ D to capture class labels to maximize the performance of an undefined model running on labeled data D L with or without unlabeled data D U = D \ D LThis data pre-selection task is difficult because no labels are initially present. In other words, the information that maximizes the performance of the undefined downstream task is unknown. Accordingly, the goal is to select l instances that are not only representative but also diverse enough to comprehensively cover the entire dataset, preventing premature loss of valuable information before label acquisition.

[0047] As shown at 412, data pre-selection involves extracting semantically significant and high-quality features from a dataset (e.g., using a feature extractor module or feature extractor circuitry 417) based on an unsupervised learning prompt. For example, feature extractor module 417 may be implemented by circuitry such as dataset pre-selection module 410. In some examples, dataset pre-selection module 410 implements one or more components configured to perform functions of the unsupervised prompt learning shown at 414. The extracted features facilitate the clustering of all unlabeled images into well-structured groups, as shown at 418 (relative to feature clustering without the prompt). A medoid point is then selected from each cluster to serve as a representative image for labeling, as shown at 419.Representativeness is achieved by using medoids, while diversity is achieved by selecting one instance from each cluster. The BLIP-2 model according to the present disclosure is adapted accordingly, as described in more detail below.

[0048] As shown in 414, the unsupervised prompt learning according to the present disclosure is designed to extract desirable features from the dataset using a learnable context 420, a BLIP-2 model 422, and two multilayer perceptron (MLP) heads for both instance-level and cluster-level transformation. These components are jointly optimized using unsupervised clustering objectives (specifically, contrastive learning at the instance-level and cluster-level). After training, the cluster assignments of all instances can be easily derived from the predicted soft labels generated by the cluster-level MLP head. In combination with the multimodal features obtained by the adapted BLIP-2 model, the most representative and diverse instances of the entire dataset D can be effectively identified.

[0049] The BLIP-2 model 422 provides an efficient pretraining strategy that leverages existing pretrained image encoders and large language models while keeping the parameters of the BLIP-2 model frozen. Training a lightweight query transformer (Q-former) bridges the modality gap between image and text. In the representation learning phase, the BLIP-2 model 422 combines the Q-former with a frozen image encoder and performs image-text pair pretraining. With three image-text pretraining objectives (contrastive image-text learning, image-based text generation, and image-text matching), a strong connection is established between the image and the input text (i.e., forcing the BLIP-2 model 422 to extract visual information from the image that is most relevant to the text and automatically generate multimodal output features). This characteristic of the BLIP-2 model 422 offers the possibility of using different prompts (e.g.input text), extract different multimodal features, and improve the data pre-selection process.

[0050] However, selecting a suitable prompt can be challenging and time-consuming, especially during data pre-selection when downstream tasks are unknown. Following the previous prompt learning approach, each context token can be modeled using a continuous vector that can be learned end-to-end from data. In particular, as shown at 414, N learnable prompt representation context vectors are defined, denoted as V = v1, v2, ... v n, each of the same dimension as the word embedding, to serve as text input for the BLIP-2 model 422. Unlike many previous methodologies that leverage cross-entropy loss as their learning objective with labeled data, the systems and methods described here implement unsupervised clustering as a learning objective with respect to undefined downstream tasks. Because the text encoder is differentiable, gradients can be backpropagated to update the context vectors. The base model of the BLIP-2 model 422 remains unchanged during the training process.

[0051] The output of the BLIP-2 model 422 can be influenced by both instance-level contrastive loss (“instance loss”) and cluster-level contrastive loss (“cluster loss”). Regarding instance loss, instance-level contrastive learning is designed to maximize the similarities between positive instance pairs while minimizing the similarities of negative pairs. Since no labels are present for any data, the goal is to maximize the similarity between differently augmented views of the same instance and treat all other augmented instances as negative examples. In particular, a randomly sampled mini-batch of an image of N images is considered. Each image, with x i is subjected to a different expansion twice, resulting in two views of the same example: xia and xib. These two images are then encoded together with a learnable context V according to f(·) of the BLIP-2 model 422 to generate multimodal representations: hia and hib. The representations are then transformed using a nonlinear transformation MLP g I (·), as shown at 424, which is called the instance level header, which zia and zib Here, sim(.,.) represents the cosine similarity. Then, the contrastive loss at the instance level of a positive pair of examples (i, j) ∈ [1, N] is defined as follows: lia=−logexp(sim(zia,zib) / τl)∑k∈{a,b}∑j=1Nexp(sim(zia,zjk) / τl), where τ I the temperature scale. The final loss corresponds to LI=12N∑iN(lia+lib), which is calculated over all positive pairs in a mini-batch.

[0052] Similar to instance-level contrastive learning, cluster-level contrastive learning aims to maximize the similarities between positive cluster pairs while minimizing the negative pairs. To create a cluster representation, another cluster-level head is used. C (·) is used to map the multimodal representation to an M-dimensional space, where M is the number of clusters. Thus, an m-th element of the feature can be interpreted as the probability that an instance belongs to the m-th cluster. For a mini-batch of N samples and M predefined clusters, the output of the cluster head is C a ∈ R N×M under the first extension, and thus Cn,ma can be interpreted as the probability that sample $n$ under extension $a$ is assigned to cluster m. The i-th column of C acan be treated as a representation of an i-th cluster under extension a. Thus, a positive cluster pair can be formed by subtracting the i-th column of C a and C b , referred to as c^ia and c^ib, is selected while the other 2M - 2 pair is left negative, as shown below: lia^=−logexp((c^ia,c^ib) / τC)∑k∈{a,b}∑j=1Nexp((c^ia,c^ik) / τC), where τ C is the cluster-level temperature parameter. To avoid a trivial solution where most instances are assigned to the same cluster, a regularization is added that boosts the entropy of the cluster assignment probability, as shown below: H(C)=−∑iM[P(c^ia)log P(c^ia)+P(c^ib)log P(c^ib)], where P(c^ik)=1N∑t=1NC^tik, k ∈ {a, b}. The final loss at cluster level is LC=12M∑i=1M(lla^+llb^)−H(C).

[0053] In this way, as described above and in Fig. As shown in Figure 4B, the optimization of the training of the BLIP-2 model 422 is a single-step, end-to-end process. The prompt and two heads are optimized simultaneously, and the overall objective function is the combination of contrastive loss at both the instance level and cluster level (e.g., L=L1+LC ).

[0054] After training, each instance is assigned a cluster number by the adapted BLIP-2 model 422 and the cluster-level MLP head, resulting in several sampling strategies. First, a cluster-level head g C (·) predicted probability is used for sampling. In this case, for each cluster, the instance with the highest confidence value is selected. Conversely, the medoid of each cluster can be selected to be labeled for the downstream task. Given the cluster X k with N kElements {x1, x2 ... x Nk} is the medoid m k the instance that represents the minimum average inequality to all instances computed on the multimodal feature h, or of the form f(·) or the project feature z for g I (·).

[0055] As shown at 416, the learned prompts obtained at 412 and 414 can be generalized across different datasets. For example, although the learned prompts (N learnable prompt representation context vectors V) originate from a dataset of pets, these same prompts can be used for dataset pre-selection for another dataset (e.g., a flower image dataset). As shown at 426, feeding the learned prompts obtained based on the pets dataset to the BLIP-2 model 422 results in improved clustering of features detected in the flower image dataset.

[0056] Fig. 4C generally illustrates results of the unsupervised prompt learning for data pre-selection (UP-DP) systems and methods of the present disclosure for downstream tasks, particularly using selected labeled instances for image classification. For example, Fig. 4C provides an intuitive understanding of the operation of UP-DP systems and methods by visualizing the learned feature representations and cluster assignment of all instances using 2D projection with Uniform Manifold Approximation and Projection (UMAP). The features extracted using the BLIP-2 model 422 with learned prompts demonstrate discrimination capabilities for distinguishing between different types of dogs and cats. For example, for a Shiba group 428, cluster assignments generated by the present cluster head categorize Shibas into distinct clusters based on background and posture, as shown at 430. The multimodal feature is extracted using the BLIP-2 model 422 with learned prompts. The features extracted with the learned prompts can distinguish between different types of pets even without the labels shown, as shown at 432.In other words, the extracted features capture more detailed information such as background, pet ownership, etc. within the same breed or species. Accordingly, as shown in . Fig. As shown in Figure 4C, UP-DP systems and methods effectively extract diverse representative points from unlabeled image data.

[0057] In this way, as described here, a learned prompt obtained using one dataset can be used across various other datasets, significantly improving feature extraction when applied in a plug-and-play manner.

[0058] Fig. 4D illustrates steps of an exemplary data pre-selection method 440 according to the principles of the present disclosure. For example, one or more processors or processing devices are configured to execute instructions to implement method 440, e.g., one or more of the processors of the systems described herein. At 442, method 440 receives an unlabeled data set. At 444, the unlabeled data set is provided to a pre-training model (e.g., a BLIP-2 model, as described above). At 446, a plurality of learnable prompt vectors (e.g., vectors V, as defined above) are provided to the pre-training model (e.g., as text inputs to the BLIP-2 model).

[0059] At 448, the method 440 (e.g., the pre-training model) generates and optimizes an unsupervised learning prompt corresponding to a multimodal feature based on the unlabeled dataset and the learnable prompt vectors. The unsupervised learning prompt represents an optimization of the learnable prompt vectors. At 450, the method 440 (e.g., the feature extractor module 417) extracts features from a dataset (e.g., the same or different dataset as the dataset fed to the pre-training model) based on the unsupervised learning prompt. For example, the extracted features correspond to clusters of unlabeled image data (e.g., unlabeled images), as shown at 418. At 452, the method 440 selects a subset of instances (“selected instances”) of the unlabeled data to be labeled. In this way, the most valuable features for labeling are selected from the unlabeled data.

[0060] At 454, labeling is performed on the selected subset of instances. At 456, labeling is performed on remaining unlabeled image data (i.e., instances that are not labeled at 454) based on the labeled selected subset of instances. For example, the labeling performed at 456 may involve using one or more classifiers (e.g., as at 458 in Fig. 4A) and can be implemented using supervised learning, semi-supervised learning, active learning, or combinations thereof. At 460, method 440 outputs and / or stores a labeled data set.

[0061] Fig. 5-11 illustrate exemplary systems and devices that may implement UP-DP systems and methods according to the present disclosure. Fig. 5 shows a schematic diagram of an interaction between a computer-controlled machine 500 and a control system 502. The computer-controlled machine 500 includes an actuator 504 and a sensor 506. The actuator 504 may include one or more actuators, and the sensor 506 may include one or more sensors. The sensor 506 is configured to sense a state of the computer-controlled machine 500. The sensor 506 may be configured to encode the sensed state into sensor signals 508 and transmit the sensor signals 508 to the control system 502. Non-limiting examples of the sensor 506 include video, radar, LiDAR, ultrasonic, and motion sensors. In some embodiments, the sensor 506 is an optical sensor configured to capture optical images of an environment proximate the computer-controlled machine 500.

[0062] The control system 502 is configured to receive sensor signals 508 from the computer-controlled machine 500. As explained below, the control system 502 may be further configured to calculate actuator control commands 510 depending on the sensor signals and to transmit the actuator control commands 510 to the actuator 504 of the computer-controlled machine 500.

[0063] As in Fig. 5, the control system 502 includes a receiving unit 512. The receiving unit 512 may be configured to receive sensor signals 508 from the sensor 506 and convert the sensor signals 508 into input signals x. In an alternative embodiment, sensor signals 508 are received directly as input signals x without the receiving unit 512. Each input signal x may be a part of each sensor signal 508. The receiving unit 512 may be configured to process each sensor signal 508 to generate each input signal x. The input signal x may include data corresponding to an image recorded by the sensor 506.

[0064] The control system 502 includes a classifier 514. The classifier 514 may be configured to classify input signals x into one or more labels using a machine learning (ML) algorithm, such as a neural network. For example, the classifier 514 corresponds to the classifier 408 described above. The classifier 514 is configured to be parameterized by parameters, such as those described above (e.g., parameter θ). Parameter θ may be stored in and provided by non-volatile storage 516. The classifier 514 is configured to determine output signals y based on input signals x. Each output signal y includes information that assigns one or more labels to each input signal x. The classifier 514 may transmit output signals y to the conversion unit 518.The conversion unit 518 is configured to convert output signals y into actuator control commands 510. The control system 502 is configured to transmit the actuator control commands 510 to the actuator 504, which is configured to actuate the computer-controlled machine 500 in response to the actuator control commands 510. In some embodiments, the actuator 504 is configured to actuate the computer-controlled machine 500 directly based on output signals y.

[0065] Upon receipt of the actuator control commands 510 by the actuator 504, the actuator 504 is configured to perform an action corresponding to the associated actuator control command 510. The actuator 504 may include control logic configured to convert the actuator control commands 510 into a second actuator control command used to control the actuator 504. In one or more embodiments, the actuator control commands 510 may be used to control a display instead of, or in addition to, an actuator.

[0066] In some embodiments, the control system 502 includes the sensor 506 instead of or in addition to the computer-controlled machine 500 including the sensor 506. The control system 502 may also include the actuator 504 instead of or in addition to the computer-controlled machine 500 including the actuator 504.

[0067] As in Fig. 5, control system 502 also includes a processor 520 and a memory 522. Processor 520 may include one or more processors. Memory 522 may include one or more storage devices. Classifier 514 (e.g., ML algorithms) of one or more embodiments may be implemented by control system 502, which includes non-volatile memory 516, processor 520, and memory 522.

[0068] Non-volatile memory 516 may include one or more persistent data storage devices, such as a hard drive, an optical drive, a tape drive, a non-volatile solid-state device, cloud storage, or any other device capable of persistently storing information. Processor 520 may include one or more devices selected from high-performance computing (HPC) systems, including high-performance cores, microprocessors, microcontrollers, digital signal processors, microcomputers, central processing units, field-programmable gate arrays, programmable logic devices, state machines, logic circuits, analog circuits, digital circuits, or any other devices that manipulate (analog or digital) signals based on computer-executable instructions located in memory 522.The memory 522 may include a single memory device or a series of memory devices, including, but not limited to, random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information.

[0069] Processor 520 may be configured to read into memory 522 and execute computer-executable instructions located in non-volatile memory 516 that implement one or more anomaly detection methodologies of one or more embodiments. Non-volatile memory 516 may include one or more operating systems and applications. Non-volatile memory 516 may store compiled and / or interpreted computer programs created using a variety of programming languages ​​and / or technologies, including, among others, and either alone or in combination, Java, C, C++, C#, Objective-C, Fortran, Pascal, Java Script, Python, Perl, and PL / SQL.

[0070] When executed by processor 520, the computer-executable instructions of non-volatile memory 516 may cause control system 502 to implement one or more of the anomaly detection methodologies disclosed herein. Non-volatile memory 516 may also include data supporting functions, features, and processes of one or more embodiments described herein.

[0071] The program code implementing the algorithms and / or methodologies described herein may be distributed individually or collectively in a variety of different forms as a program product. The program code may be distributed using a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to perform aspects of one or more embodiments. Computer-readable storage media, which are inherently non-transitory, may include volatile and non-volatile, and removable and non-removable tangible media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data.Computer-readable storage media may further include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state storage technology, portable compact disc read-only memory (CD-ROM) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be read by a computer. Computer-readable program instructions may be downloaded from a computer-readable storage medium to a computer, other type of programmable data processing device, or other device, or over a network to an external computer or external storage device.

[0072] Computer-readable program instructions stored on a computer-readable medium may be used to instruct a computer, other types of programmable data processing equipment, or other devices to operate in a specific manner such that the instructions stored on the computer-readable medium produce an article of manufacture including instructions that implement the functions, acts, and / or operations specified in the flowcharts or diagrams. In certain alternative embodiments, the functions, acts, and / or operations specified in the flowcharts and diagrams may be reordered, processed serially, and / or processed concurrently in accordance with one or more embodiments.Additionally, any of the flowcharts and / or diagrams may include more or fewer nodes or blocks than those illustrated in accordance with one or more embodiments.

[0073] The processes, methods or algorithms may be implemented in whole or in part using suitable hardware components, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), state machines, controllers or other hardware components or devices, or a combination of hardware, software and firmware components.

[0074] Fig. 6 shows a schematic diagram of a control system 502 configured to control a vehicle 600, which may be an at least partially autonomous vehicle or an at least partially autonomous robot. The vehicle 600 includes an actuator 504 and a sensor 506. The sensor 506 may include one or more video sensors, cameras, radar sensors, ultrasonic sensors, LiDAR sensors, and / or position sensors (e.g., GPS). One or more of the one or more specific sensors may be integrated into the vehicle 600. Alternatively, or in addition to one or more specific sensors identified above, the sensor 506 may include a software module configured, when executed, to determine a state of the actuator 504.A non-limiting example of a software module includes a weather information software module configured to determine current or future weather conditions near the vehicle 600 or another location.

[0075] The classifier 514 of the control system 502 of the vehicle 600 may be configured to detect objects in the vicinity of the vehicle 600 based on input signals x. In such an embodiment, the output signal y may include information characterizing the surroundings of objects of the vehicle 600. The actuator control command 510 may be determined according to this information. The actuator control command 510 may be used to avoid collisions with the detected objects.

[0076] In embodiments where vehicle 600 is an at least partially autonomous vehicle, actuator 504 may be implemented as a brake, drive system, engine, powertrain, or steering system of vehicle 600. Actuator control commands 510 may be determined to control actuator 504 such that vehicle 600 avoids collisions with detected objects. Detected objects may also be classified according to what they are most likely to be according to classifier 514, such as pedestrians or trees. Actuator control commands 510 may be determined depending on the classification. In a scenario where an adversarial attack may occur, the system described above may be further trained to better detect objects or to identify a change in lighting conditions or angle for a sensor or camera on vehicle 600.

[0077] In some embodiments where the vehicle 600 is an at least partially autonomous robot, the vehicle 600 may be a mobile robot configured to perform one or more functions, such as flying, swimming, diving, and walking. The mobile robot may be an at least partially autonomous lawnmower or an at least partially autonomous cleaning robot. In such embodiments, the actuator control command 510 may be determined such that a drive unit, a steering unit, and / or a braking unit of the mobile robot may be controlled such that the mobile robot can avoid collisions with identified objects.

[0078] In some embodiments, the vehicle 600 is an at least partially autonomous robot in the form of a gardening robot. In such an embodiment, the vehicle 600 may use an optical sensor as sensor 506 to determine a condition of plants in an environment near the vehicle 600. The actuator 504 may be a nozzle configured to spray chemicals. Depending on an identified species and / or condition of the plants, the actuator control command 510 may be determined to cause the actuator 504 to spray the plants with an appropriate amount of suitable chemicals.

[0079] The vehicle 600 may be an at least partially autonomous robot in the form of a household appliance. Non-limiting examples of household appliances include a washing machine, a stove, an oven, a microwave, or a dishwasher. In such a vehicle 600, the sensor 506 may be an optical sensor configured to detect a condition of an object to be processed by the household appliance. For example, if the household appliance is a washing machine, the sensor 506 may detect a condition of the laundry in the washing machine. The actuator control command 510 may be determined based on the detected condition of the laundry.

[0080] Fig. Figure 7 shows a schematic diagram of a control system 502 configured to control a system 700 (e.g., manufacturing machine), such as a punch-cutting device, a cutting device, or a gun drill, of a manufacturing system 702, such as part of a production line. The control system 502 may be configured to control an actuator 504 configured to control the system 700 (e.g., manufacturing machine).

[0081] A sensor 506 of the system 700 (e.g., manufacturing machine) may be an optical sensor configured to detect one or more properties of a piece of work 704. A classifier 514 may be configured to determine a state of the piece of work 704 based on one or more of the detected properties. The actuator 504 may be configured to control the system 700 (e.g., manufacturing machine) for a subsequent manufacturing step of the piece of work 704 depending on the determined state of the piece of work 704. The actuator 504 may be configured to control functions of the system 700 (e.g., manufacturing machine) at a next piece of work 706 of the system 700 (e.g., manufacturing machine) depending on the determined state of the piece of work 704.

[0082] Fig. Figure 8 shows a schematic diagram of a control system 502 configured to control a power tool 800, such as a drill or cordless drill, having an at least partially autonomous mode. The control system 502 may be configured to control an actuator 504 configured to control the power tool 800.

[0083] The sensor 506 of the power tool 800 may be an optical sensor configured to detect one or more characteristics of the work surface 802 and / or a fastener 804 being driven into the work surface 802. The classifier 514 may be configured to determine a condition of the work surface 802 and / or the fastener 804 relative to the work surface 802 based on one or more of the detected characteristics. The condition may be that the fastener 804 is flush with the work surface 802. Alternatively, the condition may be the hardness of the work surface 802. The actuator 504 may be configured to control the power tool 800 such that the driving function of the power tool 800 is adjusted depending on the determined condition of the fastener 804 relative to the work surface 802 or one or more detected characteristics of the work surface 802.For example, the actuator 504 may interrupt the driving function when the condition of the fastener 804 is flush with the working surface 802. As another non-limiting example, the actuator 504 may apply additional or lesser torque depending on the hardness of the working surface 802.

[0084] Fig. Figure 9 shows a schematic diagram of control system 502 configured to control an automated personal assistant 900. Control system 502 may be configured to control an actuator 504 configured to control automated personal assistant 900. Automated personal assistant 900 may be configured to control a household appliance, such as a washing machine, a stove, an oven, a microwave, or a dishwasher.

[0085] Sensor 506 may be an optical sensor and / or an audio sensor. The optical sensor may be configured to receive video images of gestures 904 of a user 902. The audio sensor may be configured to receive a voice command from the user 902.

[0086] The control system 502 of the automated personal assistant 900 may be configured to determine actuator control commands 510 configured to control the system 502. The control system 502 may be configured to determine the actuator control commands 510 according to sensor signals 508 from the sensor 506. The automated personal assistant 900 is configured to transmit the sensor signals 508 to the control system 502. The classifier 514 of the control system 502 may be configured to execute a gesture recognition algorithm to identify the gesture 904 performed by the user 902, to determine actuator control commands 510, and to transmit the actuator control commands 510 to the actuator 504. The classifier 514 may be configured to retrieve information from the non-volatile memory in response to the gesture 904 and to output the retrieved information in a form suitable for receipt by the user 902.

[0087] Fig. Figure 10 shows a schematic diagram of control system 502 configured to control a surveillance system 1000. Surveillance system 1000 may be configured to physically control access through a door 1002. Sensor 506 may be configured to detect a scene relevant to determining whether to grant access. Sensor 506 may be an optical sensor configured to generate and transmit image and / or video data. Such data may be used by control system 502 to detect a person's face.

[0088] The classifier 514 of the control system 502 of the surveillance system 1000 may be configured to interpret the image and / or video data by comparing identities of known individuals stored in the non-volatile memory 516, thereby determining an identity of a person. The classifier 514 may be configured to generate an actuator control command 510 in response to the interpretation of the image and / or video data. The control system 502 is configured to send the actuator control command 510 to the actuator 504. In this embodiment, the actuator 504 may be configured to lock or unlock the door 1002 in response to the actuator control command 510. In some embodiments, non-physical, logical access control is also possible.

[0089] The surveillance system 1000 may also be an observation system. In such an embodiment, the sensor 506 may be an optical sensor configured to detect a scene under observation, and the control system 502 is configured to control a display 1004. The classifier 514 is configured to determine a classification of a scene, e.g., whether the scene detected by the sensor 506 is suspicious. The control system 502 is configured to send an actuator control command 510 to the display 1004 in response to the classification. The display 1004 may be configured to adjust the displayed content in response to the actuator control command 510. For example, the display 1004 may highlight an object that the classifier 514 considers suspicious.Using an embodiment of the disclosed system, the observation system can predict the appearance of objects at specific times in the future.

[0090] Fig.11 shows a schematic diagram of control system 502 configured to control an imaging system 1100, such as an MRI scanner, an X-ray scanner, or an ultrasound scanner. Sensor 506 may be, for example, an imaging sensor. Classifier 514 may be configured to determine a classification of all or a portion of the acquired image. Classifier 514 may be configured to determine or select an actuator control command 510 in response to the classification obtained by the trained neural network. For example, classifier 514 may interpret a region of an acquired image as potentially abnormal. In this case, actuator control command 510 may be determined or selected to cause display 1102 to display the imaging and highlight the potentially abnormal region.

[0091] Although exemplary embodiments are described above, these embodiments are not intended to describe all possible forms encompassed by the claims. The terms used in the specification are terms of description rather than limitation, and it is understood that various changes may be made without departing from the spirit and scope of the disclosure. As described above, the features of various embodiments may be combined to form further embodiments of the invention that may not be explicitly described or illustrated.Although various embodiments may have been described as providing advantages or being preferred over other embodiments or prior art implementations with respect to one or more desired characteristics, those of ordinary skill in the art will recognize that one or more features or characteristics may be compromised to achieve desired overall system attributes depending on the specific application and implementation. These attributes may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, maintainability, weight, manufacturability, ease of assembly, etc.Therefore, to the extent that embodiments are described as less desirable than other embodiments or prior art implementations with respect to one or more characteristics, these embodiments are not outside the scope of the disclosure and may be desirable for certain applications.

Claims

[1] A method for performing data pre-selection for an object detection system, the method comprising: Receiving a first data set, the first data set including unlabeled data corresponding to one or more images; Providing the first data set and a plurality of learnable prompt vectors to a pre-training model, wherein the learnable prompt vectors include text inputs; Generating, using the pre-training model, an unsupervised learning prompt based on the first data set and the plurality of learnable prompt vectors, wherein the unsupervised learning prompt corresponds to a multimodal feature of the one or more images of the first data set; Extracting features from one of the first dataset and a second dataset based on the unsupervised learning prompt; Selecting and labeling a subset of instances of the extracted features; and Generate and output a labeled dataset based on the labeled subset of instances. [2] The method of claim 1, wherein the pre-training model is a Bootstrapping Language Image Pre-Training (BLIP-2) model. [3] The method of claim 1, wherein the extracted features include clusters of unlabeled image data. [4] The method of claim 3, wherein the selected subset of instances includes one or more of the clusters of unlabeled data. [5] The method of claim 4, further comprising selecting a representative image for labeling from each of the clusters of unlabeled image data. [6] The method of claim 5, wherein selecting the representative image includes selecting the representative image based on a medoid of a corresponding one of the clusters of unlabeled image data. [7] The method of claim 1, wherein generating the unsupervised learning prompt includes calculating the instance-level contrastive loss and the cluster-level contrastive loss. [8] A computing device configured to perform data pre-selection for an object detection system, the computing device including a processing device configured to execute instructions stored in a memory for: Receiving a first data set, the first data set including unlabeled data corresponding to one or more images; Providing the first data set and a plurality of learnable prompt vectors to a pre-training model, wherein the learnable prompt vectors include text inputs; Generating, using the pre-training model, an unsupervised learning prompt based on the first data set and the plurality of learnable prompt vectors, wherein the unsupervised learning prompt corresponds to a multimodal feature of the one or more images of the first data set; Extracting features from one of the first dataset and a second dataset based on the unsupervised learning prompt; Selecting and labeling a subset of instances of the extracted features; and Generate and output a labeled dataset based on the labeled subset of instances. [9] The computing device of claim 8, wherein the pre-training model is a Bootstrapping Language Image Pre-Training (BLIP-2) model. [10] The computing device of claim 8, wherein the extracted features include clusters of unlabeled image data. [11] The computing device of claim 10, wherein the selected subset of instances includes one or more of the clusters of unlabeled data. [12] The computing device of claim 8, wherein the processing device is configured to execute instructions for selecting a representative image for labeling from each of the clusters of unlabeled image data. [13] The computing device of claim 12, wherein the processing device is configured to execute instructions for selecting the representative image based on a medoid of a corresponding one of the clusters of unlabeled image data. [14] The computing device of claim 8, wherein the processing device is configured to execute instructions for computing the instance-level contrastive loss and the cluster-level contrastive loss. [15] Computer-controlled machine comprising: at least one sensor configured to generate an input image; a control system designed to perform data pre-selection for an object detection system, the control system being designed to: Receiving a first data set, wherein the first data set includes unlabeled data corresponding to one or more images, Providing the first data set and a plurality of learnable prompt vectors to a pre-training model, wherein the learnable prompt vectors include text inputs, Generating, using the pre-training model, an unsupervised learning prompt based on the first data set and the plurality of learnable prompt vectors, wherein the unsupervised learning prompt corresponds to a multimodal feature of the one or more images of the first data set, Extracting features from one of the first dataset and a second dataset based on the unsupervised learning prompt, Selecting and labeling a subset of instances of the extracted features and Generating and outputting a labeled dataset based on the labeled subset of instances, and an actuator configured to control operation of the computer-controlled machine based on the labeled data set. [16] A computer-controlled machine according to claim 15, wherein the pre-training model is a Bootstrapping Language Image Pre-Training (BLIP-2) model. [17] A computer-controlled machine according to claim 15, wherein the extracted features include clusters of unlabeled image data, and wherein the selected subset of instances includes one or more of the clusters of unlabeled data. [18] A computer-controlled machine according to claim 15, wherein the control system is arranged to select a representative image for labeling from each of the clusters of unlabeled image data. [19] A computer-controlled machine according to claim 18, wherein the control system is configured to select the representative image based on a medoid of a corresponding one of the clusters of unlabeled image data. [20] A computer-controlled machine according to claim 15, wherein the control system is configured to calculate the instance-level contrastive loss and the cluster-level contrastive loss.

Images