Unsupervised cue learning with vision-language model for data pre-selection

Abstract

Unsupervised cue learning for data preselection with a vision-language model is provided. A method of performing data pre-selection for an object detection system includes receiving a first dataset including untagged data corresponding to one or more images, providing the first dataset and a plurality of learnable cue vectors to a pre-trained model. The learnable cue vector includes a textual input. The method further includes generating an unsupervised learning cue based on the first data set and the plurality of learnable cue vectors using the pre-trained model. The unsupervised learning prompts a multi-modal feature corresponding to one or more images of the first data set. The method further includes extracting features from any one of the first data set and the second data set based on the unsupervised learning cues, selecting and tagging a subset of instances of the extracted features, and generating and outputting a tagged data set based on the tagged subset of instances.

Description

Technical Field

[0001] The present disclosure relates to data pre-selection for object detection systems and methods. Background Art

[0002] Object detection techniques such as visual object detection techniques can use various detection models to detect or predict, classify, and label objects and regions in captured images. For example, a subset of data samples is used to train one or more machine learning models for object detection and labeling. Summary of the invention

[0003] A method for performing data preselection for an object detection system includes receiving a first dataset including unlabeled data corresponding to one or more images, providing the first dataset and a plurality of learnable cue vectors to a pretrained model. The learnable cue vectors include text inputs. The method also includes generating, using the pretrained model, unsupervised learning cues based on the first dataset and the plurality of learnable cue vectors. The unsupervised learning cues correspond to multimodal features of the one or more images of the first dataset. The method also includes extracting features from either the first dataset or a second dataset based on the unsupervised learning cues, selecting and labeling a subset of instances of the extracted features, and generating and outputting a labeled dataset based on the labeled subset of instances.

[0004] In other features, the pre-trained model is a bootstrapped language-image pre-training (BLIP-2) model. The extracted features include clusters of unlabeled image data. The selected instance subset includes one or more clusters of the unlabeled data. The method also includes selecting a representative image for labeling from each cluster of the unlabeled image data. Selecting the representative image includes selecting the representative image based on a medoid of a corresponding cluster of the unlabeled image data. Generating the unsupervised learning hint includes calculating an instance-level contrastive loss and a cluster-level contrastive loss.

[0005] A computing device configured to perform data preselection for an object detection system includes a processing device configured to execute instructions stored in a memory to perform the following operations: receiving a first data set including unlabeled data corresponding to one or more images, and providing the first data set and a plurality of learnable prompt vectors to a pre-trained model. The learnable prompt vectors include text input. The processing device is also configured to execute instructions to perform the following operations: using the pre-trained model, generating unsupervised learning prompts based on the first data set and the plurality of learnable prompt vectors, the unsupervised learning prompts corresponding to multimodal features of one or more images of the first data set; extracting features from either the first data set or a second data set based on the unsupervised learning prompts; selecting and labeling a subset of instances of the extracted features; and generating and outputting a labeled data set based on the labeled subset of instances.

[0006] In other features, the pre-trained model is a bootstrapped language-image pre-training (BLIP-2) model. The extracted features include clusters of unlabeled image data. The selected instance subset includes one or more clusters of unlabeled data. The processing device is configured to execute instructions to select a representative image for labeling from each cluster of the unlabeled image data. The processing device is configured to execute instructions to select a representative image based on a center of a corresponding cluster of the unlabeled image data. The processing device is configured to execute instructions to calculate instance-level contrast loss and cluster-level contrast loss.

[0007] A computer-controlled machine includes: at least one sensor configured to generate an input image; a control system configured to perform data preselection for an object detection system, the control system configured to perform the following operations: receive a first data set including unlabeled data corresponding to one or more images; provide the first data set and a plurality of learnable prompt vectors to a pre-trained model, the plurality of learnable prompt vectors including text input; use the pre-trained model and generate unsupervised learning prompts based on the first data set and the plurality of learnable prompt vectors, the unsupervised learning prompts corresponding to multimodal features of one or more images of the first data set; extract features from either the first data set or a second data set based on the unsupervised learning prompts; select and label a subset of instances of the extracted features; and generate and output a labeled data set based on the labeled subset of instances; and an actuator configured to control the operation of the computer-controlled machine based on the labeled data set.

[0008] In other features, the pre-trained model is a bootstrapped language-image pre-training (BLIP-2) model. The extracted features include clusters of unlabeled image data, and wherein the selected instance subset includes one or more clusters of the unlabeled data. The control system is configured to select a representative image for labeling from each cluster of the unlabeled image data. The control system is configured to select the representative image based on a center of a corresponding one of the clusters of the unlabeled image data. The control system is configured to calculate an instance-level contrastive loss and a cluster-level contrastive loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A system for training a neural network according to the principles of the present disclosure is generally shown.

[0010] Figure 2 A computer-implemented method for training and utilizing a neural network in accordance with the principles of the present disclosure is generally presented.

[0011] Figure 3A An audio data tagging system according to the principles of the present disclosure is generally shown.

[0012] Figure 3BA portion of a data capture system in accordance with the principles of the present disclosure is generally shown.

[0013] Figure 3C An alternative audio data labeling system in accordance with the principles of the present disclosure is generally shown.

[0014] Figure 4A An example data pre-selection system according to the present disclosure is generally shown.

[0015] Figure 4B Data preselection according to the systems and methods of the present disclosure is generally shown.

[0016] Figure 4C Results of the disclosed unsupervised prompt learning (UP-DP) system and method for data pre-selection for downstream tasks are generally shown.

[0017] Figure 4D The steps of an example data pre-selection method according to the principles of the present disclosure are shown.

[0018] Figure 5 Depicted is a schematic diagram of the interaction between a computer-controlled machine and a control system in accordance with the principles of the present disclosure.

[0019] Figure 6 Depicted is a schematic diagram of a device according to the principles of the present disclosure. Figure 5 Schematic diagram of a control system configured to control a vehicle, which can be a partially autonomous vehicle, a fully autonomous vehicle, a partially autonomous robot, or a fully autonomous robot.

[0020] Figure 7 Depicted Figure 5 Schematic diagram of a control system configured to control a manufacturing machine, such as a punch cutter, a cutting machine, or a gun drill, of a manufacturing system (e.g., part of a production line).

[0021] Figure 8 Depicted Figure 5 Schematic diagram of a control system configured to control a power tool, such as a drill or driver, having an at least partially autonomous mode.

[0022] Fig. 9 Depicting a device configured to control an automated personal assistant Figure 5 Schematic diagram of the control system.

[0023] Fig.10 Depicted Figure 5 Schematic diagram of a control system configured to control a monitoring system, such as a controlled access system or a surveillance system.

[0024] Fig.11 Depicted Figure 5Schematic diagram of a control system configured to control an imaging system, such as an MRI device, an x-ray imaging device, or an ultrasound device. DETAILED DESCRIPTION

[0025] Embodiments of the present disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various and alternative forms. The drawings are not necessarily to scale; some drawings may be enlarged or minimized to show the details of specific components. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a representative basis for teaching those skilled in the art to adopt the embodiments in different ways. As will be understood by those of ordinary skill in the art, the various features shown and described with reference to any of the figures may be combined with the features shown in one or more other figures to produce embodiments that are not explicitly shown or described. The combination of the features shown provides representative embodiments for typical applications. However, for specific applications or implementations, various combinations and modifications of features consistent with the teachings of the present disclosure may be desirable.

[0026] Unless the context clearly indicates otherwise, "a", "an", and "the" as used herein refer to both singular and plural references. For example, a "processor" programmed to perform various functions refers to one processor programmed to perform each and all functions, or more than one processor programmed together to perform each of the various functions.

[0027] A subset of data samples is used to train vision-language machine learning models for object detection systems and methods. Data-efficient machine learning aims to identify the best subset of data samples (labeled or unlabeled) to achieve the best model performance, and has become an important research area in the era of data-hungry deep learning.

[0028] Semi-supervised learning (SSL) and active learning (AL) are two example methods for improving data-efficient learning. SSL aims to address the problem of label scarcity by improving the performance of the model by applying a limited amount of labeled data in combination with a richer pool of unlabeled data. In contrast, AL methods start with an initial set of labeled data and select the most informative data points to label, maximizing model performance with a limited label budget. However, existing AL methods require an initial labeled set to start with and require training a series of models.

[0029] Therefore, the data preselection task raises some challenges that SSL and AL fail to adequately address. In data preselection, the initial data is unlabeled and the specific downstream task is unknown. For example, for object detection, the class of the object to be detected is unknown. However, SSL and AL methods require both initial labeled data and knowledge of the specific learning task. Therefore, a method is needed to select diverse and representative instances to cover the entire dataset with the minimum number of required labels.

[0030] In some examples, basic visual-language (VL) models, such as the contrastive language-image pre-training (CLIP) model and the bootstrapped language-image pre-training (BLIP-2) model, can address some of the issues associated with SSL and AL methods. These VL models acquire comprehensive semantic knowledge by learning from a large collection of image and text pairs. Therefore, VL models can be capable of extracting meaningful features from diverse input modalities. Although VL models can be applicable to a variety of tasks, such as few-shot classification, semi-supervised learning, and selective labeling, these models may focus on unimodal features (e.g., vision) and inadvertently ignore the potential contribution of multimodal features covering both vision and language.

[0031] Designing suitable prompts for VL models can be demanding and time-consuming, often requiring trial and error. In addition, various automatic prompt learning methods usually rely on a small set of labeled or pseudo-labeled data, which is impractical for data pre-selection when the downstream task is undefined.

[0032] The data pre-selection system and method according to the present disclosure incorporate unsupervised hint learning in the visual-language model for data pre-selection. More specifically, the system and method described herein implement hint learning technology, which is configured to select instances for labeling from an unlabeled dataset in a single pass, thereby optimizing the performance of undefined downstream tasks with limited annotation budgets. Previous data pre-selection methods relied solely on visual features extracted from base models (e.g., CLIP and BLIP-2 models as described above), but largely ignored the importance of text features. Pure unsupervised hint learning according to the present disclosure is intended to enhance multimodal features extracted from VL models for data pre-selection.

[0033] As described herein, prompts can be designed so that the multimodal features extracted by the VL model provide more powerful representations for data preselection. The joint feature space of both vision and text can produce better representations for data preselection. Unsupervised prompt learning for data preselection ("UP-DP") according to the present disclosure implements an unsupervised prompt learning method that adapts a visual-language pre-trained model (e.g., BLIP-2) for data preselection. Specifically, with BLIP-2 parameters fixed or frozen, text prompts are trained to extract joint features with improved representations, thereby ensuring a diverse clustering structure covering the entire data set. Prompts learned from an example data set demonstrate significant generalization capabilities and can be directly applied to enhance feature extraction of BLIP-2 models from other data sets.

[0034] Figure 1 An example system 100 for training a neural network (e.g., of an ML model) is shown. System 100 may be configured to (and / or include circuitry configured to) implement the systems and methods of the present disclosure described in more detail below. System 100 may include an input interface for accessing training data 102 for the neural network. For example, Figure 1 As shown, the input interface can be constituted by a data storage interface 104, which can access the training data 102 from a data storage unit 106. For example, the data storage interface 104 can be a memory interface or a persistent storage interface, such as a hard disk or SSD interface, but can also be a personal, local or wide area network interface, such as a Bluetooth, Zigbee or Wi-Fi interface or an Ethernet or fiber optic interface. The data storage unit 106 can be an internal data storage unit of the system 100, such as a hard disk drive or SSD, but can also be an external data storage unit, for example, a network accessible data storage unit.

[0035] In some embodiments, data store 106 may also include a data representation 108 of an untrained version of the neural network, which may be accessed by system 100 from data store 106. However, it will be appreciated that training data 102 and data representation 108 of the untrained neural network may also be accessed from different data stores, e.g., via different subsystems of data store interface 104. Each subsystem may be of a type as described above for data store interface 104.

[0036] In some embodiments, the data representation 108 of the untrained neural network may be generated internally by the system 100 based on the design parameters of the neural network, and thus may not be explicitly stored on the data storage 106. The system 100 may also include a processor subsystem 110 that may be configured to provide an iterative function during operation of the system 100 as a replacement for a stack of neural network layers to be trained. Here, the individual layers of the replaced layer stack may have mutually shared weights and may receive as input the output of the previous layer, or, for the first layer of the layer stack, receive as input an initial activation and a portion of the input of the layer stack.

[0037] The processor subsystem 110 may be further configured to iteratively train the neural network using the training data 102. Here, the training iterations of the processor subsystem 110 may include a forward propagation portion and a backward propagation portion. The processor subsystem 110 may be configured to perform the forward propagation portion by: determining an equilibrium point of the iterative function at which the iterative function converges to a fixed point, in addition to defining other operations of the forward propagation portion that may be performed, wherein determining the equilibrium point includes using a numerical root finding algorithm to find a root solution of the iterative function minus its input; and providing the equilibrium point in place of an output of a layer stack in the neural network.

[0038] The system 100 may also include an output interface for outputting a data representation 112 of the trained neural network, which may also be referred to as trained model data 112. Figure 1 As shown, the output interface can be constituted by the data storage interface 104, which in these embodiments is an input / output ("IO") interface, through which the trained model data 112 can be stored in the data storage 106. For example, the data representation 108 defining the "untrained" neural network can be replaced during or after training, at least in part, with the data representation 112 of the trained neural network, because the parameters of the neural network, such as the weights, hyperparameters and other types of parameters of the neural network, can be adapted to reflect the training on the training data 102. Figure 1 This is also illustrated in the example diagram by reference to the same data records 108, 112 on the data store 106. In some embodiments, the data representation 112 may be stored separately from the data representation 108 defining the "untrained" neural network. In some embodiments, the output interface may be separate from the data storage interface 104, but may generally be of the type described above for the data storage interface 104.

[0039] Figure 2A data annotation / augmentation system 200 configured to (and / or including circuitry configured to) implement a system for annotating and / or augmenting data is depicted. The data annotation system 200 may include at least one computing system 202 configured to implement all or part of the systems and methods of the present disclosure explained in more detail below. The computing system 202 may include at least one processor 204 operably connected to a memory unit 208. The processor 204 may include one or more integrated circuits that implement the functionality of a central processing unit (CPU) 206. The CPU 206 may be a commercially available processing unit that implements an instruction set such as one of the x86, ARM, Power, or MIPS instruction set families. The various components of the system 200 may be implemented with the same or different circuits.

[0040] During operation, CPU 206 may execute stored program instructions retrieved from memory unit 208. The stored program instructions may include software that controls the operation of CPU 206 to perform the operations described herein. In some embodiments, processor 204 may be a system on a chip (SoC) that integrates the functionality of CPU 206, memory unit 208, network interfaces, and input / output interfaces into a single integrated device. Computing system 202 may implement an operating system for managing various aspects of operation.

[0041] Memory unit 208 may include volatile memory and nonvolatile memory for storing instructions and data. Nonvolatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when computing system 202 is disabled or loses power. Volatile memory may include static and dynamic random access memory (RAM) that stores program instructions and data. For example, memory unit 208 may store machine learning model 210 (e.g., in Figure 2 210), or an algorithm, a training data set 212 for the machine learning model 210, an original source data set 216, etc.

[0042] The computing system 202 may include a network interface device 222 configured to provide communications with external systems and devices. For example, the network interface device 222 may include a wired and / or wireless Ethernet interface defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 series of standards. The network interface device 222 may include a cellular communication interface for communicating with a cellular network (e.g., 3G, 4G, 5G). The network interface device 222 may be further configured to provide a communication interface to an external network 224 or a cloud.

[0043] The external network 224 may be referred to as the World Wide Web or the Internet. The external network 224 may establish a standard communication protocol between computing devices. The external network 224 may allow information and data to be easily exchanged between computing devices and the network. One or more servers 230 may communicate with the external network 224.

[0044] Computing system 202 may include input / output (I / O) interface 220, which may be configured to provide digital and / or analog input and output. I / O interface 220 may include an additional serial interface (eg, a universal serial bus (USB) interface) for communicating with external devices.

[0045] The computing system 202 may include a human-machine interface (HMI) device 218, which may include any device that enables the system 200 to receive control inputs. Examples of input devices may include human-machine interface inputs, such as keyboards, mice, touch screens, voice input devices, and other similar devices. The computing system 202 may include a display device 232. The computing system 202 may include hardware and software for outputting graphics and text information to the display device 232. The display device 232 may include an electronic display screen, a projector, a printer, or other suitable devices for displaying information to a user or operator. The computing system 202 may be further configured to allow interaction with a remote HMI and a remote display device through a network interface device 222.

[0046] System 200 can be implemented using one or more computing systems. Although this example depicts a single computing system 202 that implements all described features, it is intended that various features and functions be separated and implemented through multiple computing units that communicate with each other. The specific system architecture selected can depend on various factors.

[0047] The system 200 can implement a machine learning model 210 configured to analyze a raw source data set 216. For example, the CPU 206 and / or other circuits can implement the machine learning model 210. The raw source data set 216 can include raw or unprocessed sensor data, which can represent an input data set for a machine learning system. The raw source data set 216 can include video, video clips, images, audio, text-based information, and raw or partially processed sensor data (e.g., a radar map of an object). In some embodiments, the machine learning model 210 can be a deep learning or neural network algorithm designed to perform a predetermined function. For example, a neural network algorithm can be configured to identify an event or object in a video clip based on audio data.

[0048] The computer system 200 may store a training data set 212 for the machine learning model 210. The training data set 212 may represent a set of previously constructed data used to train the machine learning model 210. The machine learning model 210 may use the training data set 212 to learn weighting factors associated with a neural network algorithm. The training data set 212 may include a set of source data having corresponding achievements or results that the machine learning model 210 attempts to replicate through the learning process.

[0049] The machine learning model 210 can operate in a learning mode using the training data set 212 as input. The machine learning model 210 can be executed in multiple iterations using data from the training data set 212. For each iteration, the machine learning model 210 can update the internal weighting factors based on the achieved results. For example, the machine learning model 210 can compare the output results (e.g., annotations) with the results included in the training data set 212. Because the training data set 212 includes expected results, the machine learning model 210 can determine when the performance is acceptable. After the machine learning model 210 reaches a predetermined performance level (e.g., 100% consistency with the results associated with the training data set 212), the machine learning model 210 can be executed using data that is not in the training data set 212. The trained machine learning model 210 can be applied to a new data set to generate annotated data.

[0050] The machine learning model 210 may be configured to identify specific features in the raw source data 216. The raw source data 216 may include multiple instances or input data sets (e.g., video streams or segments including audio data) for which annotation results are desired. By way of example only, the machine learning model 210 may be configured to identify objects or events in a video segment based on audio data and annotate these events. The machine learning model 210 may be programmed to process the raw source data 216 to identify the presence of specific features. The machine learning model 210 may be configured to identify features in the raw source data 216 as predetermined features. The raw source data 216 may be obtained from a variety of sources. For example, the raw source data 216 may be actual input data collected by a machine learning system. The raw source data 216 may be machine generated for testing the system. By way of example, the raw source data 216 may include raw video and / or audio data from a camera, audio data from a microphone, and the like.

[0051] In an example, the machine learning model 210 can process the raw source data 216 and output video and / or audio data including one or more indications of the identified events. The machine learning model 210 can generate a confidence level or factor for each generated output. For example, a confidence value exceeding a predetermined high confidence threshold can indicate that the machine learning model 210 is confident that the identified event (or feature) corresponds to a particular event. A confidence value less than a low confidence threshold can indicate that the machine learning model 210 has some uncertainty that a particular feature exists.

[0052] like Figure 3A and 3B , system 300 may include an image (e.g., video) capture device 302, an audio capture array 304, and a computing system 202. The system may receive video stream data associated with a data capture environment from the image capture device 302. The system 202 may be configured to perform video object detection to identify one or more objects in a corresponding image of the video stream data. The system 202 may receive audio stream data corresponding to at least a portion of the video stream data from the audio capture array 304. The audio capture array 304 may include one or more microphones 306 or other suitable audio capture devices. The systems and methods described herein may be configured to use output from at least a first machine learning model (e.g., such as machine learning model 210 or other suitable machine learning model configured to provide output including one or more object or event detection predictions) to mark at least some objects of the video stream data and / or audio stream data.

[0053] The system 202 may calculate (e.g., using at least one probability-based function or other suitable technique or function) at least one offset value for at least a portion of the audio stream data corresponding to at least one tagged object of the video stream data based on at least one data capture characteristic. The system 202 may synchronize at least a portion of the video stream data with a portion of the audio stream data corresponding to at least one tagged object of the video stream data using at least the at least one offset value. The at least one data capture characteristic may include one or more characteristics of at least one image capture device, one or more characteristics of at least one audio capture array, one or more characteristics corresponding to a position of at least one image capture device relative to at least one audio capture array, one or more characteristics corresponding to movement of an object in the video stream data, one or more other suitable data capture characteristics, or a combination thereof.

[0054] System 202 may use one or more tags and at least one offset value of the labeled objects of the video stream data to mark at least a portion of the audio stream data corresponding to at least one labeled object of the video stream data. Each corresponding tag may include an event type, an event start indicator, and an event end indicator. System 202 may use at least some of the labeled portions of the audio stream data to generate training data. System 202 may use the training data to train a second machine learning model. System 202 may use the second machine learning model to detect one or more sounds associated with the audio data provided as input to the second machine learning model. The second machine learning model may include any suitable machine learning model and may be configured to perform any suitable functions, such as those described herein with reference to Figures 4-11.

[0055] In some embodiments, Figure 3C As generally shown in , the computing system 202 can be configured to mark audio data based on sensor data received from one or more sensors, such as those described herein or any other suitable sensor or combination of sensors. The system 202 can receive audio stream data associated with the data capture environment from the audio capture array 354 or any suitable audio capture device (such as one or more microphones 306 or other suitable audio capture devices). It should be understood that the audio capture array 354 may include features similar to the audio capture array 304 features, and may include any suitable number of audio capture devices. The system 202 can receive sensor data associated with the data capture environment from at least one sensor (e.g., such as sensor 352) asynchronous with respect to the audio capture array 354. The sensor 354 may include at least one of an induction coil, a radar sensor, a LiDAR sensor, a sonar sensor, an image capture device, any other suitable sensor, or a combination thereof. The audio capture array 354 can be positioned away from the sensor 354, positioned close to the sensor 354, or positioned to be in any suitable relationship with the sensor 354.

[0056] System 202 may use output from at least a first machine learning model (e.g., machine learning model 210 or other suitable machine learning model) to identify at least some events in sensor data. Machine learning model 210 may be configured to provide output including one or more event detection predictions based on sensor data. System 202 may synchronize at least a portion of sensor data associated with a portion of audio stream data corresponding to at least one event of sensor data. System 202 may use one or more tags extracted for corresponding events of sensor data values ​​to mark at least a portion of audio stream data corresponding to at least one event of sensor data. Each corresponding tag may include an event type, an event start indicator, and an event end indicator. System 202 may use at least some of the marked portions of audio stream data to generate training data. System 202 may use training data to train a second machine learning model. System 202 may use a second machine learning model to detect one or more sounds associated with audio data provided as input to a second machine learning model. The second machine learning model may include any suitable machine learning model and may be configured to perform any suitable functions, such as those described herein with reference to FIGS. 4-11.

[0057] Any of the systems described above and / or in more detail below may implement the data pre-selection systems and methods of the present disclosure, including unsupervised cue learning techniques in visual-language models for data pre-selection.

[0058] Figure 4A An example data pre-selection system 400 according to the present disclosure is shown. Example unimodal and multimodal feature representations obtained from example VL pre-trained models (e.g., BLIP-2 models) for various ground truth labels are shown as U-map visualizations at 402 (e.g., unimodal feature representations using only visual / image features) and 404 (e.g., multimodal feature representations using learned prompts). For comparison, the results using random prompts are shown at 406. It is worth noting that when the category information of the downstream task is unknown, the unimodal features become mixed, as shown in 402, making it challenging to identify specific categories. In contrast, when coupled with appropriate prompts, multimodal features exhibit a more dispersed and distinct distribution, as shown in 404, thereby enabling improved discrimination between categories. Therefore, as shown in 404, the VL model with appropriate prompts can provide visually better multimodal features than the unimodal features shown in 402, which can serve as an effective starting point for data pre-selection. This enhanced discrimination is critical to helping users identify the most representative and diverse instances during the data pre-selection process.

[0059] The goal of data pre-selection is to select instances for labeling from an unlabeled dataset 408 (e.g., the OxfordPets training dataset) in a single pass to maximize model performance on unknown downstream visual tasks (i.e., without knowledge of the predicted class) given a limited annotation budget. This task is driven by two practical requirements: (i) in the data acquisition phase, it is desirable to collect the minimum amount of data to support a potentially diverse set of downstream tasks (e.g., classification, action recognition, video or image detection, etc.) and training paradigms (e.g., active learning, semi-supervised learning, supervised learning, etc.); and (ii) in the annotation phase, we aim to achieve a desired level of performance on the pre-selected data with less manual annotation.

[0060] like Figure 4A As shown, the unlabeled dataset 408 is provided to a dataset pre-selection module or circuit 410, which is configured to perform data pre-selection for a data-efficient learning task, as described in more detail below. The data-efficient learning task according to the present disclosure incorporates multimodal features using learned cues, and is different from semi-supervised learning and active learning tasks, and does not require seed labels and downstream task information.

[0061] Continue to refer Figure 4A ,exist Figure 4B Data preselection according to the systems and methods of the present disclosure is described in more detail in . These systems and methods are configured to formally define data preselection tasks, provide an unsupervised hint learning framework to help pre-trained (e.g., BLIP-2) models extract improved multimodal features for data preselection, and use the adapted BLIP-2 model to select instances for downstream tasks. The workflows for data preselection, the training process of unsupervised hint learning, and the generalization of learned hints across various datasets are shown at 412, 414, and 416, respectively.

[0062] For data preselection, for illustration purposes, assume a very large dataset D consisting of d instances and an annotation budget of l. The task is to select l (l<<D) instances to label so as to yield the best performance on an undefined downstream task trained on such a partially labeled dataset. Formally, Denotes d for image x i and the corresponding category label y i (If the label is unknown, ). L represents a subset of size l of D with known class labels. The goal is to select To maximize the labeled data D L (with or without unlabeled data D U =D\D L). This data preselection task is challenging since there are no labels at the beginning. In other words, the information that will maximize the performance of the undefined downstream task is unknown. Therefore, the goal is to select l instances that are not only representative but also diverse enough to cover the entire dataset broadly, preventing the loss of valuable information prematurely before labels are acquired.

[0063] As shown in 412, data pre-selection includes extracting semantically important and high-quality features from the data set based on unsupervised learning prompts (e.g., using feature extractor module or circuit 417). For example, feature extractor module 417 can be implemented by a circuit such as data set pre-selection module 410. In some examples, data set pre-selection module 410 implements one or more components that are configured to perform the functions of unsupervised prompt learning shown in 414. As shown in 418, the extracted features facilitate clustering all unlabeled images into well-structured groups (relative to feature clustering without prompts). Subsequently, as shown in 419, a center point is selected from each cluster as a representative image of the label. The use of the center achieves representativeness, while selecting an instance from each cluster achieves diversity. The BLIP-2 model according to the present disclosure is adapted accordingly, as described in more detail below.

[0064] As shown in 414, unsupervised hint learning according to the present disclosure is configured to extract desired features from a dataset using a learnable context 420 of a BLIP-2 model 422 and two multi-layer perceptron (MLP) heads for both instance-level and cluster-level transformations. These components are jointly optimized using an unsupervised clustering objective (particularly instance-level and cluster-level contrastive learning). After training, the cluster assignments of all instances can be easily derived from the predicted soft labels produced by the cluster-level MLP heads. When combined with the multimodal features obtained from the adapted BLIP-2 model, the most representative and diverse instances from the entire dataset D can be effectively identified.

[0065] The BLIP-2 model 422 provides an effective pre-training strategy that utilizes an existing pre-trained image encoder and a large language model while the parameters of the BLIP-2 model remain frozen. Training a lightweight query transformer (Q-Former) bridges the modality gap between images and text. In the representation learning phase, the BLIP-2 model 422 connects the Q-Former to the frozen image encoder and performs image-text pair pre-training. Using three image-text pre-training objectives (image-text contrastive learning, image-based text generation, and image-text matching), a strong connection between the image and the input text is established (i.e., the BLIP-2 model 422 is forced to extract the most relevant visual information from the image to the text and automatically generate output multimodal features). This feature of the BLIP-2 model 422 provides the opportunity to change different prompts (e.g., input text), extract different multimodal features, and enhance the data pre-selection process.

[0066] However, when the downstream task is unknown, selecting appropriate hints can be demanding and time-consuming, especially for data pre-selection. According to previous hint learning methods, each context token can be modeled using a continuous vector that can be learned end-to-end from the data. Specifically, as shown in 414, N learnable hint representation context vectors are defined, denoted as V = v 1 ,v 2 ,…v n , each with the same dimensions as the word embeddings, to be used as text inputs to the BLIP-2 model 422. In contrast to many previous approaches that employ cross entropy loss as their learning objective on labeled data, the systems and methods described herein implement unsupervised clustering as a learning objective on an undefined downstream task. Since the text encoder is differentiable, gradients can be back-propagated to update the context vectors. The base model of the BLIP-2 model 422 remains unchanged during the training process.

[0067] The output of BLIP-2 model 422 may be subject to instance-level contrastive loss (“instance loss”) and cluster-level contrastive loss (“cluster loss”). With respect to instance loss, instance-level contrastive learning is designed to maximize the similarity between pairs of positive instances while minimizing the similarity between pairs of negative instances. Assuming that labels for all data are missing, the goal is to maximize the consistency between different augmented views of the same instance and treat all other augmented instances as negative examples. Specifically, consider a randomly sampled mini-batch of N images. Given x i Each image represented undergoes two different augmentations, creating two views of the same example: and The two images are then encoded along with the learnable context V according to f(·) of the BLIP-2 model 422 to generate a multimodal representation: and Next, these representations are transformed by a nonlinear transformation MLPg shown in 424 I (·) (called instance-level header) is further transformed, which produces and Here, sim(.,.) represents cosine similarity. Then the instance-level contrast loss for a positive pair of examples (i,j)∈[1,N] is defined as:

[0068]

[0069] where τ I is the temperature scalar. The final loss corresponds to It is computed across all positive pairs in a mini-batch.

[0070] Similar to instance-level contrastive learning, cluster-level contrastive learning aims to maximize the similarity between positive cluster pairs while minimizing the similarity between negative cluster pairs. To construct cluster representations, another cluster-level head g is used. C (·) to map the multimodal representation into an M-dimensional space, where M is equal to the number of clusters. Therefore, the mth element of the feature can be interpreted as the probability that the instance belongs to the mth cluster. For a mini-batch of N samples and M predefined clusters, the output of the clustering head is C under the first expansion a ∈R N×M , and therefore It can be interpreted as the probability that sample $n$ is assigned to cluster m under augmentation $a$. a The i-th column of can be regarded as the representation of the i-th cluster under the expansion of a. Therefore, by choosing C a and C b The i-th column of forms a positive cluster pair, labeled and While keeping the other 2M-2 pairs as negative, as follows:

[0071]

[0072] where τ C is the cluster-level temperature parameter. To avoid the trivial solution where most instances are assigned to the same cluster, regularization is added that encourages entropy in cluster assignment probabilities, as follows:

[0073]

[0074] in The final cluster-level loss is

[0075] In this way, as described above and as Figure 4B As shown, the training of the optimized BLIP-2 model 422 is a single-stage and end-to-end process. The hint and the two heads are optimized simultaneously, and the overall objective function is a combination of both instance-level and cluster-level contrastive losses (e.g. ).

[0076] After training, each instance is assigned a cluster number by the adapted BLIP-2 model 422 and the cluster-level MLP head, resulting in several sampling strategies. First, the cluster-level head g C (·) The predicted probability is used for sampling. In this case, for each cluster, the instance with the highest confidence score is selected. Conversely, the center of each cluster can be selected to be labeled for downstream tasks. Given a cluster with N k Members Cluster X k , center m k is the instance with the minimum average dissimilarity from all instances, which is obtained by multimodal feature h or form f(·) or for g I (·) is calculated by the projection feature z.

[0077] As shown at 416, the learned cues obtained at 412 and 414 can be generalized across various datasets. For example, although the learned cues (N learnable cues representing context vectors V) are obtained from a dataset of pets, these same cues can be used for dataset pre-selection for a different dataset (e.g., a flower image dataset). As shown at 426, providing the learned cues obtained based on the pets dataset to the BLIP-2 model 422 results in improved clustering of features detected in the flower image dataset.

[0078] Figure 4C Results of the disclosed unsupervised prompt learning (UP-DP) system and method for data preselection for downstream tasks are generally shown, in particular image classification using selected labeled examples. For example, Figure 4CAn intuitive understanding of the operation of the UP-DP system and method is shown by visualizing the learned feature representations and cluster assignments of all instances using 2D projections with Uniform Manifold Approximation and Projection (UMAP). The features extracted using the BLIP-2 model 422 with the learned cues demonstrate the ability to discriminate between different types of dogs and cats. For example, for the Shiba Inu group 428, the cluster assignments generated by our clustering head classify multiple Shiba Inus into different clusters based on background and posture, as shown in 430. Multimodal features are extracted using the BLIP-2 model 422 with the learned cues. As shown in 432, the features extracted with the learned cues can distinguish various types of pets even without labels. In other words, within the same breed or animal type, the extracted features capture more detailed information, such as background, pet posture, etc. Therefore, as shown in Figure 4C As shown, the UP-DP system and method effectively select diverse representative points from unlabeled image data.

[0079] In this way, as described herein, the learned cues obtained using a dataset can be used across different other datasets, which significantly enhances feature extraction when applied in a plug-and-play manner.

[0080] Figure 4D Steps of an example data preselection method 440 according to principles of the present disclosure are shown. For example, one or more processors or processing devices are configured to execute instructions to implement method 440, such as one or more processors of the system described herein. At 442, method 440 receives an unlabeled data set. At 444, the unlabeled data set is provided to a pre-trained model (e.g., a BLIP-2 model as described above). At 446, a plurality of learnable hint vectors (e.g., vectors V as defined above) are provided to the pre-trained model (e.g., as text input to the BLIP-2 model).

[0081] At 448, method 440 (e.g., a pre-trained model) generates and optimizes unsupervised learning hints corresponding to multimodal features based on the unlabeled dataset and the learnable hint vector. The unsupervised learning hint represents an optimization of the learnable hint vector. At 450, method 440 (e.g., feature extractor module 417) extracts features from a dataset (e.g., the same or different dataset as provided to the pre-trained model) based on the unsupervised learning hint. For example, the extracted features correspond to clusters of unlabeled image data (e.g., unlabeled images), as shown in 418. At 452, method 440 selects a subset of instances of the unlabeled data to be labeled ("selected instances"). In this way, the most valuable features are selected from the unlabeled data for labeling.

[0082] At 454, labeling is performed on the selected subset of instances. At 456, labeling is performed on the remaining unlabeled image data (i.e., the instances that were not labeled at 454) based on the labeled selected subset of instances. For example, the labeling performed at 456 may include using one or more classifiers (e.g., as in Figure 4A 458), and can be implemented using supervised learning, semi-supervised learning, active learning, or a combination thereof. At 460, method 440 outputs and / or stores the labeled data set.

[0083] Figure 5-11 Depicted are example systems and devices that may implement UP-DP systems and methods according to the present disclosure. Figure 5 A schematic diagram of the interaction between a computer-controlled machine 500 and a control system 502 is depicted. The computer-controlled machine 500 includes an actuator 504 and a sensor 506. The actuator 504 may include one or more actuators, and the sensor 506 may include one or more sensors. The sensor 506 is configured to sense a condition of the computer-controlled machine 500. The sensor 506 may be configured to encode the sensed condition into a sensor signal 508 and transmit the sensor signal 508 to the control system 502. Non-limiting examples of the sensor 506 include video, radar, LiDAR, ultrasonic, and motion sensors. In some embodiments, the sensor 506 is an optical sensor configured to sense an optical image of the environment near the computer-controlled machine 500.

[0084] The control system 502 is configured to receive sensor signals 508 from the computer controlled machine 500. As described below, the control system 502 may be further configured to calculate actuator control commands 510 based on the sensor signals and transmit the actuator control commands 510 to the actuators 504 of the computer controlled machine 500.

[0085] like Figure 5 As shown, the control system 502 includes a receiving unit 512. The receiving unit 512 can be configured to receive the sensor signal 508 from the sensor 506 and transform the sensor signal 508 into an input signal x. In an alternative embodiment, the sensor signal 508 is directly received as the input signal x without the receiving unit 512. Each input signal x can be a part of each sensor signal 508. The receiving unit 512 can be configured to process each sensor signal 508 to generate each input signal x. The input signal x can include data corresponding to the image recorded by the sensor 506.

[0086] The control system 502 includes a classifier 514. The classifier 514 can be configured to classify the input signal x into one or more labels using a machine learning (ML) algorithm, such as a neural network. For example, the classifier 514 corresponds to the above-mentioned classifier 408. The classifier 514 is configured to be parameterized by parameters, such as those described above (e.g., parameter θ). The parameter θ can be stored in and provided by the non-volatile storage unit 516. The classifier 514 is configured to determine the output signal y from the input signal x. Each output signal y includes information that assigns one or more labels to each input signal x. The classifier 514 can transmit the output signal y to a conversion unit 518. The conversion unit 518 is configured to convert the output signal y into an actuator control command 510. The control system 502 is configured to transmit the actuator control command 510 to the actuator 504, and the actuator 504 is configured to actuate the computer-controlled machine 500 in response to the actuator control command 510. In some embodiments, actuator 504 is configured to actuate computer-controlled machine 500 directly based on output signal y.

[0087] When the actuator 504 receives the actuator control command 510, the actuator 504 is configured to perform an action corresponding to the associated actuator control command 510. The actuator 504 may include control logic configured to transform the actuator control command 510 into a second actuator control command for controlling the actuator 504. In one or more embodiments, the actuator control command 510 may be used to control a display instead of or in addition to the actuator.

[0088] In some embodiments, instead of or in addition to the computer-controlled machine 500 including the sensor 506, the control system 502 includes the sensor 506. Instead of or in addition to the computer-controlled machine 500 including the actuator 504, the control system 502 may also include the actuator 504.

[0089] like Figure 5 As shown, the control system 502 also includes a processor 520 and a memory 522. The processor 520 may include one or more processors. The memory 522 may include one or more memory devices. The classifier 514 (e.g., ML algorithm) of one or more embodiments may be implemented by the control system 502, which includes a non-volatile storage unit 516, a processor 520, and a memory 522.

[0090] The non-volatile storage 516 may include one or more persistent data storage devices, such as a hard drive, an optical drive, a tape drive, a non-volatile solid-state device, cloud storage, or any other device capable of persistently storing information. The processor 520 may include one or more devices selected from a high-performance computing (HPC) system, including a high-performance core, a microprocessor, a microcontroller, a digital signal processor, a microcomputer, a central processing unit, a field programmable gate array, a programmable logic device, a state machine, a logic circuit, an analog circuit, a digital circuit, or any other device that manipulates a signal (analog or digital) based on computer executable instructions residing in the memory 522. The memory 522 may include a single memory device or multiple memory devices, including but not limited to a random access memory (RAM), a volatile memory, a non-volatile memory, a static random access memory (SRAM), a dynamic random access memory (DRAM), a flash memory, a cache memory, or any other device capable of storing information.

[0091] The processor 520 may be configured to read into the memory 522 and execute computer executable instructions residing in the non-volatile storage 516 and embodying one or more anomaly detection methods of one or more embodiments. The non-volatile storage 516 may include one or more operating systems and applications. The non-volatile storage 516 may store compiled and / or interpreted computer programs created using various programming languages ​​and / or technologies, including but not limited to Java, C, C++, C#, Objective C, Fortran, Pascal, Java Script, Python, Perl, and PL / SQL, alone or in combination.

[0092] When executed by processor 520, the computer executable instructions of non-volatile storage 516 may cause control system 502 to implement one or more of the anomaly detection methods disclosed herein. Non-volatile storage 516 may also include data supporting the functions, features, and processes of one or more embodiments described herein.

[0093] The program code embodying the algorithms and / or methods described herein can be distributed individually or collectively as a program product in a variety of different forms. The program code can be distributed using a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to implement aspects of one or more embodiments. Computer-readable storage media that are inherently non-transitory may include volatile and non-volatile and removable and non-removable tangible media implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media may also include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other solid-state memory technology, portable compact disk read-only memory (CD-ROM) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be read by a computer. Computer-readable program instructions may be downloaded from a computer-readable storage medium to a computer, another type of programmable data processing device, or another device over a network, or to an external computer or external storage device.

[0094] Computer-readable program instructions stored in a computer-readable medium can be used to direct a computer, other types of programmable data processing devices, or other devices to operate in a particular manner, so that the instructions stored in the computer-readable medium produce an article of manufacture, including instructions that implement the functions, actions, and / or operations specified in the flowchart or chart. In certain alternative embodiments, the functions, actions, and / or operations specified in the flowchart and chart can be reordered, processed serially, and / or processed concurrently according to one or more embodiments. In addition, any one of the flowchart and / or chart may include more or fewer nodes or blocks than those shown in accordance with one or more embodiments.

[0095] A process, method or algorithm may be embodied in whole or in part using appropriate hardware components (e.g., application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices) or a combination of hardware, software, and firmware components.

[0096] Figure 6A schematic diagram of a control system 502 configured to control a vehicle 600, which may be an at least partially autonomous vehicle or an at least partially autonomous robot, is depicted. The vehicle 600 includes an actuator 504 and a sensor 506. The sensor 506 may include one or more video sensors, cameras, radar sensors, ultrasonic sensors, LiDAR sensors, and / or location sensors (e.g., GPS). One or more of the one or more specific sensors may be integrated into the vehicle 600. Alternatively or in addition to the one or more specific sensors identified above, the sensor 506 may include a software module that is configured to determine the state of the actuator 504 when executed. A non-limiting example of a software module includes a weather information software module configured to determine current or future weather conditions near the vehicle 600 or other location.

[0097] The classifier 514 of the control system 502 of the vehicle 600 can be configured to detect an object near the vehicle 600 based on the input signal x. In such an embodiment, the output signal y may include information characterizing that the object is near the vehicle 600. The actuator control command 510 may be determined based on the information. The actuator control command 510 may be used to avoid a collision with the detected object.

[0098] In some embodiments, the vehicle 600 is an at least partially autonomous vehicle, and the actuator 504 may be embodied in the brakes, propulsion system, engine, drive train, or steering of the vehicle 600. Actuator control commands 510 may be determined to control the actuator 504 so that the vehicle 600 avoids colliding with a detected object. The detected objects may also be classified according to what the classifier 514 thinks they are most likely to be, such as a pedestrian or a tree. Based on the classification, the actuator control commands 510 may be determined. In scenarios where adversarial attacks may occur, the above system may be further trained to better detect objects or identify changes in lighting conditions or angles of sensors or cameras on the vehicle 600.

[0099] In some embodiments where the vehicle 600 is an at least partially autonomous robot, the vehicle 600 may be a mobile robot configured to perform one or more functions, such as flying, swimming, diving, and walking. The mobile robot may be an at least partially autonomous lawn mower or an at least partially autonomous cleaning robot. In these embodiments, the actuator control commands 510 may be determined so that a propulsion unit, a steering unit, and / or a braking unit of the mobile robot may be controlled so that the mobile robot may avoid colliding with the identified object.

[0100] In some embodiments, vehicle 600 is an at least partially autonomous robot in the form of a gardening robot. In such embodiments, vehicle 600 may use an optical sensor as sensor 506 to determine the state of plants in the environment near vehicle 600. Actuator 504 may be a nozzle configured to spray a chemical. Based on the identified species and / or the identified state of the plant, actuator control commands 510 may be determined to cause actuator 504 to spray the plant with an appropriate amount of an appropriate chemical.

[0101] The vehicle 600 may be an at least partially autonomous robot in the form of a household appliance. Non-limiting examples of household appliances include a washing machine, a stove, an oven, a microwave, or a dishwasher. In such a vehicle 600, the sensor 506 may be an optical sensor configured to detect the state of an object to be processed by the household appliance. For example, where the household appliance is a washing machine, the sensor 506 may detect the state of the laundry within the washing machine. The actuator control command 510 may be determined based on the detected state of the laundry.

[0102] Figure 7 A schematic diagram of a control system 502 of a system 700 (e.g., a manufacturing machine) configured to control a manufacturing system 702 (e.g., a portion of a production line), such as a punch cutter, a cutter, or a gun drill, is depicted. The control system 502 can be configured to control an actuator 504 configured to control the system 700 (e.g., a manufacturing machine).

[0103] The sensor 506 of the system 700 (e.g., a manufacturing machine) may be an optical sensor configured to capture one or more attributes of the manufactured product 704. The classifier 514 may be configured to determine a state of the manufactured product 704 based on the captured one or more attributes. The actuator 504 may be configured to control the system 700 (e.g., a manufacturing machine) for subsequent manufacturing steps of the manufactured product 704 based on the determined state of the manufactured product 704. The actuator 504 may be configured to control the function of the system 700 (e.g., a manufacturing machine) on the subsequent manufactured product 706 of the system 700 (e.g., a manufacturing machine) based on the determined state of the manufactured product 704.

[0104] Figure 8 A schematic diagram of a control system 502 configured to control a power tool 800 (eg, a drill or driver) having at least a partially autonomous mode is depicted. The control system 502 may be configured to control an actuator 504 configured to control the power tool 800.

[0105] The sensor 506 of the power tool 800 can be an optical sensor configured to capture one or more properties of the working surface 802 and / or the fastener 804 being driven into the working surface 802. The classifier 514 can be configured to determine the state of the working surface 802 and / or the fastener 804 relative to the working surface 802 based on the captured one or more properties. The state can be that the fastener 804 is flush with the working surface 802. Alternatively, the state can be the hardness of the working surface 802. The actuator 504 can be configured to control the power tool 800 so that the driving function of the power tool 800 is adjusted based on the determined state of the fastener 804 relative to the working surface 802 or one or more captured properties of the working surface 802. For example, if the state of the fastener 804 is flush with the working surface 802, the actuator 504 may no longer continue the driving function. As another non-limiting example, the actuator 504 can apply additional or less torque based on the hardness of the working surface 802.

[0106] Fig. 9 Depicted is a schematic diagram of a control system 502 configured to control an automated personal assistant 900. The control system 502 may be configured to control an actuator 504 configured to control the automated personal assistant 900. The automated personal assistant 900 may be configured to control a household appliance, such as a washing machine, stove, oven, microwave, or dishwasher.

[0107] Sensor 506 may be an optical sensor and / or an audio sensor. The optical sensor may be configured to receive a video image of user 902's gesture 904. The audio sensor may be configured to receive a voice command of user 902.

[0108] The control system 502 of the automated personal assistant 900 may be configured to determine an actuator control command 510 configured to control the system 502. The control system 502 may be configured to determine the actuator control command 510 based on the sensor signal 508 of the sensor 506. The automated personal assistant 900 is configured to transmit the sensor signal 508 to the control system 502. The classifier 514 of the control system 502 may be configured to execute a gesture recognition algorithm to identify a gesture 904 made by the user 902, determine the actuator control command 510, and transmit the actuator control command 510 to the actuator 504. The classifier 514 may be configured to retrieve information from the non-volatile storage in response to the gesture 904 and output the retrieved information in a form suitable for receipt by the user 902.

[0109] Fig.10A schematic diagram of a control system 502 configured to control a monitoring system 1000 is depicted. The monitoring system 1000 can be configured to physically control access through a door 1002. A sensor 506 can be configured to detect scenes relevant to a decision whether to grant access. The sensor 506 can be an optical sensor configured to generate and transmit image and / or video data. The control system 502 can use such data to detect a person's face.

[0110] The classifier 514 of the control system 502 of the monitoring system 1000 can be configured to interpret the image and / or video data by matching the identities of known persons stored in the non-volatile storage 516 to determine the identity of the person. The classifier 514 can be configured to generate an actuator control command 510 in response to the interpretation of the image and / or video data. The control system 502 is configured to transmit the actuator control command 510 to the actuator 504. In this embodiment, the actuator 504 can be configured to lock or unlock the door 1002 in response to the actuator control command 510. In some embodiments, non-physical logical access control can also be performed.

[0111] The monitoring system 1000 may also be a surveillance system. In such an embodiment, the sensor 506 may be an optical sensor configured to detect a scene under surveillance, and the control system 502 is configured to control the display 1004. The classifier 514 is configured to determine a classification of the scene, for example, whether the scene detected by the sensor 506 is suspicious. The control system 502 is configured to transmit the actuator control command 510 to the display 1004 in response to the classification. The display 1004 may be configured to adjust the displayed content in response to the actuator control command 510. For example, the display 1004 may highlight an object that the classifier 514 considers suspicious. Using embodiments of the disclosed system, the surveillance system can predict that an object will appear at a certain time in the future.

[0112] Fig.11 A schematic diagram of a control system 502 configured to control an imaging system 1100, such as an MRI device, an x-ray imaging device, or an ultrasound device, is depicted. For example, the sensor 506 can be an imaging sensor. The classifier 514 can be configured to determine a classification of all or part of a sensed image. The classifier 514 can be configured to determine or select an actuator control command 510 in response to the classification obtained by the trained neural network. For example, the classifier 514 can interpret an area of ​​the sensed image as a potential abnormality. In this case, the actuator control command 510 can be determined or selected to cause the display 1102 to display the imaging and highlight the potential abnormal area.

[0113] Although exemplary embodiments are described above, these embodiments are not intended to describe all possible forms covered by the claims. The words used in this specification are descriptive words, not restrictive words, and it should be understood that various changes can be made without departing from the spirit and scope of the present disclosure. As mentioned above, the features of various embodiments can be combined to form further embodiments of the present invention, which may not be explicitly described or illustrated. Although various embodiments may have been described as providing advantages or being superior to other embodiments or prior art implementations in terms of one or more required characteristics, it is recognized by those of ordinary skill in the art that one or more features or characteristics can be weighed to achieve the desired overall system properties, depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, applicability, weight, manufacturability, ease of assembly, etc. Therefore, insofar as any embodiment is described as being less desirable than other embodiments or prior art implementations in terms of one or more characteristics, these embodiments are not outside the scope of the present disclosure and may be desirable for specific applications.

Claims

1. A method for performing data preselection for an object detection system, the method comprising: receiving a first dataset, wherein the first dataset includes unlabeled data corresponding to one or more images; Providing a first dataset and a plurality of learnable prompt vectors to a pre-trained model, wherein the learnable prompt vectors include text input; Using the pre-trained model, generating unsupervised learning prompts based on the first dataset and a plurality of learnable prompt vectors, wherein the unsupervised learning prompts correspond to multimodal features of one or more images of the first dataset; extracting features from either the first dataset or the second dataset based on unsupervised learning prompts; Select and label a subset of instances of the extracted features; as well as Generate and output a labeled dataset based on the labeled instance subset.

2. The method of claim 1, wherein the pre-trained model is a Bootstrapped Language-Image Pre-training (BLIP-2) model. The method of claim 1 , wherein the extracted features comprise clusters of unlabeled image data. The method of claim 3 , wherein the selected subset of instances comprises one or more clusters of unlabeled data. 5 . The method of claim 4 , further comprising selecting a representative image for labeling from each cluster of the unlabeled image data. The method of claim 5 , wherein selecting the representative image comprises selecting the representative image based on a center of a corresponding one of the clusters of the unlabeled image data.

7. The method of claim 1, wherein generating unsupervised learning hints comprises computing instance-level contrastive loss and cluster-level contrastive loss.

8. A computing device configured to perform data preselection for an object detection system, the computing device comprising a processing device configured to execute instructions stored in a memory to perform the following operations: receiving a first dataset, wherein the first dataset includes unlabeled data corresponding to one or more images; Providing a first dataset and a plurality of learnable prompt vectors to a pre-trained model, wherein the learnable prompt vectors include text input; Using the pre-trained model, generating unsupervised learning prompts based on the first dataset and a plurality of learnable prompt vectors, wherein the unsupervised learning prompts correspond to multimodal features of one or more images of the first dataset; extracting features from either the first dataset or the second dataset based on unsupervised learning prompts; Select and label a subset of instances of the extracted features; as well as Generate and output a labeled dataset based on the labeled instance subset.

9. The computing device of claim 8, wherein the pre-trained model is a Bootstrapped Language-Image Pre-training (BLIP-2) model.

10. The computing device of claim 8, wherein the extracted features include clusters of unlabeled image data.

11. The computing device of claim 10, wherein the selected subset of instances comprises one or more clusters of unlabeled data.

12. The computing device of claim 8, wherein the processing device is configured to execute instructions to select a representative image for labeling from each cluster of unlabeled image data.

13. The computing device of claim 12, wherein the processing device is configured to execute instructions to select the representative image based on a center of a corresponding one of the clusters of unlabeled image data.

14. The computing device of claim 8, wherein the processing device is configured to execute instructions to calculate instance-level contrastive loss and cluster-level contrastive loss.

15. A computer-controlled machine comprising: at least one sensor configured to generate an input image; A control system is configured to perform data preselection for the object detection system, the control system being configured to perform the following operations: receiving a first dataset, wherein the first dataset includes unlabeled data corresponding to one or more images; Providing a first dataset and a plurality of learnable prompt vectors to a pre-trained model, wherein the learnable prompt vectors include text input; Using the pre-trained model, generating unsupervised learning prompts based on the first dataset and a plurality of learnable prompt vectors, wherein the unsupervised learning prompts correspond to multimodal features of one or more images of the first dataset; extracting features from either the first dataset or the second dataset based on unsupervised learning prompts; Select and label a subset of instances of the extracted features; as well as generating and outputting a labeled dataset based on the labeled instance subset; and An actuator is configured to control the operation of a computer-controlled machine based on the labeled data set.

16. The computer-controlled machine of claim 15, wherein the pre-trained model is a Bootstrapped Language-Image Pre-training (BLIP-2) model.

17. The computer-controlled machine of claim 15, wherein the extracted features comprise clusters of unlabeled image data, and wherein the selected subset of instances comprises one or more clusters of unlabeled data.

18. The computer controlled machine of claim 15, wherein the control system is configured to select a representative image for labeling from each cluster of unlabeled image data.

19. A computer controlled machine according to claim 18, wherein the control system is configured to select the representative image based on the center of a corresponding one of the clusters of unlabeled image data.

20. The computer-controlled machine of claim 15, wherein the control system is configured to calculate instance-level contrastive loss and cluster-level contrastive loss.

Images