Enhanced open world target detection using hierarchical structure learning with hyperbolic distance

Abstract

A method of performing open-world target detection includes: receiving target data, including embedded data corresponding to an embedding for a known target in a first input image; projecting the embedding into a hyperbolic embedding space, comprising embedding in a plurality of target classes, each of the plurality of target classes comprising one or more target classes; regularizing the projection embedding within the hyperbolic embedding space by moving each projection embedding closer to an embedding in a same one of the plurality of categories and farther from an embedding in a different one of the plurality of categories; receiving an unmatched query corresponding to a target in the second input image; and generating an output signal based on the hyperbolic embedding space including regularization embedding, the output signal indicating whether the target in the second input image corresponds to an unknown target in one of the target classes.

Description

Technical Field

[0001] The present disclosure relates to open-world object detection (OWOD), and more particularly to systems and methods of operating devices implementing OWOD. Background Art

[0002] Advances in object detection (OD) have opened up a plethora of real-world applications, such as robotics or self-driving cars. Breakthroughs in attention-based neural network architectures (e.g., Deformable Transformers) have yielded impressive performance in these settings. However, most of these approaches assume a fixed number of classes (closed-world assumption), which is rarely true in reality. Continuous object detection incrementally adds new classes, leading to a distribution shift of the input and the well-known phenomenon of "catastrophic forgetting," where the network forgets previously learned knowledge.

[0003] Open-world object detection (OWOD) is a challenging and realistic task that goes beyond the scope of standard object detection tasks. Open-world systems introduce the detection and integration of new classes. Therefore, OWOD involves detecting both known and unknown objects while integrating the learned knowledge for future tasks. Summary of the invention

[0004] A method for performing open-world object detection comprises: receiving object data, the object data comprising embedding data corresponding to embeddings for known objects in a first input image; projecting the embeddings into a hyperbolic embedding space, the hyperbolic embedding space comprising embeddings in a plurality of object categories, the plurality of object categories each comprising one or more object classes; regularizing the projected embeddings within the hyperbolic embedding space by moving each projected embedding closer to an embedding in a same category among the plurality of categories and further away from an embedding in a different category among the plurality of categories; receiving an unmatched query corresponding to an object in a second input image; and generating an output signal based on the hyperbolic embedding space comprising the regularized embeddings, the output signal indicating whether the object in the second input image corresponds to an unknown object in one of the object classes.

[0005] In other features, the target data includes a set of bounding boxes and class labels for known targets in a first input image. The first input image corresponds to a training image. The method also includes determining a hyperbolic contrastive loss corresponding to the hyperbolic embedding space, and regularizing the projected embedding includes regularizing the projected embedding based on the hyperbolic contrastive loss. Regularizing the projected embedding includes determining a corresponding hyperbolic mean of the embeddings in each target class. Generating the output signal includes determining a threshold distance based on the hyperbolic means, and generating the output signal based in part on the threshold distance. Generating the output signal includes determining whether the unmatched query is less than the threshold distance from at least one of the hyperbolic means.

[0006] A computing device configured to perform open-world object detection includes a processing device configured to execute instructions stored in a memory to receive object data, wherein the object data includes embedding data corresponding to an embedding for a known object in a first input image; project the embedding into a hyperbolic embedding space, the hyperbolic embedding space including embeddings in multiple object categories, each of the multiple object categories including one or more object classes; regularize the projected embeddings within the hyperbolic embedding space by moving each projected embedding closer to an embedding in the same category of the multiple categories and further away from an embedding in a different category of the multiple categories; receive an unmatched query corresponding to an object in a second input image; and generate an output signal based on the hyperbolic embedding space including the regularized embedding, the output signal indicating whether the object in the second input image corresponds to an unknown object in one of the object classes.

[0007] In other features, the target data includes a set of bounding boxes and class labels for known targets in a first input image. The first input image corresponds to a training image. The processing device is also configured to execute instructions stored in a memory to generate a hyperbolic contrastive loss corresponding to the hyperbolic embedding space, wherein regularizing the projected embedding includes regularizing the projected embedding based on the hyperbolic contrastive loss. To regularize the projected embedding, the processing device is also configured to execute instructions stored in the memory to determine the corresponding hyperbolic means of the embeddings in each of the target classes. To generate the output signal, the processing device is also configured to execute instructions stored in the memory to determine a threshold distance based on the hyperbolic means and generate the output signal based in part on the threshold distance. To generate the output signal, the processing device is also configured to execute instructions stored in the memory to determine whether the unmatched query is less than the threshold distance from at least one of the hyperbolic means.

[0008] A computer-controlled machine comprising: at least one sensor configured to generate a first input image and a second input image; a control system configured to: receive target data, the target data comprising embedding data corresponding to an embedding for a known target in the first input image; project the embedding into a hyperbolic embedding space, the hyperbolic embedding space comprising embeddings in a plurality of target categories, the plurality of target categories each comprising one or more target classes; regularize the projected embeddings within the hyperbolic embedding space by moving each projected embedding closer to an embedding in the same category of the plurality of categories and further away from an embedding in a different category of the plurality of categories; receive an unmatched query corresponding to a target in the second input image; and generate an output signal based on the hyperbolic embedding space comprising the regularized embedding, the output signal indicating whether the target in the second input image corresponds to an unknown target in one of the target classes; and an actuator configured to control operation of the computer-controlled machine in response to the output signal.

[0009] In other features, the computer-controlled machine also includes a memory storing data corresponding to the hyperbolic embedding space. The control system is also configured to generate a hyperbolic contrast loss corresponding to the hyperbolic embedding space, and wherein regularizing the projected embedding includes regularizing the projected embedding based on the hyperbolic contrast loss. To regularize the embedding, the control system is also configured to determine a corresponding hyperbolic mean of the embeddings in each of the target classes. To generate the output signal, the control system is also configured to determine a threshold distance based on the hyperbolic means and determine whether the unmatched query is less than the threshold distance from at least one of the hyperbolic means. The computer-controlled machine includes an autonomous robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A system for training a neural network according to the principles of the present disclosure is generally shown.

[0011] Figure 2 A computer-implemented method for training and utilizing a neural network in accordance with the principles of the present disclosure is generally presented.

[0012] Figure 3A An audio data tagging system according to the principles of the present disclosure is generally shown.

[0013] Figure 3B A portion of a data capture system in accordance with the principles of the present disclosure is generally shown.

[0014] Figure 3C An alternative audio data tagging system in accordance with the principles of the present disclosure is generally shown.

[0015] Figure 4Ais a functional block diagram of an example hyperbolic open world system in accordance with principles of the present disclosure.

[0016] Figure 4B is a table showing example performance of a dataset in accordance with principles of the present disclosure relative to other example datasets for various tasks.

[0017] Figure 4C A T-distributed stochastic neighbor (t-SNE) plot is shown illustrating learned embeddings using hyperbolic distance in accordance with principles of the present disclosure.

[0018] Figure 4D is a table showing an example ablation analysis quantifying the impact of various components of the disclosed techniques.

[0019] Figure 4E Depicted is an example class distance heat map quantizing hyperbolic distances in accordance with the principles of the present disclosure.

[0020] Figure 4F Depicted are example distance heat maps quantifying hyperbolic distances between known and unknown terms in accordance with principles of the present disclosure.

[0021] Figure 4G Depicted are t-SNE plots of learned embeddings using cosine and hyperbolic distance metrics in accordance with principles of the present disclosure.

[0022] Figure 4H The steps of an example hyperbolic open world method according to the principles of the present disclosure are shown.

[0023] Figure 5 Depicted is a schematic diagram of the interaction between a computer-controlled machine and a control system in accordance with the principles of the present disclosure.

[0024] Figure 6 Depicted is a Figure 5 Schematic diagram of a control system configured to control a vehicle, which can be a partially autonomous vehicle, a fully autonomous vehicle, a partially autonomous robot, or a fully autonomous robot.

[0025] Figure 7 Depicted Figure 5 Schematic diagram of a control system configured to control a manufacturing machine (e.g., a punch cutter, a cutter, or a gun drill) of a manufacturing system (e.g., part of a production line).

[0026] Figure 8 Depicted Figure 5 Schematic diagram of a control system configured to control a power tool, such as a drill or driver, having an at least partially autonomous mode.

[0027] Fig. 9Depicting a device configured to control an automated personal assistant Figure 5 Schematic diagram of the control system.

[0028] Fig.10 Depicted Figure 5 Schematic diagram of a control system configured to control a surveillance system, such as a controlled access system or a monitoring system.

[0029] Fig.11 Depicted Figure 5 Schematic diagram of a control system configured to control an imaging system, such as an MRI device, an x-ray imaging device, or an ultrasound device. DETAILED DESCRIPTION

[0030] Embodiments of the present disclosure are described herein. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various alternative forms. The drawings are not necessarily drawn to scale; some features may be enlarged or minimized to show the details of a particular component. Therefore, the specific structural and functional details disclosed herein should not be interpreted as restrictive, but only as a representative basis for teaching those skilled in the art to adopt the embodiments in various ways. As will be understood by those of ordinary skill in the art, the various features illustrated and described with reference to any one of the drawings may be combined with the features illustrated in one or more other drawings to produce embodiments that are not explicitly illustrated or described. The combination of features shown provides representative embodiments of typical applications. However, for specific applications or implementations, various combinations and modifications of features consistent with the teachings of the present disclosure may be required.

[0031] Unless the context clearly indicates otherwise, "a", "an", and "the" as used herein refer to both singular and plural referents. For example, a "processor" programmed to perform various functions refers to one processor programmed to perform each function, or more than one processor programmed together to perform each of the various functions.

[0032] Open-world object detection (OWOD) involves detecting both known (previously seen and classified during training) and unknown (not previously seen and classified and / or outside of known classes) objects. However, the level of "unknown" varies significantly depending on the context. For example, in autonomous driving scenarios, trees are often seen as part of the background, but may also be important in a home context.

[0033] The OWOD framework includes both detection tasks and continuous learning. However, a major challenge in this framework is the lack of annotations for unknown objects, which leads to a bias towards known labels and potential confusion between unknown items and background. This bias severely hinders the accurate identification of unknown objects and creates a major obstacle in the detection process.

[0034] Previous attempts to address this problem involve various techniques such as relying on learned objectness scores to re-label the background as potential unknowns, clustering classes to better isolate the unknowns, and decoupled approaches where the class and localization heads are separated. The goal is to remove the labeling information and instead capture shared features that make the labeled items relevant as objects.

[0035] However, these techniques fail to address the critical issue of defining what constitutes an “unknown” target. Currently, there is no clear definition or prior knowledge available to effectively distinguish between unknown targets and background, and the interpretation varies greatly depending on the context. For example, in a driving scenario, “debris on the road” can be considered an unknown target that is relevant to the task. In contrast, in a camera monitoring context, “debris on the road” can be considered part of the background. Without considering the context, previous techniques can only learn about low-level features (such as texture or shape) to distinguish between known and unknown. Therefore, modeling the hierarchy and similarity between known and unknown items is not performed either at the image level or the dataset level.

[0036] Thus, systems and methods according to the principles of the present disclosure define a hierarchical relationship between objects to be discovered and known items or objects. The hierarchy is characterized by classes that share the same semantic context and belong to the same category (e.g., vehicles, animals, electronic devices, etc.). This hierarchical relationship enables the retrieval of common features and facilitates the discovery of unknown objects. For example, a model trained on objects relevant to driving scenarios may adequately detect stop signs or traffic lights, but is not expected to recognize unrelated objects, such as sofas or other furniture.

[0037] Thus, as implemented in the systems and methods of the present disclosure, external or contextual information is already embedded in the known classes. In other words, there is a semantic or underlying structural relationship between the unknown and known items to be discovered. The systems and methods described herein use a regularizer (e.g., a superclass regularizer) configured according to the principles of the present disclosure to learn and model a hierarchical representation of known items. Utilizing the learned hierarchical representation enables the use of adaptive re-labeling (similarity distance-based re-labeling modules, circuits, etc.) to detect unknown targets.

[0038] In this way, hierarchical relationships between items are learned so that representations of known objects can be effectively used to discover unknown items. For example, items belonging to the same family (or category) may be closer to each other and farther away from different families (e.g., closer to other animals and farther away from vehicles). To capture this structural hierarchy, the systems and methods described herein compute and operate based on hyperbolic distances (which map hierarchical latent structures, such as graphs or trees, and serve as natural distance metrics). This mapping in hyperbolic space exhibits the desirable property of capturing the degree of match between unknown and known items, thereby enhancing the detection of unknown objects.

[0039] Figure 1 An example system 100 for training a neural network (e.g., of an ML model) is shown. System 100 may be configured to (and / or include circuitry configured to) implement the systems and methods of the present disclosure described in more detail below. System 100 may include an input interface for accessing training data 102 for the neural network. For example, Figure 1 As shown, the input interface can be constituted by a data storage interface 104, which can access the training data 102 from a data storage 106. For example, the data storage interface 104 can be a memory interface or a permanent storage interface, such as a hard disk or SSD interface, but can also be a personal area network, a local area network, or a wide area network interface, such as a Bluetooth, Zigbee or Wi-Fi interface or an Ethernet or fiber optic interface. The data storage 106 can be an internal data storage of the system 100, such as a hard disk drive or SSD, or an external data storage, such as a network accessible data storage.

[0040] In some embodiments, data store 106 may further include a data representation 108 of an untrained version of the neural network, which may be accessed by system 100 from data store 106. However, it should be understood that data representation 108 of the untrained neural network and training data 102 may also each be accessed from different data stores, such as via different subsystems of data storage interface 104. Each subsystem may be of the type described above for data storage interface 104.

[0041] In some embodiments, the data representation 108 of the untrained neural network may be generated internally by the system 100 based on the design parameters of the neural network and may not be explicitly stored on the data storage 106. The system 100 may also include a processor subsystem 110 that may be configured to provide an iterated function as a replacement for the layer stack of the neural network to be trained during operation of the system 100. Herein, the layers of the replaced layer stack may have mutually shared weights and may receive the output of the previous layer as input, or, for the first layer of the layer stack, receive an initial activation and a portion of the input of the layer stack.

[0042] The processor subsystem 110 may also be configured to iteratively train the neural network using the training data 102. Herein, iterations of training performed by the processor subsystem 110 may include a forward propagation portion and a backward propagation portion. The processor subsystem 110 may be configured to perform the forward propagation portion by, among other operations, defining a forward propagation portion that may be performed, determining an equilibrium point of the iteration function where the iteration function converges to a fixed point (where determining the equilibrium point includes using a numerical root finding algorithm to find a root solution of the iteration function minus its input), and by providing the equilibrium point as a surrogate for the output of a stack of layers in the neural network.

[0043] The system 100 may also include an output interface for outputting a data representation 112 of the trained neural network. This data may also be referred to as training model data 112. For example, Figure 1 As shown, the output interface can be constituted by a data storage interface 104, wherein in these embodiments, the interface is an input / output ("IO") interface, through which the trained model data 112 can be stored in the data storage 106. For example, during or after training, the data representation 108 defining the "untrained" neural network can be at least partially replaced by the data representation 112 of the trained neural network, as the parameters of the neural network (e.g., weights, hyperparameters, and other types of parameters of the neural network) can be adapted to reflect the training on the training data 102. This is also Figure 1 106 by reference numerals 108, 112, which represent the same data record on the data store 106. In some embodiments, the data representation 112 may be stored separately from the data representation 108 defining the "untrained" neural network. In some embodiments, the output interface may be separate from the data store interface 104, but may generally be of the type described above for the data store interface 104.

[0044] Figure 2 A data annotation / augmentation system 200 configured to (and / or including circuitry configured to) implement a system for annotating and / or augmenting data is depicted. The data annotation system 200 may include at least one computing system 202 configured to implement all or part of the systems and methods of the present disclosure explained in more detail below. The computing system 202 may include at least one processor 204 operably connected to a memory unit 208. The processor 204 may include one or more integrated circuits that implement the functionality of a central processing unit (CPU) 206. The CPU 206 may be a commercially available processing unit that implements an instruction set, such as one of the x86, ARM, Power, or MIPS instruction set families. The various components of the system 200 may be implemented with the same or different circuits.

[0045] During operation, CPU 206 may execute stored program instructions retrieved from memory unit 208. The stored program instructions may include software that controls the operation of CPU 206 to perform the operations described herein. In some embodiments, processor 204 may be a system on a chip (SoC) that integrates the functionality of CPU 206, memory unit 208, network interfaces, and input / output interfaces into a single integrated device. Computing system 202 may implement an operating system for managing various aspects of operation.

[0046] Memory unit 208 may include volatile memory and non-volatile memory for storing instructions and data. Non-volatile memory may include solid-state memory, such as NAND flash memory, magnetic and optical storage media, or any other suitable data storage device that retains data when computing system 202 is deactivated or powered off. Volatile memory may include static and dynamic random access memory (RAM) that stores program instructions and data. For example, memory unit 208 may store machine learning model 210 (e.g., in Figure 2 210), or an algorithm, a training data set 212 for the machine learning model 210, an original source data set 216, etc.

[0047] The computing system 202 may include a network interface device 222 configured to provide communications with external systems and devices. For example, the network interface device 222 may include a wired and / or wireless Ethernet interface defined by the Institute of Electrical and Electronics Engineers (IEEE) 802.11 family of standards. The network interface device 222 may include a cellular communication interface for communicating with a cellular network (e.g., 3G, 4G, 5G). The network interface device 222 may also be configured to provide a communication interface to an external network 224 or cloud.

[0048] The external network 224 may be referred to as the World Wide Web or the Internet. The external network 224 may establish a standard communication protocol between computing devices. The external network 224 may allow information and data to be easily exchanged between computing devices and the network. One or more servers 230 may communicate with the external network 224.

[0049] Computing system 202 may include input / output (I / O) interface 220, which may be configured to provide digital and / or analog input and output. I / O interface 220 may include an additional serial interface (eg, a universal serial bus (USB) interface) for communicating with external devices.

[0050] The computing system 202 may include a human-machine interface (HMI) device 218, which may include any device that enables the system 200 to receive control inputs. Examples of input devices may include human-machine interface inputs such as keyboards, mice, touch screens, voice input devices, and other similar devices. The computing system 202 may include a display device 232. The computing system 202 may include hardware and software for outputting graphics and text information to the display device 232. The display device 232 may include an electronic display screen, a projector, a printer, or other suitable devices for displaying information to a user or operator. The computing system 202 may also be configured to allow interaction with a remote HMI and a remote display device via a network interface device 222.

[0051] System 200 can be implemented using one or more computing systems. Although this example depicts a single computing system 202 that implements all of the described features, it is intended that the various features and functions can be separated and implemented by multiple computing units that communicate with each other. The specific system architecture selected may depend on a variety of factors.

[0052] The system 200 may implement a machine learning model 210 configured to analyze a raw source data set 216. For example, the CPU 206 and / or other circuitry may implement the machine learning model 210. The raw source data set 216 may include raw or unprocessed sensor data, which may represent an input data set for a machine learning system. The raw source data set 216 may include video, video clips, images, audio, text-based information, and raw or partially processed sensor data (e.g., a radar map of a target). In some embodiments, the machine learning model 210 may be a deep learning or neural network algorithm designed to perform a predetermined function. For example, a neural network algorithm may be configured to identify an event or target in a video clip based on audio data.

[0053] The computer system 200 may store a training data set 212 for the machine learning model 210. The training data set 212 may represent a set of previously constructed data for training the machine learning model 210. For example, a training data set 212 according to the present disclosure may include a plurality of automatically collected ground truth measurements and associated data. The machine learning model 210 may use the training data set 212 to learn weighting factors associated with a neural network algorithm. The training data set 212 may include a set of source data having corresponding achievements or results that the machine learning model 210 attempts to replicate through the learning process.

[0054] The machine learning model 210 can operate in a learning mode using the training data set 212 as input. The machine learning model 210 can be executed in multiple iterations using data from the training data set 212. For each iteration, the machine learning model 210 can update the internal weighting factors based on the results obtained. For example, the machine learning model 210 can compare the output results (e.g., annotations) with those included in the training data set 212. Since the training data set 212 includes expected results, the machine learning model 210 can determine when the performance is acceptable. After the machine learning model 210 reaches a predetermined performance level (e.g., 100% consistent with the results associated with the training data set 212), the machine learning model 210 can be executed using data that is not in the training data set 212. The trained machine learning model 210 can be applied to a new data set to generate annotated data.

[0055] The machine learning model 210 may be configured to identify specific features in the original source data 216. The original source data 216 may include multiple instances or input data sets (e.g., video streams or segments including audio data) for which annotation results are desired. By way of example only, the machine learning model 210 may be configured to identify targets or events in video segments and annotate events based on audio data. The machine learning model 210 may be programmed to process the original source data 216 to identify the presence of specific features. The machine learning model 210 may be configured to identify features in the original source data 216 as predetermined features. The original source data 216 may be obtained from a variety of sources. For example, the original source data 216 may be actual input data collected by a machine learning system. The original source data 216 may be machine generated for testing a system. As an example, the original source data 216 may include original video and / or audio data from a camera, audio data from a microphone, and the like.

[0056] In an example, the machine learning model 210 can process the raw source data 216 and output video and / or audio data including one or more indications of the identified events. The machine learning model 210 can generate a confidence level or factor for each generated output. For example, a confidence value exceeding a predetermined high confidence threshold can indicate that the machine learning model 210 is confident that the identified event (or feature) corresponds to a particular event. A confidence value less than a low confidence threshold can indicate that the machine learning model 210 has some uncertainty that a particular feature exists.

[0057] As in Figure 3A and 3B, system 300 may include an image (e.g., video) capture device 302, an audio capture array 304, and a computing system 202. The system may receive video stream data associated with a data capture environment from the image capture device 302. The system 202 may be configured to perform video target detection to identify one or more targets in a corresponding image of the video stream data. The system 202 may receive audio stream data corresponding to at least a portion of the video stream data from the audio capture array 304. The audio capture array 304 may include one or more microphones 306 or other suitable audio capture devices. The systems and methods described herein may be configured to use output from at least a first machine learning model (e.g., such as machine learning model 210 or other suitable machine learning model configured to provide output including one or more target or event detection predictions) to mark at least some targets of the video stream data and / or the audio stream data.

[0058] The system 202 may calculate at least one offset value for at least a portion of the audio stream data corresponding to at least one marked target of the video stream data based on at least one data capture characteristic (e.g., using at least one probability-based function or other suitable technique or function). The system 202 may synchronize at least a portion of the video stream data with a portion of the audio stream data corresponding to the at least one marked target of the video stream data using at least the at least one offset value. The at least one data capture characteristic may include one or more characteristics of at least one image capture device, one or more characteristics of at least one audio capture array, one or more characteristics corresponding to a position of at least one image capture device relative to at least one audio capture array, one or more characteristics corresponding to movement of a target in the video stream data, one or more other suitable data capture characteristics, or a combination thereof.

[0059] System 202 may use one or more tags and at least one offset value of the tag target of the video stream data to mark at least a portion of the audio stream data corresponding to at least one tag target of the video stream data. Each corresponding tag may include an event type, an event start indicator, and an event end indicator. System 202 may use at least some of the marked portions of the audio stream data to generate training data. System 202 may use the training data to train a second machine learning model. System 202 may use the second machine learning model to detect one or more sounds associated with the audio data provided as input to the second machine learning model. The second machine learning model may include any suitable machine learning model and may be configured to perform any suitable functions, such as those described herein with respect to Figures 4-11.

[0060] In some embodiments, such as in Figure 3CAs generally shown in , the computing system 202 can be configured to tag audio data based on sensor data received from one or more sensors (such as those described herein or any other suitable sensor or combination of sensors). The system 202 can receive audio stream data associated with the data capture environment from the audio capture array 354 or any suitable audio capture device (such as one or more of the microphones 306 or other suitable audio capture devices). It should be understood that the audio capture array 354 may include features similar to those of the audio capture array 304, and may include any suitable number of audio capture devices. The system 202 can receive sensor data associated with the data capture environment from at least one sensor (e.g., sensor 352) that is asynchronous with respect to the audio capture array 354. The sensor 354 may include at least one of an induction coil, a radar sensor, a LiDAR sensor, a sonar sensor, an image capture device, any other suitable sensor, or a combination thereof. The audio capture array 354 can be positioned away from the sensor 354, positioned close to the sensor 354, or positioned in any suitable relationship with the sensor 354.

[0061] The system 202 may use output from at least a first machine learning model, such as the machine learning model 210 or other suitable machine learning models, to identify at least some events in the sensor data. The machine learning model 210 may be configured to provide an output including one or more event detection predictions based on the sensor data. The system 202 may synchronize at least a portion of the sensor data associated with a portion of the audio stream data corresponding to at least one event of the sensor data. The system 202 may use one or more tags extracted for the corresponding events of the sensor data values ​​to mark at least a portion of the audio stream data corresponding to at least one event of the sensor data. Each corresponding tag may include an event type, an event start indicator, and an event end indicator. The system 202 may use at least some of the marked portions of the audio stream data to generate training data. The system 202 may use the training data to train a second machine learning model. The system 202 may use the second machine learning model to detect one or more sounds associated with the audio data provided as input to the second machine learning model. The second machine learning model may include any suitable machine learning model and may be configured to perform any suitable functions, such as those described herein with respect to Figures 4-11.

[0062] Any of the systems described above and / or in more detail below may implement the systems and methods of the present disclosure to perform OWOD.The systems and methods of the present disclosure may be referred to as Hyperbolic Open World ("Hyp-OW") systems and methods.

[0063] Figure 4A4 is a functional block diagram of an exemplary Hyp-OW system 400 according to the present disclosure. The system 400 includes one or more convolutional neural networks, such as a deformable transformer 402, which is configured to receive and process input corresponding to an observed scene (e.g., an input image 404). For example, the deformable transformer 402 implements an object detection model that performs a deformable convolution on a specific region of the input image 404. The output 406 of the deformable transformer is a set of bounding boxes and class labels for objects detected in the input image 404, which are provided to a classification system 408 according to the present disclosure.

[0064] The classification system 408 includes a classifier 410 (e.g., a classifier configured to classify an input signal into one or more labels using a machine learning (ML) algorithm such as a neural network), a hyperbolic contrastive loss module 412 configured to learn a hierarchical representation of each class, and a superclass regularizer module 414 configured to regularize the classes so that classes belonging to the same category are closer to each other in an embedding space (e.g., a hyperbolic embedding space) while being farther away from classes of different categories. An adaptive relabeling module 416 utilizes the learned hierarchy to relabel proposed candidates as unknown if they meet certain criteria described in more detail below.

[0065] For example, various embeddings 418 (e.g., vector representations, which may be referred to as embedding data) of images or image regions are positioned in an embedding space 420. Ensuring that classes belonging to the same category (e.g., vehicles, animals, electronic devices, etc.) are closer to each other in the embedding space 420 and farther away from classes from different categories, new (unmatched) embeddings may be moved closer to other classes in the same category. An adaptive relabeling module 416 receives known items and unknown candidates (e.g., from an input image 422) and may selectively relabel the embeddings (e.g., based on a comparison between a distance d of the unmatched embedding from a predetermined or calculated position (e.g., a centroid position) of the class within the embedding space 420 and a threshold distance δ). For example, if the distance d is less than or equal to the threshold distance δ, the embedding may be relabeled. Conversely, if the distance d is greater than the threshold distance δ, the embedding may not be relabeled.

[0066] In this manner, the Hyp-OW system 400 is configured to effectively capture the hierarchical relationships between known classes and unknown classes, thereby leading to improved object detection performance.

[0067] In an embodiment, each embedding is projected into a hyperbolic space with an exponential map (i.e., a hyperbolic embedding space) according to the following formula:

[0068]

[0069] For each embedding query q, the corresponding projection z into the hyperbolic space is defined as follows:

[0070]

[0071] The hyperbolic contrastive loss module 412 is configured to learn a hierarchical representation for each class to bring the embeddings of the same class closer together while pushing each class away from the embeddings of different classes. i c corresponds to queries that match class c∈K. For each query z i c , samples a positive example from the buffer M. The query z where i∈A={B∪M} i c is the query embedding of class c, which can come from batch B or buffer M. Sample z i+ corresponds to positive samples, while z i- The sample corresponds to the negative sample. In order to attract the embeddings of the query and positive samples while repelling the embeddings from the negative samples, the hyperbolic contrastive loss can be written as follows:

[0072]

[0073] where τ1 is the temperature, and

[0074] The superclass regularizer module 414 is configured to bring classes from the same category closer to each other and push them away from classes belonging to different categories. p , in order to approximate the embedding of category p, according to To calculate the class c∈S p Each embedding The hyperbolic mean of corresponds to the Lorentz factor.

[0075] For each embedding query z i c , sample the category embedding H from the buffer M p (c∈S p ), and use the same hyperbolic contrastive loss according to the following formula:

[0076]

[0077] The adaptive relabeling module 416 is configured to implement the adaptive relabeling for queries z from the incoming batch B that have been matched with the ground-truth labels. m and queries z that do not match any ground truth labels (which may be referred to as “proposal candidates”) u An adaptive re-labeling scheme for Z ccorresponds to the hyperbolic mean of each class c from the buffer M and represents the centroid in the hyperbolic embedding space of each class. Compute matching query z m and the distance between the centroid, to identify the maximum distance δ according to the following formula B :

[0078]

[0079] Calculate each unmatched query z u With all the centroids Z c and choose the distance between them for each z u If the minimum distance is less than δ B , we can assume that the unmatched query may contain unknown items. Therefore, each unmatched query z u Can be relabeled in response to the following conditions being true:

[0080]

[0081] Unmatched queries can then be re-labeled as unknown and forwarded for classification.

[0082] The final loss L can be calculated according to the following formula:

[0083] L = l cls +L bbox +αL hyp +βL reg , where coefficients α and β weight the hyperbolic contrastive loss and superclass regularizer respectively.

[0084] Figure 4B Table 424 is shown, which shows example performance of a dataset ("Hyp-OW") of the systems and methods of the present disclosure for various tasks relative to other example datasets. As shown, the dataset can be evaluated for low, medium, and high difficulty levels. Evaluation metrics include U-Recall (e.g., unknown recall, shown as a percentage representation of unknown objects found) for unknown detection and mAP for known detection. Hyp-OW shows consistent improvements over other techniques in both unknown and known detection at all difficulty levels.

[0085] Figure 4CT-distributed random neighbor (t-SNE) graphs 426 and 428 are shown, which show the learned embeddings implemented using the hyperbolic distance, where the superclass regularizer module 414 according to the present disclosure is used (at 426) and without the superclass regularizer module 414 according to the present disclosure (at 428). Each class is represented based on the corresponding category. As shown, the use of the superclass regularizer module 414 as described herein implements a more different grouping of classes belonging to the same category, thereby enhancing the hierarchy and effectively separating the embeddings of different categories.

[0086] Figure 4D Table 432 is shown, which shows an example ablation analysis quantifying the impact of various components of the Hyp-OW technique of the present disclosure (e.g., results obtained using cosine distance, results obtained without using superclass regularizer module 414, and results obtained without using adaptive relabeling). The results show that the adaptive relabeling module 416 improves the accuracy of known detections (mAP), while the superclass regularizer module 414 improves the accuracy of unknown detections (U-Recall). In contrast, using cosine distance (i.e., not hyperbolic distance) results in reduced accuracy of unknown detections.

[0087] Figure 4E Example class distance heatmaps 436 and 438 are shown that quantify the hyperbolic distance (e.g., from 1.0 to 1.5) between class embeddings (e.g., vehicles, animals, outdoor, furniture) and individual classes (e.g., bicycles, cars, motorcycles, airplanes, etc.), where lighter shades represent smaller distances (and greater similarities) and darker shades represent larger distances (and less similarities). Results obtained using the superclass regularizer module 414 are shown at 436, and results obtained without the superclass regularizer module 414 are shown at 438. Heatmap 436 shows that the use of the superclass regularizer module 414 enhances the distances within and between classes. For example, as shown, the classes within groups 440, 442, 444, and 446 are closer to their respective classes and farther away from other classes. Therefore, the use of the superclass regularizer module 414 separates different classes from each other in hyperbolic space and groups classes in the same class more closely together in hyperbolic space. In contrast, without the use of the superclass regularizer module 414 as shown at 438, the distances between classes in the hyperbolic space appear less distinct, as indicated by the weaker shading intensity.

[0088] Figure 4FAn example distance heatmap 450 is shown that quantifies the hyperbolic distance between known items and unknown items for a task (e.g., Task 1) of hierarchical segmentation, where lighter shades represent smaller distances (and greater similarity) and darker shades represent larger distances (and smaller similarity). As shown, smaller distances are observed between known and unknown items that belong to the same category (as indicated by box 452 in some examples). Therefore, unknown items can be effectively detected and labeled (e.g., by adaptive relabeling module 416) based on their similarity to known items.

[0089] Figure 4G t-SNE plots 456 and 458 of the learned embeddings using two different distance metrics: cosine distance (at 456) and hyperbolic distance (at 458) are shown. The use of hyperbolic distance shapes the embeddings in a hierarchical manner so that classes belonging to the same category present more cohesive groupings (i.e., improved clustering) and embeddings of different categories are effectively separated.

[0090] Figure 4H Steps of an exemplary hyperbolic open-world object detection method 460 in accordance with principles of the present disclosure are shown. At 462, the method 460 receives object data corresponding to an input image. For example, the method 460 receives a set of bounding boxes and class labels for objects detected in the input image, which correspond to embeddings of known items. At 464, the method 460 projects the embeddings into a hyperbolic space. At 466, the method 460 determines a hyperbolic contrast loss as described above.

[0091] At 468, method 460 regularizes the embeddings in the hyperbolic space. As used herein, "regularization" refers to regularizing the classes so that classes belonging to the same category are closer to each other in the embedding space (e.g., the hyperbolic embedding space) and farther away from classes of different categories (e.g., using the superclass regularizer module 414 as described above). For example, regularizing the embeddings according to the principles of the present disclosure includes determining a hyperbolic mean of the embeddings in a given class.

[0092] At 470, the method 460 performs adaptive re-labeling (e.g., using the adaptive re-labeling module 416) on the proposed unknown candidates (e.g., unmatched queries) from the input image. For example, the proposed unknown candidates that are less than a predetermined distance (e.g., the threshold distance described above) from at least one calculated centroid of all classes are re-labeled as unknown.

[0093] As described above, systems and methods according to the principles of the present disclosure emphasize semantic relationships between known and unknown items. For example, a Hyp-OW dataset according to the principles of the present disclosure is configured to define a high-level hierarchy between items that emphasize semantic relationships. As described above, the Hyp-OW system and method: (i) applies a hyperbolic contrastive loss to learn and model a hierarchy by moving embeddings of the same class closer together and moving embeddings away from embeddings of different classes (e.g., using a hyperbolic contrastive loss module 412); (ii) moves classes from the same category closer together while pushing classes away from classes of different categories (e.g., using a superclass regularizer module 414); and (iii) implements adaptive relabeling to dynamically relabel proposed candidates as unknown using semantic similarity distance to exploit the learned hierarchy (e.g., using an adaptive relabeling module 416).

[0094] Figure 5-11 Depicted are example systems and devices that may implement OWOD systems and methods according to the present disclosure. Figure 5 A schematic diagram of the interaction between a computer-controlled machine 500 and a control system 502 is depicted. The computer-controlled machine 500 includes an actuator 504 and a sensor 506. The actuator 504 may include one or more actuators, and the sensor 506 may include one or more sensors. The sensor 506 is configured to sense a condition of the computer-controlled machine 500. The sensor 506 may be configured to encode the sensed condition into a sensor signal 508 and send the sensor signal 508 to the control system 502. Non-limiting examples of the sensor 506 include video, radar, LiDAR, ultrasound, and motion sensors. In some embodiments, the sensor 506 is an optical sensor that is configured to sense an optical image of an environment close to the computer-controlled machine 500.

[0095] The control system 502 is configured to receive sensor signals 508 from the computer controlled machine 500. The control system 502 may also be configured to calculate actuator control commands 510 from the sensor signals and send the actuator control commands 510 to the actuators 504 of the computer controlled machine 500, as described below.

[0096] like Figure 5As shown, the control system 502 includes a receiving unit 512. The receiving unit 512 can be configured to receive the sensor signal 508 from the sensor 506 and transform the sensor signal 508 into an input signal x. In an alternative embodiment, without the receiving unit 512, the sensor signal 508 is directly received as the input signal x. Each input signal x can be a part of each sensor signal 508. The receiving unit 512 can be configured to process each sensor signal 508 to generate each input signal x. The input signal x can include data corresponding to the image recorded by the sensor 506.

[0097] The control system 502 includes a classifier 514. The classifier 514 can be configured to classify the input signal x into one or more labels using a machine learning (ML) algorithm (e.g., a neural network). The classifier 514 is configured to be parameterized by parameters such as those described above (e.g., parameter θ). The parameter θ can be stored in and provided by the non-volatile storage 516. The classifier 514 is configured to determine the output signal y from the input signal x. Each output signal y includes information that assigns one or more labels to each input signal x. The classifier 514 can send the output signal y to a conversion unit 518. The conversion unit 518 is configured to convert the output signal y into an actuator control command 510. The control system 502 is configured to send the actuator control command 510 to the actuator 504, and the actuator 504 is configured to actuate the computer-controlled machine 500 in response to the actuator control command 510. In some embodiments, the actuator 504 is configured to actuate the computer-controlled machine 500 directly based on the output signal y.

[0098] When the actuator 504 receives the actuator control command 510, the actuator 504 is configured to perform an action corresponding to the associated actuator control command 510. The actuator 504 may include control logic configured to transform the actuator control command 510 into a second actuator control command for controlling the actuator 504. In one or more embodiments, the actuator control command 510 may be used to control a display instead of or in addition to the actuator.

[0099] In some embodiments, the control system 502 includes the sensor 506 instead of or in addition to the computer-controlled machine 500 including the sensor 506. The control system 502 may also include the actuator 504 instead of or in addition to the computer-controlled machine 500 including the actuator 504.

[0100] like Figure 5As shown, the control system 502 also includes a processor 520 and a memory 522. The processor 520 may include one or more processors. The memory 522 may include one or more memory devices. The classifier 514 (e.g., ML algorithm) of one or more embodiments may be implemented by the control system 502 including the non-volatile storage 516, the processor 520, and the memory 522.

[0101] The non-volatile storage 516 may include one or more persistent data storage devices, such as a hard drive, an optical drive, a tape drive, a non-volatile solid-state device, cloud storage, or any other device capable of persistently storing information. The processor 520 may include one or more devices selected from a high-performance computing (HPC) system, including a high-performance core, a microprocessor, a microcontroller, a digital signal processor, a microcomputer, a central processing unit, a field programmable gate array, a programmable logic device, a state machine, a logic circuit, an analog circuit, a digital circuit, or any other device that manipulates signals (analog or digital) based on computer-executable instructions residing in the memory 522. The memory 522 may include a single memory device or multiple memory devices, including but not limited to random access memory (RAM), volatile memory, non-volatile memory, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, cache memory, or any other device capable of storing information.

[0102] The processor 520 may be configured to read into the memory 522 and execute computer executable instructions residing in the non-volatile storage 516 and embodying one or more anomaly detection methods of one or more embodiments. The non-volatile storage 516 may include one or more operating systems and applications. The non-volatile storage 516 may store content compiled and / or interpreted from a computer program created using various programming languages ​​and / or techniques, including but not limited to Java, C, C++, C#, Objective C, Fortran, Pascal, JavaScript, Python, Perl, and PL / SQL, alone or in combination.

[0103] When executed by processor 520, the computer executable instructions of non-volatile storage 516 may cause control system 502 to implement one or more of the anomaly detection methods disclosed herein. Non-volatile storage 516 may also include data supporting the functions, features, and processes of one or more embodiments described herein.

[0104] The program code embodying the algorithm and / or method described herein can be distributed separately or jointly as a program product in various forms. The program code can be distributed using a computer-readable storage medium having a computer-readable program instruction for causing a processor to perform one or more embodiments. The inherently non-transitory computer-readable storage medium may include volatile and non-volatile, removable and non-removable tangible media implemented by any method or technology for storing information (e.g., computer-readable instructions, data structures, program modules, or other data). The computer-readable storage medium may also include RAM, ROM, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, or other solid-state memory technology, portable compact disk read-only memory (CD-ROM) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices, or any other medium that can be used to store the required information and can be read by a computer. The computer-readable program instruction can be downloaded from the computer-readable storage medium to a computer, another type of programmable data processing device, or another device, or downloaded to an external computer or external storage device via a network.

[0105] The computer-readable program instructions stored in the computer-readable medium can be used to instruct a computer, other types of programmable data processing devices, or other devices to operate in a particular manner so that the instructions stored in the computer-readable medium produce a manufactured product including instructions for implementing the functions, actions, and / or operations specified in the flowchart or figure. In certain alternative embodiments, the functions, actions, and / or operations specified in the flowchart and figure can be reordered, processed serially, and / or processed simultaneously according to one or more embodiments. In addition, any of the flowchart and / or figure may include more or fewer nodes or blocks than those illustrated in accordance with one or more embodiments.

[0106] A process, method or algorithm may be implemented in whole or in part using appropriate hardware components (e.g., application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), state machines, controllers, or other hardware components or devices) or a combination of hardware, software, and firmware components.

[0107] Figure 6A schematic diagram of a control system 502 configured to control a vehicle 600 is depicted, which may be an at least partially autonomous vehicle or an at least partially autonomous robot. The vehicle 600 includes an actuator 504 and a sensor 506. The sensor 506 may include one or more video sensors, cameras, radar sensors, ultrasonic sensors, LiDAR sensors, and / or location sensors (e.g., GPS). One or more of the one or more specific sensors may be integrated into the vehicle 600. Alternatively or in addition to the one or more specific sensors identified above, the sensor 506 may include a software module configured to determine the state of the actuator 504 when executed. A non-limiting example of a software module includes a weather information software module that is configured to determine the current or future state of the weather near the vehicle 600 or other location.

[0108] The classifier 514 of the control system 502 of the vehicle 600 may be configured to detect an object in the vicinity of the vehicle 600 based on the input signal x. In such an embodiment, the output signal y may include information characterizing the vicinity of the object of the vehicle 600. The actuator control command 510 may be determined based on the information. The actuator control command 510 may be used to avoid a collision with the detected object.

[0109] In some embodiments, the vehicle 600 is an at least partially autonomous vehicle, and the actuator 504 may be embodied in the brakes, propulsion system, engine, drive train, or steering of the vehicle 600. The actuator control command 510 may be determined so that the actuator 504 is controlled so that the vehicle 600 avoids colliding with a detected target. The detected targets may also be classified based on what the classifier 514 thinks they are most likely to be, such as a pedestrian or a tree. The actuator control command 510 may be determined based on the classification. In scenarios where a hostile attack may occur, the above system may be further trained to better detect targets or identify changes in the angle or lighting conditions of the sensors or cameras on the vehicle 600.

[0110] In some embodiments where the vehicle 600 is an at least partially autonomous robot, the vehicle 600 may be a mobile robot configured to perform one or more functions, such as flying, swimming, diving, and walking. The mobile robot may be an at least partially autonomous lawn mower or an at least partially autonomous cleaning robot. In such embodiments, the actuator control commands 510 may be determined such that a propulsion unit, a steering unit, and / or a braking unit of the mobile robot may be controlled such that the mobile robot may avoid colliding with an identified target.

[0111] In some embodiments, vehicle 600 is an at least partially autonomous robot in the form of a gardening robot. In such embodiments, vehicle 600 can use an optical sensor as sensor 506 to determine the state of plants in the environment near vehicle 600. Actuator 504 can be a nozzle configured to spray a chemical. Based on the identified species and / or the identified state of the plants, actuator control commands 510 can be determined to cause actuator 504 to spray the plants with an appropriate amount of an appropriate chemical.

[0112] The vehicle 600 may be an at least partially autonomous robot in the form of a household appliance. Non-limiting examples of household appliances include a washing machine, a stove, an oven, a microwave, or a dishwasher. In such a vehicle 600, the sensor 506 may be an optical sensor configured to detect the state of an object to be processed by the household appliance. For example, in the case where the household appliance is a washing machine, the sensor 506 may detect the state of the laundry in the washing machine. The actuator control command 510 may be determined based on the detected state of the laundry.

[0113] Figure 7 A schematic diagram of a control system 502 is depicted, the control system 502 being configured to control a system 700 (e.g., a manufacturing machine), such as a punch cutter, a cutter, or a gun drill, of a manufacturing system 702 (e.g., a portion of a production line). The control system 502 may be configured to control an actuator 504, which is configured to control the system 700 (e.g., a manufacturing machine).

[0114] The sensor 506 of the system 700 (e.g., a manufacturing machine) may be an optical sensor configured to capture one or more characteristics of the manufactured product 704. The classifier 514 may be configured to determine the state of the manufactured product 704 from the one or more captured characteristics. The actuator 504 may be configured to control the system 700 (e.g., a manufacturing machine) for subsequent manufacturing steps of the manufactured product 704 according to the determined state of the manufactured product 704. The actuator 504 may be configured to control the function of the system 700 (e.g., a manufacturing machine) on the subsequent manufactured product 706 of the system 700 (e.g., a manufacturing machine) according to the determined state of the manufactured product 704.

[0115] Figure 8 A schematic diagram of a control system 502 configured to control a power tool 800 , such as a drill or driver, having an at least partially autonomous mode is depicted. The control system 502 may be configured to control an actuator 504 configured to control the power tool 800 .

[0116] The sensor 506 of the power tool 800 may be an optical sensor configured to capture one or more characteristics of the working surface 802 and / or the fastener 804 driven to the working surface 802. The classifier 514 may be configured to determine the state of the working surface 802 and / or the fastener 804 relative to the working surface 802 based on the one or more captured characteristics. The state may be that the fastener 804 is flush with the working surface 802. Alternatively, the state may be the hardness of the working surface 802. The actuator 504 may be configured to control the power tool 800 so that the driving function of the power tool 800 is adjusted based on the determined state of the fastener 804 relative to the working surface 802 or one or more captured characteristics of the working surface 802. For example, if the state of the fastener 804 is flush with the working surface 802, the actuator 504 may interrupt the driving function. As another non-limiting example, the actuator 504 may apply additional or less torque based on the hardness of the working surface 802.

[0117] Fig. 9 Depicted is a schematic diagram of a control system 502 configured to control an automated personal assistant 900. The control system 502 may be configured to control an actuator 504, which is configured to control the automated personal assistant 900. The automated personal assistant 900 may be configured to control a household appliance, such as a washing machine, stove, oven, microwave, or dishwasher.

[0118] Sensor 506 may be an optical sensor and / or an audio sensor. The optical sensor may be configured to receive a video image of gesture 904 of user 902. The audio sensor may be configured to receive a voice command of user 902.

[0119] The control system 502 of the automated personal assistant 900 can be configured to determine an actuator control command 510 configured to control the system 502. The control system 502 can be configured to determine the actuator control command 510 based on the sensor signal 508 of the sensor 506. The automated personal assistant 900 is configured to send the sensor signal 508 to the control system 502. The classifier 514 of the control system 502 can be configured to execute a gesture recognition algorithm to identify a gesture 904 made by the user 902, determine the actuator control command 510, and send the actuator control command 510 to the actuator 504. The classifier 514 can be configured to retrieve information from non-volatile storage in response to the gesture 904, and output the retrieved information in a form suitable for receipt by the user 902.

[0120] Fig.10A schematic diagram of a control system 502 configured to control a surveillance system 1000 is depicted. The surveillance system 1000 can be configured to physically control access through a door 1002. A sensor 506 can be configured to detect scenes relevant to deciding whether to grant access. The sensor 506 can be an optical sensor configured to generate and transmit image and / or video data. The control system 502 can use such data to detect a person's face.

[0121] The classifier 514 of the control system 502 of the monitoring system 1000 can be configured to interpret the image and / or video data by matching the identities of known persons stored in the non-volatile storage 516 to determine the identity of the person. The classifier 514 can be configured to generate and send actuator control commands 510 in response to the interpretation of the image and / or video data. The control system 502 is configured to send the actuator control commands 510 to the actuator 504. In this embodiment, the actuator 504 can be configured to lock or unlock the door 1002 in response to the actuator control commands 510. In some embodiments, non-physical, logical access control is also possible.

[0122] The monitoring system 1000 may also be a surveillance system. In such an embodiment, the sensor 506 may be an optical sensor configured to detect a scene under surveillance, and the control system 502 is configured to control the display 1004. The classifier 514 is configured to determine a classification of the scene, such as whether the scene detected by the sensor 506 is suspicious. The control system 502 is configured to send an actuator control command 510 to the display 1004 in response to the classification. The display 1004 may be configured to adjust the displayed content in response to the actuator control command 510. For example, the display 1004 may highlight a target that is considered suspicious by the classifier 514. Using embodiments of the disclosed system, the surveillance system can predict the appearance of a target at some time in the future.

[0123] Fig.11 A schematic diagram of a control system 502 configured to control an imaging system 1100 (e.g., an MRI device, an x-ray imaging device, or an ultrasound device) is depicted. The sensor 506 may be, for example, an imaging sensor. The classifier 514 may be configured to determine a classification of all or part of a sensed image. The classifier 514 may be configured to determine or select an actuator control command 510 in response to the classification obtained by the trained neural network. For example, the classifier 514 may interpret an area of ​​the sensed image as a potential anomaly. In this case, the actuator control command 510 may be determined or selected to cause the display 1102 to display the imaging and highlight the potential abnormal area.

[0124] Although exemplary embodiments are described above, these embodiments are not intended to describe all possible forms included in the claims. The words used in the specification are descriptive words, not restrictive words, and it should be understood that various changes can be made without departing from the spirit and scope of the present disclosure. As mentioned above, the features of the various embodiments can be combined to form other embodiments of the present invention that may not be explicitly described or shown. Although various embodiments have been described as providing advantages or preferred over other embodiments or prior art implementations in terms of one or more desired characteristics, it should be recognized by those of ordinary skill in the art that one or more features or characteristics can be compromised to achieve the desired overall system properties, depending on the specific application and implementation. These properties may include, but are not limited to, cost, strength, durability, life cycle cost, marketability, appearance, packaging, size, maintainability, weight, manufacturability, ease of assembly, etc. Therefore, to the extent that any embodiment is described as not as ideal as other embodiments or prior art implementations in terms of one or more features, these embodiments are not outside the scope of the present disclosure and may be ideal for specific applications.

Claims

1. A method for performing open-world object detection, the method comprising: receiving object data, wherein the object data comprises embedding data corresponding to an embedding for a known object in a first input image; projecting the embedding into a hyperbolic embedding space, wherein the hyperbolic embedding space includes embeddings in a plurality of target categories, and wherein each of the plurality of target categories includes one or more target classes; Regularizing the projected embeddings in the hyperbolic embedding space, wherein regularizing the projected embeddings in the hyperbolic embedding space comprises moving each projected embedding (i) closer to an embedding in the same category among the multiple categories and (ii) further away from an embedding in a different category among the multiple categories; receiving an unmatched query corresponding to an object in a second input image; and An output signal is generated based on a hyperbolic embedding space including a regularized embedding, the output signal indicating whether an object in the second input image corresponds to an unknown object in one of the object classes.

2. The method according to claim 1, wherein: The object data includes a set of bounding boxes and class labels for known objects in a first input image.

3. The method according to claim 1, wherein: The first input image corresponds to a training image.

4. The method of claim 1 , further comprising determining a hyperbolic contrastive loss corresponding to the hyperbolic embedding space, wherein regularizing the projected embedding comprises regularizing the projected embedding based on the hyperbolic contrastive loss.

5. The method according to claim 1, wherein: Regularizing the projected embeddings includes determining a respective hyperbolic mean of the embeddings in each of the target classes.

6. The method according to claim 5, wherein: Generating the output signal includes (i) determining a threshold distance based on the hyperbolic mean and (ii) generating the output signal based in part on the threshold distance.

7. The method according to claim 6, wherein: Generating the output signal includes determining whether the unmatched query is less than the threshold distance from at least one of the hyperbolic means.

8. A computing device configured to perform open-world object detection, the computing device comprising a processing device configured to execute instructions stored in a memory to: receiving object data, wherein the object data comprises embedding data corresponding to an embedding for a known object in a first input image; projecting the embedding into a hyperbolic embedding space, wherein the hyperbolic embedding space includes embeddings in a plurality of target categories, and wherein each of the plurality of target categories includes one or more target classes; regularizing the projected embeddings in the hyperbolic embedding space, wherein regularizing the projected embeddings in the hyperbolic embedding space includes moving each projected embedding (i) closer to an embedding in a same category among the plurality of categories and (ii) further away from an embedding in a different category among the plurality of categories; receiving an unmatched query corresponding to an object in a second input image; and An output signal is generated based on a hyperbolic embedding space including a regularized embedding, the output signal indicating whether an object in the second input image corresponds to an unknown object in one of the object classes.

9. The computing device according to claim 8, wherein: The object data includes a set of bounding boxes and class labels for known objects in a first input image.

10. The computing device according to claim 8, wherein: The first input image corresponds to a training image.

11. The computing device of claim 8, wherein the processing device is further configured to execute instructions stored in a memory to generate a hyperbolic contrastive loss corresponding to the hyperbolic embedding space, wherein regularizing the projected embedding comprises regularizing the projected embedding based on the hyperbolic contrastive loss.

12. The computing device according to claim 8, wherein: To regularize the projected embeddings, the processing device is further configured to execute instructions stored in the memory to determine a respective hyperbolic mean of the embeddings in each of the target classes.

13. The computing device of claim 12, wherein: To generate the output signal, the processing device is further configured to execute instructions stored in the memory to (i) determine a threshold distance based on the hyperbolic mean and (ii) generate the output signal based in part on the threshold distance.

14. The computing device of claim 13, wherein: To generate the output signal, the processing device is further configured to execute instructions stored in the memory to determine whether the unmatched query is less than the threshold distance from at least one of the hyperbolic means.

15. A computer-controlled machine comprising: at least one sensor configured to generate a first input image and a second input image; A control system configured to: receiving object data, wherein the object data comprises embedding data corresponding to an embedding for a known object in a first input image; projecting the embedding into a hyperbolic embedding space, wherein the hyperbolic embedding space includes embeddings in a plurality of target categories, and wherein each of the plurality of target categories includes one or more target classes; Regularizing the projected embeddings in the hyperbolic embedding space, wherein regularizing the projected embeddings in the hyperbolic embedding space comprises moving each projected embedding (i) closer to an embedding in the same category among the multiple categories and (ii) further away from an embedding in a different category among the multiple categories; receiving an unmatched query corresponding to an object in a second input image; and generating an output signal based on a hyperbolic embedding space including a regularized embedding, the output signal indicating whether an object in the second input image corresponds to an unknown object in one of the object classes; and An actuator is configured to control operation of the computer-controlled machine in response to the output signal.

16. The computer controlled machine of claim 15, further comprising a memory storing data corresponding to the hyperbolic embedding space.

17. The computer controlled machine of claim 15, wherein: The control system is further configured to generate a hyperbolic contrastive loss corresponding to the hyperbolic embedding space, wherein regularizing the projected embedding includes regularizing the projected embedding based on the hyperbolic contrastive loss.

18. The computer controlled machine of claim 15, wherein: To regularize the embeddings, the control system is further configured to determine a respective hyperbolic mean of the embeddings in each of the target classes.

19. A computer controlled machine according to claim 18, wherein: To generate the output signal, the control system is further configured to (i) determine a threshold distance based on the hyperbolic means and (ii) determine whether the unmatched query is less than the threshold distance from at least one of the hyperbolic means.

20. The computer controlled machine of claim 15, wherein: The computer controlled machines include autonomous robots.

Images