Singular sampling of the dataset

Abstract

In one embodiment of the present invention, a technique for sampling a dataset is described. The technique includes obtaining a plurality of embeddings corresponding to a plurality of data points in the dataset. The technique further includes constructing a tree structure using the plurality of embeddings. The step of constructing the tree structure includes generating a first node that stores a first set of embeddings from the plurality of embeddings and generating a first plurality of nodes as child nodes of the first node. Each node in the first plurality of nodes stores a subset of another embedding from the first set of embeddings. The technique further includes traversing the tree structure to sample a subset of embeddings from the plurality of embeddings and generating a sampled dataset including a subset of data points corresponding to the subset of embeddings.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority to U.S. patent application Ser. No. 18 / 066,024, entitled "Differential Sampling of Datasets," filed Dec. 14, 2022. The subject matter of this related application is incorporated herein by reference. [Technical Field]

[0002] FIELD Embodiments of the present disclosure relate generally to data processing and machine learning, and more particularly to techniques for performing unique sampling of a dataset. [Background technology]

[0003] Advances in the field of machine learning and the increasing availability of computing power have led to a rapid expansion of machine learning applications. However, many machine learning models (e.g., deep neural networks) require large amounts of labeled data for training and validation. Such labeled data typically includes examples tagged with various labels. For example, labeled images tagged with object type (e.g., people, vehicles) and location can be used to train machine learning models to perform object classification and localization.

[0004] Labeled data is obtained by relying on humans to tag data with appropriate labels. However, such manual data labeling is time- and resource-intensive, and few tools existed to date to assist the data labeling process. Furthermore, with advances in data collection and storage technologies, the amount of data that can be labeled is also increasing. However, because resources available for data labeling are typically limited, it has been necessary to reduce the size of the data by filtering or sampling before sending it for labeling.

[0005] However, when filtering or sampling data for labeling using conventional techniques, there is a risk of generating labeled data that does not reflect the diversity or "uniqueness" of the data elements contained in the original dataset. For example, a standard method for selecting data points to be labeled from a large-scale dataset is sometimes to randomly sample data points from the large-scale dataset. While this random sampling method typically produces sampled data points with a distribution similar to that of the original large-scale dataset, it may exclude "unique" data points (corresponding to "outliers") that occur with low frequency in the original large-scale dataset. Therefore, when a machine learning model is trained using these sampled data points and their corresponding labels, the machine learning model cannot learn from these excluded data points, which may result in reduced predictive performance or bias.

[0006] Recently, active learning techniques have been developed to improve the generation of labeled datasets used to train machine learning models. These active learning techniques aim to prioritize labeling data that contributes most to the predictive performance of the machine learning model, maximizing its impact on the model's predictive performance. For example, active learning techniques can train a machine learning model using a small "seed" dataset of labeled data. The predictions generated by the machine learning model based on the seed dataset can then be used to compute multiple heuristics on a larger dataset of unlabeled data points. These heuristics can then be used to select additional data points from the large dataset for labeling. These heuristics can be chosen to optimize criteria such as the diversity of the labeled data, the predictive performance of the machine learning model, and the machine learning model's confidence in the predictions generated from the data points.

[0007] However, active learning techniques aimed at increasing the diversity of labeled data tend to be computationally complex and resource-intensive. For example, active learning methods can attempt to select a subset of labeled data points from a large dataset consisting of unlabeled data points so that the performance of a machine learning model trained using a subset of labeled data points approaches the performance of a machine learning model trained using the entire large-scale dataset. To select such a subset of data points, active learning methods can calculate the distance (bipartite distance) between each latent representation for every possible combination of data points in the large-scale dataset. However, the number of such bipartite distances increases exponentially as the number of data points in the large-scale dataset increases, and therefore, as the number of data points increases and the latent representations of the data points become more complex, it can become practically impossible to perform such bipartite distance calculations. Summary of the Invention [Problem to be solved by the invention]

[0008] As is apparent from the above discussion, there is a need in the art for more effective techniques for generating labeled data. [Means for solving the problem]

[0009] In one embodiment of the present invention, a technique for sampling a dataset is described. The technique includes obtaining a plurality of embeddings corresponding to a plurality of data points in the dataset. The technique further includes constructing a tree structure using the plurality of embeddings. The step of constructing the tree structure includes generating a first node that stores a first set of embeddings from the plurality of embeddings and generating a first plurality of nodes as child nodes of the first node. Each node in the first plurality of nodes stores a subset of another embedding from the first set of embeddings. The technique further includes traversing the tree structure to sample a subset of embeddings from the plurality of embeddings and generating a sampled dataset including a subset of data points corresponding to the subset of embeddings.

[0010] The technical advantages of the technology disclosed herein over the prior art include the ability to suppress duplicate data points and highly similar data points in a subset obtained by sampling a subset of data points from a large dataset, thereby increasing the diversity of the data points in the subset obtained by sampling. Therefore, by training a machine learning model using data points sampled using the technology disclosed herein and their corresponding labels, it is possible to obtain a machine learning model with higher predictive performance and less bias than a machine learning model trained using data points randomly sampled from a large dataset. Another technical advantage of the technology disclosed herein is the ability to efficiently sample a subset of data points by scanning a tree structure constructed based on a large dataset. This reduces the time and resource burden compared to conventional methods that calculate the distance between embeddings for all possible combinations of data points to increase the diversity of data points sampled from a dataset. These technical advantages result in one or more technical improvements over prior art methods. [Brief explanation of the drawings]

[0011] Having briefly summarized the concept of the present invention above, in order to provide a detailed understanding of the features of the various embodiments described above, the concept of the present invention will now be described in more detail with reference to various embodiments, some of which are also illustrated in the accompanying drawings. It should be noted that the accompanying drawings merely illustrate exemplary embodiments of the concept of the present invention, and therefore should not be construed as limiting the scope of the present disclosure in any way, and that there may be other embodiments that have equivalent effects. [Figure 1] 1 is a conceptual diagram of a system configured to implement one or more embodiments. [Figure 2] FIG. 1 is a flow diagram illustrating steps of a method for processing a data labeling request, according to various embodiments. [Figure 3] FIG. 2 illustrates a more detailed view of the server shown in FIG. 1, according to various embodiments. [Figure 4] FIG. 4 illustrates a more detailed configuration of the processing engine and sampling engine shown in FIG. 3, according to various embodiments. [Figure 5A] FIG. 5 illustrates an exemplary tree structure generated by the processing engine shown in FIG. 4, according to various embodiments. [Figure 5B] FIG. 5B illustrates a step in traversing the exemplary tree structure shown in FIG. 5A, according to various embodiments. [Figure 5C] FIG. 5B illustrates a step in traversing the exemplary tree structure shown in FIG. 5A, according to various embodiments. [Figure 5D] FIG. 5B illustrates a step in traversing the exemplary tree structure shown in FIG. 5A, according to various embodiments. [Figure 6] 1 is a flow diagram illustrating steps of a method for sampling from a dataset, according to various embodiments. [Figure 7] FIG. 1 is a flow diagram illustrating steps in a method for sampling images from a dataset based on embeddings representing objects depicted in the images, according to various embodiments. [Figure 8]FIG. 1 is a flow diagram illustrating steps in a method for sampling images from a dataset based on embeddings representing objects depicted in the images, according to various other embodiments. [Figure 9] FIG. 1 is a flow diagram illustrating steps in a method for sampling images from a dataset based on embeddings representing objects depicted in the images, according to various other embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0012] In the following description, numerous specific details are set forth to provide a more thorough understanding of various embodiments. However, it will be apparent to one skilled in the art that the concepts of the present invention may be practiced even if one or more of these specific details are omitted.

[0013] System Overview 1 illustrates a system 100 configured to implement one or more aspects of the present disclosure. As illustrated, the system 100 includes, but is not limited to, a server 130 and multiple client devices 1021-1022. N Each of the client devices 1021 to 102 N The server 130 and the client devices 1021 to 1022 may be individually referred to as "client devices 102." N work together and, upon request from the customer, client devices 1021-102 N The server 130, the client devices 1021 to 1022, and the like enable data labeling by the user. N, and customer devices 110 communicate via network 150. Network 150 may be a suitable network, such as a wide area network (WAN) such as the Internet or a local area network (LAN). For ease of explanation, FIG. 1 shows a single server 130 and a single customer device 110, but system 100 may generally include any number of servers, customer devices, and client devices. These may be physical computing systems or virtual computing systems running, for example, in a data center or in the cloud.

[0014] As shown, server 130 exposes a task application programming interface (task API) 132, which allows customers to submit data and data labeling requests using API calls. Any suitable data and labeling requests can be submitted to server 130 via such API calls. For example, in the case of an autonomous vehicle, photo data acquired by onboard sensors, LIDAR (light detection and ranging) data, radar (radio detection and ranging) data, or any combination thereof, can be uploaded from customer device 110 to server 130, and a request can be made to tag specific types of objects (e.g., vehicles, bicycles, pedestrians, etc.) within the data. GPS (global positioning system) data can also be uploaded and included as part of the LIDAR data.

[0015] In some embodiments, the server application 134 running on the server 130 requires that data and data labeling requests submitted via API calls meet predefined restrictions. For example, restrictions may be placed on the classes of objects that can be labeled (e.g., vehicles, pedestrians, buildings, etc.), the type and size of the data, etc.

[0016] The server application 134 processes the data received via the task API 132 and transmits the processed data to the client devices 1021-1022 based on the customer's request. N Data labeling applications running on 1041-104 N At this time, the processed data is transmitted to the client devices 1021 to 1022. N The server application 134 may perform any suitable processing on the received data. For example, in some embodiments, the server application 134 may process the received photo data, LIDAR data, and radar data in various formats into the data labeling applications 1041-1044. N The server application 134 may convert the received data into a single format that can be read and processed by the server application 134. For example, the server application 134 may also compress the received data to reduce its size. For purposes of illustration, the server application 134 is illustrated as a single application, but it should be understood that in alternative embodiments, the functions of the server application 134 may be performed by multiple applications or other types of software.

[0017] Data Labeling Applications 1041-104 N(each individually referred to hereinafter as the “data labeling application 104”) processes and renders data received from the server application 134 and displays it on a user interface (UI). In some embodiments, the data labeling application 104 visualizes three-dimensional (3D) data (e.g., LIDAR data, radar data, or both) by rendering one or more colored point clouds, allowing a user to navigate and view the point clouds from various perspectives. For example, in some embodiments, the data labeling application 104 can downsample to obtain an aggregated point cloud that includes only highly informative points. For example, based on a user instruction, the data labeling application 104 can blend the colors of point clouds from different data sources (e.g., photo data, label data, LIDAR intensity data, etc.). Furthermore, the data labeling application 104 can not only display the rendered point cloud on the UI, but also simultaneously display the photos associated with the point cloud.

[0018] In some embodiments, the data labeling application 104 provides tools to assist the user in performing the data labeling task. For example, tools may be provided that allow the user to use a mouse and keyboard to draw annotations in the form of cuboids, bounding boxes, etc., or to label points that belong to specific objects. Further tools may be provided to assist the user in performing the data labeling task, such as tools to automatically adjust the position and orientation of a user-specified cuboid, or tools to propagate a user-specified cuboid from a keyframe to other frames.

[0019] Figure 2 is a flow diagram illustrating method steps for processing data labeling requests, according to various embodiments. Note that although the method steps are described with reference to the system shown in Figure 1, one skilled in the art will understand that in other embodiments, any system can be configured to perform the method steps in any order.

[0020] As shown, method 200 begins at step 202, where server application 134 receives data and a corresponding data labeling request via an API call. The data can be in any suitable format acceptable to server application 134. For example, server application 134 may require data to be submitted in one or more JSON (JavaScript Object Notation) files. Similarly, data labeling requests may need to meet certain restrictions regarding the classes of objects that can be labeled (e.g., vehicles, pedestrians, buildings, etc.).

[0021] The server application 134 then processes the received data in step 204. The server application 134 may perform any suitable processing on the received data. In some embodiments, the processing may include compressing the received data, converting it to a format readable by a data labeling application, or both. For example, the received data may be converted into a data format that represents each point in the 3D point cloud as an (x, y, z) coordinate in a list format with a corresponding timestamp.

[0022] Next, in step 206, the server application 134 sends the processed data and instructions indicating one or more data labeling tasks to one or more data labeling applications based on the received request. Although one data labeling application 104 is shown for ease of explanation, it should be understood that the server application 134 can send the processed data and instructions indicating data labeling tasks over the network to any number of data labeling applications running on different client devices.

[0023] In step 208, the data labeling application 104 generates and displays a rendering of the received data. For example, the data labeling application 104 may display the rendered point cloud on a UI, allowing the user to navigate and view the point cloud from various viewpoints. Additionally or alternatively, the UI may display text and images, and the data labeling application 104 may provide tools to assist in labeling the rendered point cloud, text, images, and other data.

[0024] Next, in step 210, the data labeling application 104 receives labeling operations to be performed on the data contained in the rendering results. For example, a user can scan a two-dimensional (2D) or 3D scene to identify an object of interest, specify the object's location using a mouse, and use the mouse and keyboard to draw a box or rectangle that precisely surrounds the object. In this case, the user can also move forward and backward in time to see where the object moves over time and label the object in all frames corresponding to each point in time. Furthermore, as described above, the data labeling application 104 provides tools to assist the user in labeling in addition to tools that enable such labeling operations. For example, such assistive tools include a tool that automatically adjusts the position and orientation of a rectangle specified by the user, and a tool that propagates a rectangle from a frame specified as a keyframe to other frames.

[0025] In step 212, the data labeling application 104 returns the labeled data to the server application 134. The labeled data may be sent over a network, such as the Internet, to the server application 134, which may then return the labeled data to the customer. In some embodiments, the labeled data may be processed, such as through optional validation processes, before being returned to the customer.

[0026] 3 illustrates a more detailed configuration of server 130 shown in FIG. 1, according to various embodiments. Note that the server 130 described herein is merely exemplary, and other technically feasible configurations are within the scope of the present invention. For example, the hardware and software components of server 130 may be implemented on client device 102 or customer device 110. Alternatively, for example, the functionality of server 130 may be implemented by running multiple instances of server 130 on nodes within a data center, cluster, or cloud computing environment.

[0027] As shown, server 130 includes, but is not limited to, a central processing unit (CPU) 302, a system memory 304, and a display processor 312, with system memory 304 coupled to display processor 312 via memory bridge 305 and communication path 313. Memory bridge 305 is further coupled to I / O (input / output) bridge 307 via communication path 306, which is further coupled to switch 316.

[0028] In operation, I / O bridge 307 is configured to receive user input information from input devices 308, such as a keyboard or mouse, and forward the input information to CPU 302 for processing via communication path 306 and memory bridge 305. Switch 316 is also configured to enable connections between I / O bridge 307 and other components of server 130, such as network adapter 318 and various add-in cards 320, 321, etc.

[0029] I / O bridge 307 is coupled to system disk 314. System disk 314 can be configured to store content, applications, and data used by CPU 302 and display processor 312. Generally, system disk 314 provides non-volatile storage for applications and data and can be comprised of a fixed or removable hard disk drive, a flash memory device, or other magnetic, optical, or solid-state storage device, such as a CD-ROM (compact disc read-only memory), a DVD-ROM (digital versatile disc read-only memory), a Blu-ray® disc, or an HD-DVD (high definition DVD). Additionally, although not explicitly shown, other components, such as port connections such as a universal serial bus, a compact disc drive, a digital versatile disc drive, or a film recording device, can also be connected to I / O bridge 307.

[0030] Other port connections, such as USB, and other components (not shown), such as a film recording device, may also be connected to I / O bridge 307. For example, an audio processor may be used to generate analog or digital audio output based on instructions and data provided by CPU 302, system memory 304, or system disk 314. The communication paths interconnecting the various components shown in FIG. 3 may be implemented using any suitable protocol. As known in the art, such protocols may include any bus or point-to-point communication protocol, such as Peripheral Component Interconnect (PCI), PCI Express (PCI-E), Accelerated Graphics Port (AGP), HyperTransport, etc., and the protocols used between connected devices may be different.

[0031] In various embodiments, memory bridge 305 may be a northbridge chip and I / O bridge 307 may be a southbridge chip. Furthermore, communication paths within server 130, such as communication paths 306 and 313, may be implemented using any technologically appropriate protocol, including, for example, any bus protocol or point-to-point communication protocol, such as AGP (Accelerated Graphics Port), HyperTransport, etc., as known in the art.

[0032] Display processor 312 is coupled to memory bridge 305 via a communication path such as a bus (e.g., a PCI Express link, an Accelerated Graphics Port link, or a HyperTransport link). In one embodiment, display processor 312 is a graphics subsystem including at least one graphics processing unit (GPU) and graphics memory. The graphics memory includes display memory (e.g., a frame buffer) for storing pixel data for each pixel of an output image. The graphics memory may be integrated into the same device as the GPU, connected as a separate device from the GPU, or implemented within system memory 304.

[0033] Display processor 312 delivers pixels at regular intervals to display device 310 (e.g., a screen or conventional CRT, plasma, OLED, SED, or LCD-based monitor or television). Display processor 312 may also output pixels to a film recorder adapted to reproduce the computer-generated images on photographic film. Display processor 312 may provide analog or digital signals to display device 310. In various embodiments, one or more of the various graphical user interfaces illustrated in the drawings accompanying this specification are displayed to one or more users via display device 310, and the one or more users may input data into these various graphical user interfaces and receive visual output.

[0034] Also connected to I / O bridge 307 is system disk 314, which may be configured to store content, applications, and data used by CPU 302 and display processor 312. Generally, system disk 314 provides non-volatile storage for applications and data and may be comprised of a fixed or removable hard disk drive, a flash memory device, CD-ROM, DVD-ROM, "Blu-ray," HD-DVD, or other magnetic, optical, or solid-state storage device.

[0035] Switch 316 is configured to enable connections to other components such as I / O bridge 307, such as network adapter 318 and various add-in cards 320, 321. Network adapter 318 enables server 130 to communicate with other systems over electronic communications networks, which may include wired or wireless communications over a local area network or a wide area network such as the Internet.

[0036] Other port connections, such as USB, and other components (not shown), such as a film recording device, may also be connected to I / O bridge 307. For example, an audio processor may be used to generate analog or digital audio output based on instructions and data provided by CPU 302, system memory 304, or system disk 314. The communication paths interconnecting the various components shown in FIG. 3 may be implemented using any suitable protocol. As known in the art, such protocols may include any bus or point-to-point communication protocol, such as PCI (Peripheral Component Interconnect), PCI Express (PCI-E), AGP (Accelerated Graphics Port), HyperTransport, etc., and the protocols used between connected devices may be different.

[0037] In one embodiment, display processor 312 incorporates circuitry optimized for graphics and video processing, including, for example, video output circuitry, constituting a graphics processing unit (GPU). In another embodiment, display processor 312 incorporates circuitry optimized for general-purpose processing. In yet another embodiment, display processor 312 may be integrated with one or more other system elements, such as memory bridge 305, CPU 302, and I / O bridge 307, to form a system on chip (SoC). In yet another embodiment, display processor 312 may be omitted, and the functionality of display processor 312 may be realized by software executed by CPU 302.

[0038] Pixel data may be provided directly from CPU 302 to display processor 312. In some embodiments of the invention, instructions, data, or both representing a scene are provided to a render farm (a group of server computers, each similar to server 130) via network adapter 318 or system disk 314. The render farm then uses the provided instructions, data, or both to generate one or more rendered images of the scene. These rendered images may be stored digitally on a computer-readable medium and, optionally, sent back to server 130 for display. Similarly, stereo image pairs processed by display processor 312 may be output to another system for display, stored on system disk 314, or stored digitally on a computer-readable medium.

[0039] Alternatively, CPU 302 may provide instructions, data, or both, defining desired output images to display processor 312, which may then generate pixel data for one or more output images based on the instructions and data. This may include determining and adjusting for disparity (offset) between stereo image pairs. The instructions and data defining the desired output images may be stored in system memory 304 or in graphics memory within display processor 312. In one embodiment, display processor 312 provides 3D rendering capabilities that enable it to generate pixel data for output images based on instructions and data defining a scene's geometry, lighting, texture, motion, camera parameters, and the like. Display processor 312 may also include one or more programmable execution units capable of executing shader programs, tone mapping programs, and the like.

[0040] Additionally, in other embodiments, CPU 302 or display processor 312 may be replaced or supplemented by any technically feasible type of processing device configured to process data and execute program code, including, for example, a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc. In various embodiments, any of the operations or functions described herein may be performed by CPU 302, display processor 312, one or more other processing devices, or any combination of these types of processing devices.

[0041] Based on the provided data and instructions, the CPU 302, the render farm, the display processor 312, or any combination thereof, may use any surface or volume rendering technique known in the art, including rasterization, scanline rendering, REYES (micropolygon rendering), ray casting, ray tracing, neural rendering, image-based rendering techniques, or any combination of these or other rendering or image processing techniques known in the art, to create one or more rendered images.

[0042] Alternatively, other embodiments are contemplated in which server 130 comprises a robot or robotic device, and further comprises CPU 302 and / or other processing unit or device, and system memory 304. In such embodiments, server 130 may include or omit other elements shown in FIG. 3. System memory 304 and / or other memory units or devices included with server 130 may also store instructions that, when executed, cause the robot or robotic device represented by server 130 to perform one or more actions, steps, tasks, etc.

[0043] It should be understood that the systems illustrated herein are illustrative and that variations and modifications are possible. Connection topologies, such as the number and arrangement of bridges, the number of CPUs, and the number of parallel processing subsystems, can be modified as needed. For example, in some embodiments, system memory 304 can be directly connected to CPU 302 without using memory bridge 305, with other devices communicating with system memory 304 via memory bridge 305 and CPU 302. Another alternative topology involves connecting display processor 312 to I / O bridge 307 or directly to CPU 302 rather than memory bridge 305. In still other embodiments, I / O bridge 307 and memory bridge 305 can be integrated into a single chip rather than being one or more separate devices. Additionally, certain embodiments may omit one or more components shown in FIG. 3 . For example, switch 316 may be omitted, with network adapter 318 and add-in cards 320 and 321 connected directly to I / O bridge 307.

[0044] In one or more embodiments, server 130 is configured to execute a processing engine 322 and a sampling engine 324 that reside in system memory 304. Processing engine 322 and sampling engine 324 may be stored on a storage device, such as system disk 314, and loaded into system memory 304 at runtime.

[0045] In some embodiments, the processing engine 322 and the sampling engine 324 operate to perform singular sampling on the dataset. In singular sampling, a subset of data points is sampled from the dataset to maximize the variety, or "singularity," of the sampled data points. More specifically, the processing engine 322 constructs a tree structure using embeddings corresponding to the data points in the dataset. In this tree structure, a parent node stores an embedding corresponding to a set of data points. This set of data points is defined by a region, such as a hypersphere, that the embedding occupies in a multidimensional vector space. Meanwhile, two or more child nodes derived from the parent node store subsets of the embedding set of the parent node. These subsets are disjoint. Similarly to the parent node, the subset of data points stored in each child node is also defined by a region, such as a hypersphere, in the multidimensional vector space. The tree structure is constructed so that the size of the area represented by each node is minimized, and the area defining the data points contained in a child node is smaller than the area defining the data points contained in its parent node.

[0046] The sampling engine 324 traverses the tree structure to sample a subset of data points from the dataset. Starting from the root node and progressing downward, the sampling engine 324 traverses the tree structure, sampling each node in the tree structure to sample an embedding representative of that node. After sampling an embedding from a node, the sampling engine 324 marks that node and all other nodes containing that embedding (e.g., each node along the path from the node to a leaf node that represents a region where that embedding may exist) as sampled. The sampling engine 324 repeats this process until certain conditions are met, such as sampling all nodes, sampling a predetermined number of embeddings, the maximum possible distance between an embedding in the subset and the remaining embeddings in the dataset being below a threshold, or other criteria being met. The sampling engine 324 then passes the subset of data points represented by the sampled embeddings to a labeling or other process.

[0047] Singular sampling of the dataset FIG. 4 illustrates a more detailed configuration of the processing engine 322 and sampling engine 324 shown in FIG. 3 , according to various embodiments. As described above, the processing engine 322 and sampling engine 324 are capable of generating a sampled dataset 428 consisting of data points 420(1) through 420(Y) (each of which may be individually referred to as a “data point 420”) from a large-scale dataset 402 consisting of data points 404(1) through 404(X) (each of which may be individually referred to as a “data point 404”). For example, when a dataset 402 consisting of 100,000 data points 404 is provided, the processing engine 322 and sampling engine 324 can generate a sampled dataset 428 consisting of a plurality of data points 420. The data points 420 correspond to a specific percentage (e.g., 5%, 10%, etc.) or a specific number (e.g., 4096, 5000, etc.) of the data points 404 included in the dataset 402.

[0048] The data points 404 included in the dataset 402 may include various types and formats of data. For example, the data points 404 may include images, text, audio, video, point clouds, meshes, time series data, other types of data in high-dimensional space, or any combination thereof. Additionally, one or more data points 404 may be generated or defined as a subset of one or more other data points 404. For example, the dataset 402 may include a first set of data points 404 corresponding to a set of images. The dataset 402 may also include a second set of data points 404 corresponding to objects depicted in those images.

[0049] In one or more embodiments, dataset 402 includes data points 404 that can be processed, analyzed, or used in one or more applications. A sampled dataset 428 generated from dataset 402 is then configured to assist the application in using the data from dataset 402 to achieve its objectives, satisfy its constraints, or otherwise support its use. For example, dataset 402 may include images and objects within images used to train machine learning models that perform computer vision tasks. Such computer vision tasks include object recognition, object detection, semantic segmentation, object tracking, event detection, pose estimation, motion estimation, and other types of computer vision tasks. Because many of these computer vision tasks require labeled data and resources available for data labeling are typically limited, processing engine 322 and sampling engine 324 may generate sampled dataset 428 to reduce the number of data points sent for labeling.

[0050] More specifically, the processing engine 322 generates or identifies embeddings 408(1) through 408(X) (each of which may be individually referred to as an "embedding 408") corresponding to data points 404 in the dataset 402. For example, the processing engine 322 may convert each data point 404 in the dataset 402 into a corresponding embedding 408 using one or more of various types of neural networks or machine learning models, such as a convolutional neural network, a fully connected neural network, a recurrent neural network, a residual neural network, a transformer neural network, an autoencoder, a variational autoencoder, or a generative adversarial network. As another example, the processing engine 322 may obtain the embeddings 408 corresponding to the data points 404, the mapping between the embeddings 408 and the corresponding data points 404, or both, from a data store of some type, such as a database or a key-value store.

[0051] The processing engine 322 organizes the embeddings 408 into a tree structure 406. The tree structure 406 includes hierarchically organized nodes 414(1) through 414(3), each of which may be individually referred to as a "node 414." In one or more embodiments, the tree structure 406 may be a ball tree, a KD tree, or any other tree structure for spatially organizing and partitioning the embeddings 408 into disjoint subsets. Each node 414 in the tree structure 406 represents a region or partition, such as a hypersphere or hyperplane, in the latent space associated with the embedding 408. Each node 414 contains a subset of the embeddings 408 that reside within that region. Each non-leaf node 414 is further associated with two or more child nodes, which spatially partition the set of embeddings 408 contained within the non-leaf node 414 into two or more disjoint subsets of embeddings 408. Each child node represents a smaller area than the area represented by the corresponding non-leaf parent node 414 .

[0052] For example, a root node 414 of a ball tree corresponding to the tree structure 406 may include embeddings 408 corresponding to all data points 404 in the dataset 402, and the root node 414 represents a hypersphere encompassing all these embeddings 408. The root node 414 may have two child nodes 414, and these two child nodes 414 may include two disjoint subsets of embeddings 408. The union of these two disjoint subsets of embeddings 408 corresponds to the universal set of embeddings 408 generated from all data points 404 in the dataset 402. Each child node 414 also represents a hypersphere that defines the corresponding set of embeddings 408, and the hypersphere is defined by a centroid and a radius. Additional layers of child nodes may be included in the ball tree. These additional layers allow the set of embeddings 408 and their corresponding hyperspheres associated with a higher-level parent node 414 to be further subdivided into smaller, non-intersecting subsets of embeddings 408 and their corresponding hyperspheres, and each embedding 408 is stored in a node 414 on the path from the root node 414 to a leaf node 414 in the ball tree.

[0053] As shown in FIG. 4 , processing engine 322 generates tree structure 406 based on distances 410(1)-410(Z) (sometimes individually referred to as “distances 410”) associated with embeddings 408. In one or more embodiments, distances 410 may be calculated as Mahalanobis distances between embeddings 408, Mahalanobis distances between each embedding 408 and one or more distributions of embeddings 408, or both. For example, processing engine 322 may apply a whitening transform to the matrix of embeddings 408. Using this whitened matrix, processing engine 322 may then calculate the Mahalanobis distances between embeddings 408 or between a given embedding 408 and some or all of the distributions of embeddings 408 as efficiently computable Euclidean distances 410. Furthermore, the processing engine 322 can use the Euclidean distance 410 thus determined to divide the set of embeddings 408 belonging to a given parent node 414 in the tree structure 406 into two or more subsets of embeddings 408 belonging to two or more child nodes 414. The processing engine 322 can also generate a ball tree corresponding to the tree structure 406 by performing a process to identify the radius of the hypersphere represented by each node 414, a process to identify one or more embeddings 408 that are closest to or farthest from the center of gravity of each hypersphere, and the like.

[0054] In one or more embodiments, the way in which the tree structure 406 is generated, the placement and / or configuration of the nodes 414 within the resulting tree structure 406, are controlled using a set of construction criteria 412. The construction criteria 412 include various parameters that affect how the embeddings 408 are organized within the tree structure 406, such as, but not limited to, the depth of the tree structure 406, the number of nodes 414 within the tree structure 406, the number of embeddings 408 in each leaf node 414 of the tree structure 406, the type of distance 410 (e.g., Euclidean distance, Mahalanobis distance, cosine distance, etc.) used to construct the tree structure 406, how the embeddings 408 are divided and assigned to which nodes 414 in the tree structure 406, and the type of tree structure 406.

[0055] In one or more embodiments, the construction criteria 412 specify memory constraints for generating the tree structure 406, various parameters for generating the tree structure 406 under those memory constraints, or both. For example, the construction criteria 412 may specify memory limits associated with one or more instances of the processing engine 322, a maximum number of embeddings 408 that can be stored in memory in a given instance of the processing engine 322, or both. In this case, the processing engine 322 may divide the set of embeddings 408 corresponding to all data points 404 in the dataset 402 into multiple subsets of embeddings 408, each subset of embeddings 408 not exceeding the memory limit. Furthermore, the processing engine 322 may generate independent "subtrees" (or other types of tree structures) based on each subset of embeddings 408, and all of these subtrees may be grouped under a single root node 414 of the tree structure 406.

[0056] Once the tree structure 406 is created, the sampling engine 324 generates a sampled data set 428 by performing a traversal 416 of the tree structure 406. During the traversal 416, the sampling engine 324 generates a set of sampled embeddings 418 by sequentially selecting embeddings 408 from each node 414 in the tree structure 406, starting from the root node 414 and progressing down to the nodes 414 below. To determine the order in which to visit the nodes 414 when generating the sampled embeddings 418, during the traversal 416, the sampling engine 324 keeps track of the set of sampled nodes 424, the set of nodes 426 with some sampled child nodes, and the set of nodes 430 with all sampled child nodes.

[0057] A sampled node 424 corresponds to a node 414 that contains one or more sampled embeddings 418. A node 426 with some sampled child nodes corresponds to a sampled node 424 that has at least one sampled child node and at least one unsampled child node. A node 430 with all sampled child nodes corresponds to a sampled node 424 with all its child nodes sampled.

[0058] At each step of the traversal 416, the sampling engine 324 identifies a node 414 to sample and selects a specific embedding 408 within that node 414 to add to the set of sampled embeddings 418. For example, the sampling engine 324 may randomly select an embedding 408 from the node 414, select the embedding 408 closest to the centroid of the hypersphere or region represented by the node 414, or select another specific embedding 408 from the node 414. The sampling engine 324 then adds all nodes 414 containing the selected embedding 408 to the set of sampled nodes 424. This includes all nodes 414 on the path from the identified node 414 to the leaf nodes 414 in the tree structure 406. The sampling engine 324 also updates the set of nodes 426 with some sampled child nodes and the set of nodes 430 with all sampled child nodes based on the updated set of sampled nodes 424. For example, if an update to the set of sampled nodes 424 results in a new, unsampled child node 414 being added to a given node 414, the sampling engine 324 can move the given node 414 from the set of nodes 426, some of whose children have been sampled, to the set of nodes 430, all of whose children have been sampled.

[0059] In one or more embodiments, the sampling engine 324 performs the traversal 416 by retrieving sampled embeddings 418 by traversing the nodes 414 in an order that decreases the maximum distance 434 between any embedding 408 in the set of sampled embeddings 418 and any embedding 408 not in the set of sampled embeddings 418. This maximum distance 434 corresponds to the diameter of the largest hypersphere represented by a node in the set of nodes 426 whose child nodes have been sampled. To decrease this maximum distance 434, after sampling the root node 414 in the first step of the traversal 416, the sampling engine 324 selects the child node 414 of a given node 414 that represents the largest hypersphere in the set of nodes 426 whose child nodes have been sampled as the node 414 to sample in the next step of the traversal 416. Once all of the child nodes 414 for a given node 414 have been sampled, the sampling engine 324 moves the given node 414 to the set of all-child-sampled nodes 430. This surfaces a new node 414 representing the next-largest hypersphere in the set of some-child-sampled nodes 426. Once the set of some-child-sampled nodes 426 is empty (e.g., after all of its nodes 414 have been moved to the set of all-child-sampled nodes 430), the maximum distance 434 is set to the diameter of the largest hypersphere represented by a leaf node in the set of sampled nodes 424.

[0060] FIG. 5A illustrates an exemplary tree structure 406 generated by the processing engine 322 shown in FIG. 4 , according to various embodiments. As shown in FIG. 5A , the exemplary tree structure 406 includes five nodes, nodes 414(1) through 414(5). The root node 414(1) of the exemplary tree structure 406 represents a hypersphere with a radius of 100 that encompasses the set of embeddings 408 representing all data points 404 in a given dataset 402. Nodes 414(2) and 414(3) are child nodes of the root node 414(1) and represent hyperspheres with radii of 80 and 30, respectively. Nodes 414(2) and 414(3) also divide the set of embeddings 408 associated with the root node 414(1) into two non-intersecting subsets of embeddings 408 defined by their corresponding hyperspheres.

[0061] Nodes 414(4) and 414(5) are children of root node 414(2) and represent hyperspheres of radius = 60 and radius = 50, respectively. Nodes 414(4) and 414(5) also partition the set of embeddings 408 associated with node 414(2) into two disjoint subsets of embeddings 408 defined by their corresponding hyperspheres.

[0062] FIG. 5B illustrates one step in traversing the exemplary tree structure 406 shown in FIG. 5A, according to various embodiments. More specifically, FIG. 5B illustrates the result of a first traversal step, resulting in a first embedding being sampled from root node 414(1). Path 502 includes root node 414(1), node 414(2), and node 414(4), indicating that the first embedding also exists at these nodes. As a result, nodes 414(1), 414(2), and 414(4) are added to the set of sampled nodes 424. Because nodes 414(1) and 414(2) each contain one unsampled child node, 414(3), and one unsampled child node, 414(5), both nodes 414(1) and 414(2) are added to the set of nodes 426 whose children have been sampled.

[0063] Additionally, maximum distance 434 is initialized to a value that reflects the state in which the first embedding was sampled from node 414(1). Specifically, maximum distance 434 is set to 200 because any embedding not included in the set of sampled embeddings 418 is within twice the radius of the hypersphere represented by node 414(1) of the first embedding.

[0064] Figure 5C illustrates one step in traversing the exemplary tree structure 406 shown in Figure 5A, according to various embodiments. More specifically, Figure 5C illustrates the second traversal step resulting in a second embedding sampled from node 414(3).

[0065] Node 414(3) is selected for sampling in the second traversal step because node 414(1) represents the largest hypersphere in the set of nodes 426 with some child nodes already sampled, and node 414(3) is an unsampled child of node 414(1). Node 414(3) can be identified by following path 504 from node 414(1) to node 414(3). Once the second embedding is sampled from node 414(3), node 414(3) is marked as sampled, and node 414(1) is moved from the set of nodes 426 with some child nodes already sampled to the set of nodes 430 with all child nodes already sampled. Additionally, maximum distance 434 is reduced to 160. This is because node 414(2) is the node corresponding to the largest hypersphere in the set of nodes 426 that have some child nodes that have been sampled, and any embedding that is not in the set of sampled embeddings 418 is within twice the radius of the hypersphere represented by node 414(2) of any node in the set of sampled embeddings 418.

[0066] Figure 5D illustrates a step in traversing the exemplary tree structure 406 shown in Figure 5A, according to various embodiments. Specifically, Figure 5D illustrates the third traversal step resulting in a third embedding sampled from node 414(5).

[0067] Node 414(5) is selected for sampling in the third traversal step because node 414(2) represents the largest hypersphere in the set of nodes 426, some of whose children have been sampled, and node 414(5) is an unsampled child of node 414(2). Node 414(5) can be identified by tracing path 506 from node 414(2) to node 414(5). Once the third embedding is sampled from node 414(5), node 414(5) is marked as sampled, and node 414(2) is moved from the set of nodes 426, some of whose children have been sampled, to the set of nodes 430, all of whose children have been sampled. Maximum distance 434 is also updated to 120, or twice the radius of the hypersphere of leaf node 414(4), the largest sampled leaf node. This is because the set of node 426, some of whose children have already been sampled, is now empty.

[0068] 4, the sampling engine 324 continues to perform the traversal 416, adding new embeddings 408 to the collection of sampled embeddings 418, until one or more sampling criteria 432 are met. The sampling criteria 432 may include various conditions that affect how the traversal 416 is performed and whether the traversal 416 is used to generate the sampled embeddings 418, such as, but not limited to, a pre-specified number of data points 420 to include in the sampled data set 428, one or more thresholds for maximum distance 434, sampling from all nodes 414 in the tree structure 406, or sampling from a predetermined number or percentage of nodes 414 in the tree structure 406. Additionally, if the sampling criteria 432 indicate that more sampled embeddings 418 need to be generated after sampling all nodes 414 in the tree structure 406 has been completed (e.g., if the pre-specified number of data points 420 to be included in the sampled data set 428 exceeds the number of leaf nodes 414 in the tree structure 406), the sampling engine 324 continues to obtain embeddings 408 from leaf nodes 414 or other nodes 414 in the tree structure 406 and add them to the set of sampled embeddings 418 until the sampling criteria 432 are met.

[0069] As mentioned above, to address memory limitations associated with generating the tree structure 406, the tree structure 406 can include more than two subtrees, each generated from a different subset of embeddings 408. Construction of the tree structure 406 is then completed by grouping these subtrees under a single root node 414. When traversing 416 this type of tree structure 406, the sampling engine 324 first samples the embeddings from the top node 414 of each subtree. Once the sampling engine 324 has obtained sampled embeddings 418 from the top nodes 414 of all subtrees, the root node 414 of the tree structure 406 and the top nodes 414 of all subtrees are placed in a set of sampled nodes 424. Furthermore, the root node 414 of the tree structure 406 is moved from the set of nodes 426 with some sampled children to the set of nodes 430 with all sampled children. The top nodes 414 of all subtrees are also placed in the set of nodes 426 with some sampled children. The sampling engine 324 then performs a traversal 416 of pairs of nodes 414 that are descendants of the top node 414 of the subtree, similar to traversing a binary tree, to obtain additional sampled embeddings 418. Such traversal is performed based on the set of sampled nodes 424, the set of nodes 426 with some sampled children, the set of nodes 430 with all sampled children, the maximum distance 434 value associated with the sampled embeddings 418, or any combination thereof.

[0070] If the sampling criteria 432 are met, the sampling engine 324 converts the sampled embedding 418 into data points 420 in a sampled dataset 428. For example, the sampling engine 324 can use the mapping between the embedding 408 and the data points 404 to identify the set of data points 420 that the sampled embedding 418 represents. The sampling engine 324 can also generate a sampled dataset 428 that organizes these data points 420 in a list format.

[0071] As described above, the data points 404 may include a first type of data point (e.g., an image) under which one or more instances of a second type of data point (e.g., an object depicted in the image) may be grouped. In this type of scenario, the number of instances of the first type of data point included in the sampled dataset 428 may be set to maximize the diversity of the second type of data point. For example, consider training a machine learning model that recognizes or detects objects in images. A training dataset for such a machine learning model is generated by selecting a predetermined number or percentage of images from the large image dataset 402 to include in the sampled dataset 428. In this case, the images included in the sampled dataset 428 are selected to provide a high diversity of objects used to train the machine learning model.

[0072] To generate a sampled data set 428 that optimizes the diversity of the second type of data points 404 while maintaining a predetermined number of first type data points 420 (e.g., as specified by sampling criteria 432), the processing engine 322 can construct a tree structure 406 using embeddings 408 of the second type of data points. The sampling engine 324 can then select the first type of data points 420 based on the second type of data points 404 represented by the sampled embeddings 418 to construct the sampled data set 428. For example, the processing engine 322 can use an object detection model to generate bounding boxes for objects included in images in the data set 402 and to obtain bounding boxes or crops of the objects from the data set 402. The processing engine 322 can then use an image embedding model, such as a convolutional neural network, to convert the set of image regions defined by these bounding boxes or crops into a set of object embeddings 408. The processing engine 322 can then construct a tree structure 406 using the distances 410 associated with the object embeddings 408. A sampling engine 324 can then perform a traversal 416 on this tree structure 406 to obtain sampled embeddings 418 that represent a high-diversity set of objects depicted in the images. The sampling engine 324 can then identify sets of images in which these sets of objects appear and include the images in a sampled dataset 428.

[0073] Note that because an image may depict multiple objects, the number of objects represented by the sampled embeddings 418 may exceed the number of images they depict. Thus, even if a predetermined number of sampled embeddings 418 are obtained from the tree structure 406, the sampled dataset 428 may contain fewer images than the predetermined number.

[0074] While retrieving sampled embeddings 418 representing objects contained in images from the tree structure 406, the sampling engine 324 may maintain a list of images depicting the objects represented by the sampled embeddings 418 during the scan 416, so as to ensure that the sampled dataset 428 contains a predetermined number of images (e.g., the number specified by the sampling criteria 432). The sampling engine 324 may continue to add new embeddings to the collection of sampled embeddings 418 until the number of images in the list meets the sampling criteria 432 for the sampled dataset 428.

[0075] Additionally or alternatively, the sampling engine 324 determines whether a particular embedding 408 sampled from the current node 414 of the tree structure 406 represents an object depicted in an image already included in the list. If so, the sampling engine 324 continues to add embeddings 408 belonging to the same node 414 to the set of sampled embeddings 418 until it finds an embedding that represents an object in an image not included in the list. For example, the sampling engine 324 may sample one or more embeddings 408 in ascending order of distance 410 from the centroid of the hypersphere represented by that node 414. If the sampling engine 324 determines that the sampled embedding 408 represents an object in an image not included in the list, it adds the image to the image list. The sampling engine 324 repeats this process until the number of images in the list meets the sampling criteria 432 for the sampled dataset 428.

[0076] Additionally or alternatively, the sampling engine 324 adds a new embedding 408 to the set of sampled embeddings 418 by identifying an image in which the object represented by the new embedding 408 appears and further identifying other objects appearing in the image. The sampling engine 324 then identifies all nodes 414 in the tree structure 406 that contain embeddings 408 corresponding to those other objects and adds those nodes 414 to the set of sampled nodes 424. The sampling engine 324 is thus configured to next sample an embedding 418 that represents an object appearing in an image different from the image associated with the previously sampled embedding 418. The sampling engine 324 can continue to add new embeddings to the set of sampled embeddings 418 in this manner until the number of images in the image list satisfies the sampling criteria 432 for the sampled dataset 428.

[0077] Once the sampled dataset 428 is generated, the sampling engine 324 provides the sampled dataset 428 for use in generating a training dataset. This training dataset includes some or all of the data points 420 included in the sampled dataset 428 and their corresponding labels 422(1) through 422(Y) (each of which may be individually referred to as a “label 422”). For example, the sampling engine 324 may output a list of identifiers, values, and other representations of the data points 420 to a user interface, a file, an email, a notification, another data source, or any combination thereof. A user may submit one or more of the output data points 420 for labeling (e.g., using the system 100 shown in FIG. 1 ). Once labels 422 have been generated for data points 420, one or more portions of the data points 420 and their corresponding labels 422 can be combined to form datasets, such as training datasets, testing datasets, and validation datasets, that are used to improve or evaluate the performance of one or more machine learning models on one or more tasks.

[0078] In one or more embodiments, the sampling engine 324 generates or outputs the sampled dataset 428 such that the labeled dataset generated from the sampled dataset 428 is highly diverse. More specifically, as described above, the embedding 418 is sampled from an unsampled node 414, and the embedding closest to the centroid of the hypersphere (or region) represented by the unsampled node 414 may be taken as the sampled embedding 418. The first sampled embedding obtained from the root node 414 of the tree structure 406 is closest to the “average” of all embeddings 408 associated with the dataset 402. Thus, the first sampled embedding is the most “representative” embedding, representing the distribution of embeddings 408 generated from the data points 404 included in the dataset 402. However, as the traversal 416 progresses from the root node 414 to the lower nodes 414 in the tree structure 406, the sampled embeddings 418 obtained from the lower nodes 414 become representative of the distribution of a subset of the embeddings 408 associated with the respective node 414 and become less representative of the distribution of all the data points 404 in the dataset 402. Furthermore, the lower nodes 414 in the tree structure 406 represent hyperspheres (or regions) farther from the center of gravity of the hyperspheres (or regions) represented by the higher nodes 414 in the tree structure 406, and therefore the sampled embeddings 418 obtained from these lower nodes 414 will represent data points 404 that are less frequent (equivalent to outliers). Therefore, including such embeddings can increase the diversity of the set of sampled embeddings 418.

[0079] As a result, the sampling engine 324 can create a sampled data set 428 that lists the data points 420 corresponding to the sampled embeddings 418 obtained by traversing 416 the tree structure 406 in the order in which they were obtained. Additionally or alternatively, the sampling engine 324 can output a maximum distance 434 measurement associated with each data point in the list. This allows the sampling engine 324 to set a "cutoff" for the number of data points 420 to be labeled. This cutoff allows for a trade-off between the diversity of labeled data points 420 (as indicated by the maximum distance 434 value) and the overhead associated with labeling these data points 420.

[0080] For example, the sampling engine 324 can generate a graph that plots maximum distance 434 as a function of the number of data points 420 included in the sampled data set 428. In this graph, a predetermined number N of data points 420 can be mapped to the first N sampled embeddings 418 obtained by traversing 416 the tree structure 406. Using this plot, the sampling engine 324 or a user can select a value for N that balances the diversity of the corresponding data points with the overhead associated with labeling (or other processing) those data points. The sampling engine 324 or a user can then generate labels for those N data points using the system 100 shown in FIG. 1 or one or more labeling techniques.

[0081] 6 is a flow diagram illustrating steps of a method for sampling from a dataset, according to various embodiments. Note that the method steps are described with reference to the systems shown in Figures 1-4, but one skilled in the art will understand that any system configured to perform the method steps in any order is within the scope of the present disclosure.

[0082] As shown, in step 602, the processing engine 322 obtains a set of embeddings corresponding to a set of data points included in the dataset. For example, the processing engine 322 can use one or more machine learning models to convert multidimensional data, such as text, images, audio, video, point clouds, or meshes, into an embedding representation in a low-dimensional vector space. Alternatively, the dataset can include a first set of data points corresponding to a set of images and a second set of data points corresponding to objects in the images, where the images may contain one or more objects. The processing engine 322 can use an object detection model to generate bounding boxes for the objects in the images. Furthermore, the processing engine 322 can use an image embedding model, such as a convolutional neural network, to convert a set of regions in the images defined by these bounding boxes into a set of embeddings.

[0083] In step 604, processing engine 322 constructs a tree structure using these embeddings. For example, processing engine 322 can calculate vector distance measures, such as Mahalanobis distance, Euclidean distance, or cosine similarity, between embeddings or between a given embedding and a distribution or centroid associated with a set of embeddings. Processing engine 322 can use these vector distance measures to organize the embeddings and generate types of tree structures, such as ball trees or KD trees. Each node in the tree structure represents a region in a vector space, such as a hypersphere, and contains one or more embeddings that reside within that region. Each non-leaf node in the tree structure contains two or more child nodes that correspond to different subregions of the region represented by the non-leaf node. Each child node represents a smaller region than its corresponding parent node and contains a subset of the embeddings contained in the parent node that belong to the region represented by the child node.

[0084] In step 606, the sampling engine 324 samples a subset of embeddings from the set of embeddings by traversing the tree structure. More specifically, the sampling engine 324 samples embeddings from each node in the tree structure, starting from the root node and proceeding downward. Furthermore, the sampling engine 324 tracks the set of sampled nodes, the set of sampled nodes with some sampled child nodes, and the set of sampled nodes with all sampled child nodes. In each traversal step, the sampling engine 324 identifies a node from the set of nodes with some sampled child nodes whose embeddings occupy the largest area in vector space. The sampling engine 324 then samples embeddings from the identified node's unsampled child nodes. The sampling engine 324 continues traversing until certain conditions are met. These conditions include sampling a predetermined number of embeddings from the tree structure, the maximum distance between an embedding included in the subset of sampled embeddings and an embedding not included in the subset of sampled embeddings being below a threshold, or other criteria being met. Note that if all nodes with some sampled children have been moved to the set of nodes with all sampled children, and if more embeddings still need to be sampled from the tree structure, sampling engine 324 samples additional embeddings from already sampled nodes in the tree structure. Additionally or alternatively, sampling engine 324 may sample embeddings representing objects depicted in the image based on criteria related to the generation of the image dataset, as described in more detail below in the discussion of Figures 7-9.

[0085] In step 608, the sampling engine 324 generates a sampled dataset that includes a subset of data points corresponding to the subset of embeddings sampled in step 606, a set of labels corresponding to the data points, or both. For example, the sampling engine 324 can list the subset of data points corresponding to the embeddings sampled by traversing the tree structure. The sampling engine 324 can also show how the diversity or representativeness of the top N data points changes for various values ​​of N. The sampling engine 324 or a user can then select a value of N that strikes a good balance between the diversity of the corresponding data points and the overhead associated with labeling (or other processing) the data points. The sampling engine 324 can also generate labels for the N data points using one or more labeling techniques.

[0086] 7 is a flow diagram illustrating method steps for sampling images from a dataset based on embeddings representing objects depicted in the images, according to various embodiments. Note that the method steps are described with reference to the systems shown in FIGS. 1-4, but one skilled in the art will understand that any system configured to perform the method steps in any order is within the scope of the present disclosure.

[0087] As shown, in step 702, the sampling engine 324 samples embeddings representing objects depicted in the image from nodes in the tree structure. For example, the sampling engine 324 may sample embeddings from the root node of the tree structure, from unsampled child nodes of a sampled node, or from both.

[0088] In step 704, the sampling engine 324 marks the node and one or more of its descendant nodes that contain the embedding as sampled. For example, the sampling engine 324 may add all nodes on the path from the node to the leaf node that contains the embedding to the set of sampled nodes.

[0089] In step 706, the sampling engine 324 adds the image to a subset of images already sampled from the dataset, for example, the sampling engine 324 adds the image to a list of images depicting the object represented by the embedding sampled from the tree structure.

[0090] In step 708, the sampling engine 324 determines whether the subset of images meets the sampling criteria. For example, the sampling engine 324 can compare the number of images in the subset with a pre-specified number of images to sample from a larger image dataset. If the images in the subset do not meet the sampling criteria, the sampling engine 324 continues to sample embeddings from the tree structure and add images corresponding to the embeddings to the subset of sampled images from the dataset by repeating steps 702-706. However, if an image depicting the object represented by the sampled embedding is already present in the subset of images, the sampling engine 324 does not add the same image to the subset of images in step 706. Furthermore, if all nodes in the tree structure have been marked as sampled but additional images still need to be added to the subset of images, the sampling engine 324 can sample from already sampled nodes in the tree structure. In this manner, the sampling engine 324 continues to sample embeddings from the tree structure and update the subset of images until the number of images in the subset reaches a pre-specified number or the images in the subset meet other sampling criteria.

[0091] 8 is a flow diagram illustrating method steps for sampling images from a dataset based on embeddings representing objects depicted in the images, according to various other embodiments. Note that the method steps will be described with reference to the systems shown in FIGS. 1-4, but one skilled in the art will understand that any system configured to perform the method steps in any order is within the scope of the present disclosure.

[0092] As shown, in step 802, the sampling engine 324 samples embeddings representing objects depicted in the image from nodes in the tree structure. For example, the sampling engine 324 may sample embeddings from the root node of the tree structure, from unsampled child nodes of a sampled node, or from both.

[0093] In step 804, the sampling engine 324 identifies a set of other objects depicted in the image. For example, the sampling engine 324 can perform a search using the image's identifiers to obtain a set of identifiers for all objects contained in the image.

[0094] In step 806, the sampling engine 324 marks as sampled the set of nodes in the tree structure that includes the embedding sampled in step 802 and any additional embeddings that represent other objects depicted in the image identified in step 804. For example, for each embedding, the sampling engine 324 may identify the nodes on the path from the root node to the leaf node that contains that embedding. The sampling engine 324 may add to the set of sampled nodes any identified nodes that are not already included in the set of sampled nodes.

[0095] In step 808, the sampling engine 324 adds the image to a subset of images already sampled from the dataset. For example, the sampling engine 324 can add the image to a list of images depicting the object represented by the embedding sampled from the tree structure.

[0096] In step 810, the sampling engine 324 determines whether the subset of images meets the sampling criteria. For example, the sampling engine 324 can compare the number of images in the subset with a pre-specified number of images to sample from a larger image dataset. If the images in the subset do not meet the sampling criteria, the sampling engine 324 continues to sample other embeddings from the tree structure and add images corresponding to the other embeddings to the subset of sampled images from the dataset by repeating steps 802-808. If an image depicting the object represented by the sampled embedding is already present in the subset of images, the sampling engine 324 does not add the same image to the subset of images in step 808. Furthermore, if all nodes in the tree structure have been marked as sampled but additional images still need to be added to the subset of images, the sampling engine 324 can sample from already sampled nodes in the tree structure. In this manner, the sampling engine 324 continues to sample embeddings from the tree structure and update the subset of images until the number of images in the subset reaches a pre-specified number or the images in the subset meet other sampling criteria.

[0097] 9 is a flow diagram illustrating method steps for sampling images from a dataset based on embeddings representing objects depicted in the images, according to various other embodiments. Note that the method steps will be described with reference to the systems shown in FIGS. 1-4, but one skilled in the art will understand that any system configured to perform the method steps in any order is within the scope of the present disclosure.

[0098] As shown, in step 902, the sampling engine 324 samples embeddings representing objects depicted in the image from nodes in the tree structure. For example, the sampling engine 324 may sample embeddings from the root node of the tree structure, from unsampled child nodes of a sampled node, or from both.

[0099] In step 904, the sampling engine 324 determines whether the image is included in the subset of images associated with previously sampled embeddings. For example, the sampling engine 324 can compare the image's identifier to a list of identifiers for images depicting the object represented by previously sampled embeddings from the tree structure.

[0100] If the image is in the subset of images associated with an already sampled embedding, sampling engine 324 repeats step 902 one or more times to sample another embedding belonging to the same node. For example, sampling engine 324 may perform step 902 by sampling from the node the embedding that is next closest to the centroid of the hypersphere represented by the node. Additionally, sampling engine 324 may repeat step 904 one or more times to determine whether the newly sampled embedding corresponds to an image in the subset of images associated with an already sampled embedding.

[0101] If the sampling engine 324 finds that the newly sampled embedding represents an object that is not included in any of the images in the subset of images associated with the previously sampled embeddings, the process proceeds to step 906. In step 906, the sampling engine 324 marks the set of nodes in the tree structure that contain the embedding as sampled. For example, the sampling engine 324 may identify nodes on a path from the node to a leaf node that contains the embedding. The sampling engine 324 may add identified nodes that are not already included in the set of sampled nodes to the set of sampled nodes.

[0102] In step 908, the sampling engine 324 adds the image to the subset of images associated with the previously sampled embedding. For example, the sampling engine 324 can add the image's identifier from the tree structure to a list of identifiers for images depicting the object represented by the sampled embedding.

[0103] In step 910, the sampling engine 324 determines whether the subset of images meets the sampling criteria. For example, the sampling engine 324 can compare the number of images in the subset with a pre-specified number of images to sample from the larger image dataset. If the images in the subset do not meet the sampling criteria, the sampling engine 324 continues to sample embeddings from the tree structure and add images corresponding to the embeddings to the subset of sampled images from the dataset by repeating steps 902-908. If all nodes in the tree structure have been marked as sampled but more images still need to be added to the subset of images, the sampling engine 324 can sample from nodes in the tree structure that have already been sampled. In this manner, the sampling engine 324 continues to sample embeddings from the tree structure and update the subset of images until the pre-specified number of images is reached or the images in the subset meet other sampling criteria.

[0104] In summary, the technique disclosed herein performs singular sampling on a dataset. In this singular sampling, a subset of data points is sampled from the dataset to maximize the variety, or "singularity," of the sampled data points. Specifically, a tree structure, such as a ball tree, is constructed using embeddings corresponding to the data points in the dataset. In this tree structure, a parent node stores embeddings corresponding to a set of data points. This set of data points is defined by a region, such as a hypersphere, that the embedding occupies in a multidimensional vector space. Meanwhile, two or more child nodes derived from the parent node store subsets of the embedding set of the parent node. These subsets are configured so that they do not intersect with each other. Similarly, the subsets of data points stored in each child node are also defined by a region, such as a hypersphere, in the multidimensional vector space. The tree structure is constructed so that the size of the region represented by each node is minimized, and the region defining the data points included in the child node is smaller than the region defining the data points included in its parent node.

[0105] A subset of data points is then sampled from the dataset by traversing the tree structure. The tree structure is traversed starting from the root node and progressing downward to sample an embedding representative of each node in the tree structure. Once an embedding from a node has been sampled, that node and all other nodes containing that embedding (e.g., each node along the path from that node to a leaf node that represents a region where that embedding may exist) are marked as sampled. This process is repeated for other unsampled nodes until a predetermined condition is met, such as sampling all nodes, sampling a predetermined number of embeddings, the maximum possible distance between an embedding in the subset and the rest of the embeddings in the dataset being below a threshold, or some other criteria being met. The subset of data points represented by the sampled embeddings can then be passed on to a labeling or other process as a "high-diversity" representation of the dataset.

[0106] The technical advantages of the technology disclosed herein over the prior art include the ability to suppress duplicate data points and highly similar data points in a subset obtained by sampling a subset of data points from a large dataset, thereby increasing the diversity of the data points in the subset obtained by sampling. Therefore, by training a machine learning model using data points sampled using the technology disclosed herein and their corresponding labels, it is possible to obtain a machine learning model with higher predictive performance and less bias than a machine learning model trained using data points randomly sampled from a large dataset. Another technical advantage of the technology disclosed herein is the ability to efficiently sample a subset of data points by scanning a tree structure constructed based on a large dataset. This reduces the time and resource burden compared to conventional methods that calculate the distance between embeddings for all possible combinations of data points to increase the diversity of data points sampled from a dataset. These technical advantages result in one or more technical improvements over prior art methods.

[0107] Section 1 In some embodiments, a computer-implemented method for sampling a dataset is provided, the method comprising: obtaining a plurality of embeddings corresponding to a plurality of data points in the dataset; and constructing a tree structure using the plurality of embeddings, wherein constructing the tree structure comprises generating a first node storing a first set of embeddings in the plurality of embeddings; and generating a first plurality of nodes as child nodes of the first node, each node in the first plurality of nodes storing a subset of another embedding in the first set of embeddings, the method further comprising traversing the tree structure to sample a subset of embeddings from the plurality of embeddings; and generating a sampled dataset comprising a subset of data points corresponding to the subset of embeddings.

[0108] Section 2 2. The computer-implemented method of claim 1, wherein constructing the tree structure using the plurality of embeddings comprises generating the tree structure based on a plurality of distances associated with the plurality of embeddings.

[0109] Section 3 3. The computer-implemented method of claim 1 or 2, wherein the plurality of distances includes Mahalanobis distances.

[0110] Section 4 4. The computer-implemented method of any of claims 1 to 3, wherein sampling the subset of embeddings comprises: sampling a first embedding from the first node; and, in response to sampling the first embedding, marking the first node and one or more descendant nodes of the first node that contain the first embedding as sampled.

[0111] Section 5 5. The computer-implemented method of claim 1, wherein sampling the subset of embeddings further comprises: sampling a second embedding from a second node included in the first plurality of nodes that has not been marked as sampled; and, in response to sampling the second embedding, marking the second node and one or more descendant nodes of the second node that contain the second embedding as sampled.

[0112] Section 6 6. The computer-implemented method of any of claims 1 to 5, wherein sampling the subset of embeddings further comprises selecting the second node based on the size of the region represented by the first node.

[0113] Section 7 7. The computer-implemented method of any of claims 1 to 6, wherein sampling the subset of embeddings further comprises sampling a third embedding from the second node after marking all nodes in the tree structure as sampled.

[0114] Section 8 8. The computer-implemented method of any of claims 1 to 7, wherein sampling the first embedding includes identifying the embedding closest to the center of the region represented by the first node as the first embedding.

[0115] Section 9 9. The computer-implemented method of any of claims 1 to 8, wherein the plurality of data points comprises at least one of an image or an object depicted in an image.

[0116] Section 10 10. The computer-implemented method of any of claims 1 to 9, wherein the tree structure comprises a ball tree.

[0117] Section 11 In some embodiments, one or more non-transitory computer-readable media are provided that store instructions that, when executed by one or more processors, cause the one or more processors to obtain a plurality of embeddings corresponding to a plurality of data points in a dataset and build a tree structure using the plurality of embeddings, where building the tree structure involves generating a first node that stores a first set of embeddings in the plurality of embeddings and generating a first plurality of nodes as child nodes of the first node, each node in the first plurality of nodes storing a subset of another embedding in the first set of embeddings, and further cause the one or more processors to sample a subset of embeddings from the plurality of embeddings by traversing the tree structure and generate a sampled dataset including a subset of data points corresponding to the subset of embeddings.

[0118] Section 12 12. The one or more non-transitory computer-readable media of claim 11, wherein constructing the tree structure using the plurality of embeddings includes dividing the plurality of embeddings into two or more subsets of embeddings, generating two or more subtree structures corresponding to the two or more subsets of embeddings, and consolidating the two or more subtree structures under a root node.

[0119] Section 13 13. The one or more non-transitory computer-readable media of claim 11 or 12, wherein constructing the tree structure using the plurality of embeddings comprises applying a whitening transform to the plurality of embeddings to generate a plurality of whitening-transformed embeddings; calculating a plurality of Euclidean distances associated with the plurality of whitening-transformed embeddings; and generating the tree structure based on the plurality of Euclidean distances.

[0120] Section 14 14. The one or more non-transitory computer-readable media of any of clauses 11-13, wherein sampling the subset of embeddings includes sampling from the first node a first embedding that is closest to the centroid of the hypersphere represented by the first node; and in response to sampling the first embedding, marking the first node and one or more descendant nodes of the first node that include the first embedding as sampled.

[0121] Section 15 15. The one or more non-transitory computer-readable media of any of clauses 11-14, wherein sampling the subset of embeddings further comprises: sampling a second embedding from a second node included in the first plurality of nodes and not marked as sampled; and, in response to sampling the second embedding, marking the second node and one or more descendant nodes of the second node that include the second embedding as sampled.

[0122] Section 16 16. The one or more non-transitory computer-readable media of any of claims 11 to 15, wherein the one or more descendant nodes include a leaf node in the tree structure.

[0123] Section 17 17. The one or more non-transitory computer-readable media of any of clauses 11-16, wherein sampling a subset of the embeddings comprises continuing to sample embeddings from the tree structure until a threshold on the number of data points to be included in the sampled data set is reached.

[0124] Section 18 18. The one or more non-transitory computer-readable media of any of clauses 11-17, wherein generating the sampled data set includes determining the number of data points to include in the subset of data points based on a maximum distance between any embedding in the subset of embeddings and any embedding included in the plurality of embeddings but excluded from the subset of embeddings.

[0125] Section 19 19. The one or more non-transitory computer-readable media of any of clauses 11-18, wherein generating the sampled dataset comprises selecting a subset of the data points to include in the sampled dataset based on the order in which the traversal of the tree structure sampled the subset of embeddings.

[0126] Section 20 In some embodiments, a system is provided comprising one or more memories storing instructions and one or more processors coupled to the one or more memories, the one or more processors executing the instructions to obtain a plurality of embeddings corresponding to a plurality of data points in a dataset and to build a tree structure using the plurality of embeddings, the building of the tree structure being performed by generating a first node that stores a first set of embeddings in the plurality of embeddings and generating a first plurality of nodes as child nodes of the first node, each node in the first plurality of nodes storing a subset of another embedding in the first set of embeddings, the instructions further configuring the one or more processors to traverse the tree structure to sample a subset of embeddings from the plurality of embeddings and to generate a training dataset including a subset of data points corresponding to the subset of embeddings and a set of labels for the subset of data points.

[0127] Any combination of any element recited in any claim and / or any element described in this specification, regardless of the form of such combination, is intended to fall within the scope and protection contemplated by the present invention.

[0128] The various embodiments described above have been presented for illustrative purposes and are not intended to be exhaustive or limiting of the embodiments of the present disclosure. Those skilled in the art will recognize that many modifications and variations can be made without departing from the scope and spirit of the embodiments described herein.

[0129] Aspects of the present disclosure may be embodied as a system, method, or computer program product. Accordingly, aspects of the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be collectively referred to herein as a "module," a "system," or a "computer." Also, any hardware and / or software technique, process, function, component, engine, module, or system described in this disclosure may be implemented as a single circuit or set of circuits. Furthermore, aspects of the present disclosure may take the form of a computer program product embodied as computer-readable program code recorded on one or more computer-readable medium(s).

[0130] One or more computer-readable media may be utilized in any combination. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific (but non-exhaustive) examples of computer-readable storage media include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. As used herein, a computer-readable storage medium may refer to any tangible medium capable of storing a program for use by or in connection with a system, apparatus, or device that executes instructions.

[0131] Aspects of the present disclosure have been described above with reference to flowchart diagrams and / or block diagrams illustrating methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It will be understood that each block of the flowchart diagrams and / or block diagrams, and combinations of blocks in the flowchart diagrams and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a programmable data processing device, such as a general-purpose computer or a special-purpose computer, to produce a machine. The instructions, when executed by the processor of the computer or programmable data processing device, cause the implementation of the functions or acts specified in one or more blocks of the flowchart diagrams and / or block diagrams. Such a processor may include, but is not limited to, a general-purpose processor, a special-purpose processor, an application-specific processor, or a field-programmable gate array.

[0132] The flowcharts and block diagrams shown in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams can be considered to represent a module, segment, or portion of code, including one or more executable instructions for implementing the logical function(s) defined in the block. It should also be noted that in some alternative implementations, the functions shown in the blocks can be executed in an order different from that shown in the figures. For example, two blocks shown as consecutive blocks may actually be executed substantially simultaneously, or in some cases, the blocks may be executed in the reverse order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, can be implemented by a dedicated hardware-based system that performs the functions or operations defined in the block, or a combination of dedicated hardware and computer instructions.

[0133] It should be noted that while the foregoing description is directed to embodiments of the present disclosure, other and further embodiments of the present disclosure may be devised without departing from the basic scope thereof, which scope is defined by the claims set forth below. [Explanation of symbols]

[0134] 100 systems 102 client devices 104 Data Labeling Applications 110 Customer Devices 130 servers 132 Task Application Programming Interface (Task API) 134 Server Applications 150 Network 302 Central Processing Unit (CPU) 304 system memory 305 Memory Bridge 306, 313 communication paths 307 Input / Output (I / O) Bridge 308 Input Devices 310 Display Devices 312 Display Processor 314 System Disk 316 Switch 318 Network Adapter 320, 321 add-in card 322 Processing Engine 324 Sampling Engine 402 datasets 404, 420 data points 406 Tree Structure 408 Embedded 410 Distances Associated with Embeddings 412 Tree Structure Construction Criteria 414 nodes 416 Traversing a Tree Structure 418 Sampled Embedding 422 Label 424 sampled nodes 426 Nodes with some child nodes already sampled 428 sampled datasets 430 Nodes where all child nodes have been sampled 432 Sampling Criteria 434 Maximum distance 502, 504, 506 passes

Claims

1. 1. A computer-implemented method for sampling a dataset, the method comprising: obtaining a plurality of embeddings corresponding to a plurality of data points included in the dataset; constructing a tree structure using the plurality of embeddings; and constructing the tree structure comprises: creating a first node that stores a first set of embeddings included in the plurality of embeddings; generating a first plurality of nodes as child nodes of the first node, each node of the first plurality of nodes storing a subset of another embedding in the first set of embeddings; The method comprises: sampling a subset of embeddings from the plurality of embeddings by traversing the tree structure; generating a sampled data set comprising a subset of data points corresponding to a subset of the embedding; The method further comprises:

2. 2. The computer-implemented method of claim 1, wherein constructing the tree structure using the multiple embeddings comprises generating the tree structure based on multiple distances associated with the multiple embeddings.

3. The computer-implemented method of claim 2 , wherein the plurality of distances comprises Mahalanobis distances.

4. sampling a subset of the embeddings, sampling a first embedding from the first node; In response to sampling the first embedding, marking the first node and one or more descendant nodes of the first node that include the first embedding as sampled; 10. The computer-implemented method of claim 1, comprising:

5. sampling a subset of the embeddings, sampling a second embedding from a second node included in the first plurality of nodes that is not marked as sampled; In response to sampling the second embedding, marking the second node and one or more descendant nodes of the second node that include the second embedding as sampled; The computer-implemented method of claim 4 further comprising:

6. 6. The computer-implemented method of claim 5, wherein sampling the subset of embeddings further comprises selecting the second node based on the size of the region represented by the first node.

7. 6. The computer-implemented method of claim 5, wherein sampling the subset of embeddings further comprises sampling a third embedding from the second node after marking all nodes in the tree structure as sampled.

8. 5. The computer-implemented method of claim 4, wherein sampling the first embedding comprises identifying as the first embedding an embedding that is closest to a center of a region represented by the first node.

9. The computer-implemented method of claim 1 , wherein the plurality of data points comprises at least one of an image or an object depicted in an image.

10. The computer-implemented method of claim 1 , wherein the tree structure comprises a ball tree.

11. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to: obtaining a plurality of embeddings corresponding to a plurality of data points in the dataset; constructing a tree structure using the plurality of embeddings; and constructing the tree structure, creating a first node that stores a first set of embeddings included in the plurality of embeddings; generating a first plurality of nodes as child nodes of the first node, each node of the first plurality of nodes storing a subset of another embedding in the first set of embeddings; The instructions may be to the one or more processors: sampling a subset of embeddings from the plurality of embeddings by traversing the tree structure; generating a sampled data set comprising a subset of data points corresponding to a subset of the embedding; and one or more non-transitory computer-readable media further performing the steps of:

12. constructing the tree structure using the plurality of embeddings, dividing the plurality of embeddings into two or more subsets of embeddings; generating two or more subtree structures corresponding to the two or more subsets of embeddings; and aggregating the two or more subtree structures under a root node.

13. constructing the tree structure using the plurality of embeddings, applying a whitening transform to the plurality of embeddings to generate a plurality of whitening-transformed embeddings; calculating a plurality of Euclidean distances associated with the plurality of whitened transformed embeddings; generating the tree structure based on the plurality of Euclidean distances; 12. The one or more non-transitory computer-readable media of claim 11, comprising:

14. sampling a subset of the embeddings, sampling from the first node a first embedding that is closest to the centroid of the hypersphere that the first node represents; In response to sampling the first embedding, marking the first node and one or more descendant nodes of the first node that include the first embedding as sampled; 12. The one or more non-transitory computer-readable media of claim 11, comprising:

15. sampling a subset of the embeddings, sampling a second embedding from a second node included in the first plurality of nodes that is not marked as sampled; In response to sampling the second embedding, marking the second node and one or more descendant nodes of the second node that include the second embedding as sampled; 15. The one or more non-transitory computer-readable media of claim 14, further comprising:

16. The one or more non-transitory computer-readable media of claim 14 , wherein the one or more descendant nodes comprise leaf nodes in the tree structure.

17. 12. The one or more non-transitory computer-readable media of claim 11, wherein sampling the subset of embeddings comprises continuing to sample embeddings from the tree structure until a threshold is reached on the number of data points to be included in the sampled data set.

18. 12. The one or more non-transitory computer-readable media of claim 11, wherein generating the sampled data set comprises determining a number of data points to include in the subset of data points based on a maximum distance between any embedding included in the subset of embeddings and any embedding included in the plurality of embeddings but excluded from the subset of embeddings.

19. 12. The one or more non-transitory computer-readable media of claim 11, wherein generating the sampled dataset comprises selecting a subset of the data points to include in the sampled dataset based on an order in which the traversal of the tree structure sampled the subset of embeddings.

20. one or more memories for storing instructions; one or more processors coupled to the one or more memories; A system comprising: The one or more processors execute the instructions to: obtaining a plurality of embeddings corresponding to a plurality of data points in the dataset; constructing a tree structure using the plurality of embeddings; and constructing the tree structure, creating a first node that stores a first set of embeddings included in the plurality of embeddings; generating a first plurality of nodes as child nodes of the first node, each node of the first plurality of nodes storing a subset of another embedding in the first set of embeddings; The instructions may be to the one or more processors: sampling a subset of embeddings from the plurality of embeddings by traversing the tree structure; generating a training dataset comprising a subset of data points corresponding to the subset of embeddings and a set of labels for the subset of data points; The system is further configured to:

Images