Sampling techniques for data clustering

Abstract

Techniques and solutions are described for clustering observations in a dataset with reduced memory consumption. A first observation is received. A distance is determined between the first observation and the first cluster. It is determined that the distance satisfies a threshold, and a first observation is added to a first sample of the first cluster. A first set of average values is calculated for the first sample using the first observed data and at least a second observed data of the first sample. A second set of averages is calculated for the plurality of elements of the first cluster using the first set of averages for the first sample and at least a third set of averages for at least a second sample of the first cluster. An identifier of the first observation is added to a set of observation identifiers classified in the first cluster.

Description

Technical Field

[0001] In general, the present disclosure relates to data clustering techniques. Certain implementations provide for defining clusters using summary information of data observations included in the clusters. Background Art

[0002] There are a variety of scenarios where it may be of interest to determine which portions of a video contain specific elements, such as a face. For example, in security applications, it may be useful to view a camera lens to identify portions of the lens that include an image of a specific person. Manually scanning a camera lens may be time consuming, tedious, and prone to error. As another example, streaming video content is becoming increasingly popular. It may be beneficial to provide users with more information about video content or to allow users to interact with video content (which may not be practical in other media formats).

[0003] Video processing techniques can be very time consuming and require a lot of processor time. For example, analyzing a video for the presence of a specific individual may require storing and analyzing thousands of video samples. In the case of streaming movies, consider an average movie that has a running time of two hours and is presented at a frame rate of 24 frames per second. This content contains 172,800 frames. In turn, processing each frame can generate many values ​​to be stored and processed. In the case of facial recognition techniques, a face can be detected in a given frame, and then a vector of 128 or 512 elements can be used to extract facial features. Therefore, there is room for improvement. Summary of the invention

[0004] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0005] Techniques and solutions for clustering observations in a data set are described. A first observation is received. A distance is determined between the first observation and a first cluster. It is determined that the distance satisfies a threshold, and the first observation is added to a first sample of the first cluster. A first set of means is calculated for the first sample using data of the first observation and data of at least a second observation of the first sample. A second set of means is calculated for a plurality of elements of the first cluster using the first set of means for the first sample and at least a third set of means for at least a second sample of the first cluster. An identifier of the first observation is added to a set of observation identifiers classified in the first cluster. Using the sample as summary information for individual operations classified in the first cluster reduces memory consumption compared to using data of discrete observations classified in the first cluster to calculate the second set of means.

[0006] In one aspect, a clustering method is provided. A first observation instance for an input data set is received. The first observation instance includes a first plurality of values ​​of a corresponding plurality of elements of the input data set. The first plurality of values ​​of the plurality of elements describe a feature of interest in the input data set.

[0007] Determining a distance between the first observation and a first cluster of samples having a location determined using observations classified within the first cluster Determining that the distance satisfies a threshold Based on the determining that the distance satisfies the threshold, adding the first observation to a first sample associated with the first cluster.

[0008] For the first sample, a first set of means for the plurality of elements is calculated using the first plurality of values ​​of the first observation and at least a second plurality of values ​​of the plurality of elements of at least a second observation in the first sample. A second set of means for the plurality of elements of the cluster is calculated using the first set of means for the first cluster and at least a third set of means for at least a second sample associated with the first cluster. Using the sample as summary information for individual observations classified within the first cluster reduces memory consumption compared to calculating the second set of means using data for discrete observations classified within the first cluster. Based on the determination that the distance satisfies the threshold, an identifier for the first observation is added to a set of observation identifiers classified in the first cluster.

[0009] The present disclosure also includes computing systems and tangible, non-transitory computer-readable storage media configured to perform the above methods or including instructions for performing the above methods.As described herein, various other features and advantages may be incorporated into the present technology as desired. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a diagram showing a processing pipeline for clustering image data.

[0011] Figure 2A and 2B A flow chart of the clustering technique using observational summary information is presented.

[0012] Figure 3 A flow chart of a method for determining whether clusters should be merged is presented.

[0013] Figure 4 Mathematical details are provided for examples of how observation summary information can be used in clustering operations.

[0014] Figure 5 is a diagram illustrating information that may be maintained for clusters associated with image data.

[0015] Figure 6 is a graph showing the relationship between observations, samples, and clusters.

[0016] Fig. 7A and 7B Code for an example implementation of the disclosed technology is presented.

[0017] Figure 8 is a flow chart of an example clustering operation according to an embodiment of the present disclosure.

[0018] Fig. 9 is a diagram of an example computing system in which some described embodiments may be implemented.

[0019] Fig.10 is an example cloud computing environment that can be used with the techniques described in this article DETAILED DESCRIPTION

[0020] Example 1 - Overview

[0021] There are a variety of scenarios where it may be of interest to determine which portions of a video contain specific elements, such as a face. For example, in security applications, it may be useful to view a camera lens to identify portions of the lens that include an image of a specific person. Manually scanning a camera lens may be time consuming, tedious, and prone to error. As another example, streaming video content is becoming increasingly popular. It may be beneficial to provide users with more information about video content or to allow users to interact with video content (which may not be practical in other media formats).

[0022] Video processing techniques can be very time consuming and require a lot of processor time. For example, analyzing a video for the presence of a specific individual may require storing and analyzing thousands of video samples. In the case of streaming movies, consider an average movie that has a running time of two hours and is presented at a frame rate of 24 frames per second. This content contains 172,800 frames. In turn, processing each frame can generate many values ​​to be stored and processed. In the case of facial recognition techniques, a face can be detected in a given frame, and then a vector of 128 or 512 elements can be used to extract facial features. Therefore, there is room for improvement.

[0023] The disclosed techniques provide improved image recognition and clustering, including techniques that reduce processing time and resources. For example, a sampling technique is provided in which multiple observations (such as image frames) are assigned to samples associated with different feature clusters (where each sample is a set of observations), where the feature clusters can represent different image elements, such as a specific human being (and in particular, facial images based on a specific human being). Once a determined number of elements have been received for a given sample, the average of the observations in the sample is stored, and the data for individual observations is deleted. Observation identifiers (such as identifiers of specific images / video frames) can be maintained in a list of observations associated with a particular cluster. The process can then be repeated as additional observations are assigned to various samples of various clusters.

[0024] Techniques for determining when new clusters should be formed and when clusters should be merged are also provided. For example, an observation can be analyzed to determine whether it is sufficiently "close" to an existing cluster, such as using the Euclidean distance of a vector of values ​​associated with the observation and a vector associated with a given cluster (such as a vector representing the centroid of the cluster). If the observation is not within the determined distance or other measure of "closeness", a new cluster can be created for the observation.

[0025] In the case of merging clusters, the centroids of the two clusters can be analyzed to determine whether they are close enough to meet the merging criteria, such as having a Euclidean distance between their centroids that is less than a threshold amount. If so, the clusters can be merged, including defining the centroid of the merged cluster and combining the observation lists associated with the merged clusters. In at least some cases, the merging operation is performed after the observation clustering has been completed or completed to a desired degree, because it may be difficult to split the cluster into multiple clusters.

[0026] Various data used in the disclosed technology can be maintained in suitable computer-implemented structures. For example, input data sets, clusters, samples, and observations can be maintained in data structures (such as stacks, queues, graphs, heaps, trees, etc.), or in instances of data types (such as vectors or instances of abstract or complex data types).

[0027] Although the disclosed technology can find use in a variety of scenarios, a specific example is provided to illustrate how the disclosed technology can be used. In this example, a movie is to be analyzed to determine the specific actors or actresses present in the movie. The end result of the process is a cluster corresponding to the specific actor or actress, and a list of frames of the movie that fall within one of those clusters.

[0028] Frames of a movie can be extracted as individual images. The images can be processed using face detection techniques. Images corresponding to a particular face in a given image can then be processed using feature extraction techniques to obtain a data set describing the face in a given portion of the image. This data set is then compared to the set of clusters built and modified during "training" (the processing of the movie). If a given facial image has data similar enough to an existing cluster, it can be added to the list of observations associated with the cluster (and therefore identified as containing a frame of the actor or actress associated with the cluster), and the data associated with the cluster can be recalculated.

[0029] Taking a particular cluster of a particular actor or actress as an example, the center of the cluster can be defined, for example, based on samples that have been classified as belonging to the cluster. As new observations are added to the sample, the value of the sample and the definition of the center of the cluster can be modified. Once the sample reaches a certain number of samples, the data of the individual observations (images) in the sample are removed, and only summary information of the sample is retained, such as the average value of the observations originally in the sample (or the center point of higher dimensional data).

[0030] The list of frames in a particular cluster can be used for various purposes, including providing a user with a list of movie clips featuring a particular actor or actress. In some embodiments, the user can select to view one of these segments, where the first frame in the set of consecutive frames can be used as the starting point for video playback. In response to a user request, a portion of a movie corresponding to a selected time window can be provided, such as in a streaming video file.

[0031] In another example, the surveillance video may be processed as described for the movie scene above. The image of the person of interest may be compared to the clusters to determine if the image falls within the cluster, which indicates that the person of interest is in the processed surveillance video. The user may then be provided with a list of times / time windows in which the person of interest appears in the video.

[0032] The disclosed techniques may provide a number of advantages. Since the clustering technique is dynamic, the number of clusters may be determined at run time. In the case of videos of unknown content, it may be difficult to determine a static number of clusters prior to image processing. However, clustering is also tunable in that a distance parameter may be adjusted to determine when clusters should be merged, or to determine when a new cluster should be created when an observed distance does not fall within a threshold distance of an existing cluster.

[0033] Compared to techniques that maintain or use all observation data, the disclosed techniques also use reduced memory because once the sample size has been reached, only sample mean information (used as summary information for observations classified within a given cluster) is maintained, rather than individual observation data. Both memory resources and processor resources are saved because the number of calculations used to calculate the sample means that the mean (centroid) of the cluster is limited to the set sample size (rather than, for example, using all individual observations in the cluster to calculate its centroid), where a new sample is created once the sample observation set is full. Therefore, a given observation is only used a maximum number of times equal to the maximum number of observations in the sample. This can be beneficial compared to techniques that can use observation data until the results converge or meet a determined number of iterations (which is not bounded by the sample size). Similarly, although calculations for a given cluster can continue until the processing of the data set is completed, processing the observations in the sample set helps to reduce the processing load, such as reducing the amount corresponding to the determined size of the sample. Since the number of calculations can be significantly reduced, the disclosed techniques can allow faster processing, processing using less powerful hardware, or processing data sets that may be too large to be actually processed using other techniques.

[0034] Example 2 - Example processing pipeline for image feature clustering

[0035] Figure 1 An example end-to-end processing scenario 100 is shown in which the disclosed techniques may be used. Although the processing scenario 100 will be described with respect to image / video data, it should be understood that, in particular disclosed clustering techniques, the disclosed techniques may be used in other kinds of data processing.

[0036] The image or video information 104 is provided to a face detection component 108. In some cases, the face detection component 108 uses images rather than videos. In this case, the processing step may involve extracting an image from the video input 116 at 112, wherein the extracted image is provided to the face detection component 108.

[0037] The face detection component 108 can perform one or both of detecting faces in an image and extracting the detected faces from the image. In this context, "detecting" a face means determining that a portion of an image corresponds to a face, rather than determining an identity associated with the face or otherwise providing any information about what the face represents, or whether the face is related to any other faces that may have been extracted. Suitable techniques for face detection include multi-task cascaded convolutional neural networks (MTCNNs), including as described in "Joint Face Detection and alignment using Multi-Task Cascaded Convolutional Networks" by Zhang et al. (arXiv 1604.02878 (2016)). Additional face detection techniques are described in "Human Face Detection Techniques: A Comprehensive Review and Future Research Directions" by Hasan et al. Electronics, 10: 2354 (2021).

[0038] The output of the face detection component 108 is facial image data 120, which may be a portion of an image corresponding to a detected face, or an overall image in which a face appears, but where the location of the face in the image is provided along with the overall image. The facial image data 120 is processed by a feature extraction component 124. The feature extraction component 124 provides as output facial feature data 128 (or embeddings), which describe properties of the facial image and can be used to compare different facial images. Note that while the facial feature data 128 can be used for facial recognition, the disclosed techniques can be used to cluster facial images regardless of when or whether the facial feature data is associated with a known face / identity.

[0039] Various techniques may be used as or as part of the face extraction component 124. One suitable technique is described in Schroff et al., "FaceNet: A Unified Embedding for Face Recognition and Clustering," arXiv 1503.03832 (2015). Additional techniques that may be used to generate embeddings include Ageitgey (https: / / github.com / ageitgey / face_recognition), CompreFace (Exadel, Inc.), DeepFace (https: / / github.com / serengil / deepface), InsightFace (https: / / github.com / deepinsight / insightface), and InsightFace-REST (https: / / github.com / SthPhoenix / InsightFace-REST), Rekognition (Amazon.com, Inc.). Although many feature extraction components use machine learning techniques (including the use of deep neural networks) to provide embeddings, the present disclosure is not limited to such techniques. In particular, the facial feature data 128 may be latent features (such as generated using a convolutional neural network), or may be non-latent features (e.g., directly describing features such as distances between different facial features, locations of various facial anchor points, etc.).

[0040] Suitable technologies exist for both face detection and feature extraction, including the Mobile Vision API from Google, now included in ML Kit from Google.

[0041] The facial feature data 128 is provided to a clustering component 132, which may use the techniques described in the present disclosure. The clustering component 132 provides a clustering output 136. Depending on the particular use case, the clustering output 136 may include various information. Typically, for clusters determined from processing the input image / video 104, the clustering output 136 includes a cluster identifier and information about the video frames / images containing images that fall within the cluster. The frame / image information may include a frame number / image identifier.

[0042] In the case of a video (or a time series of images), the clustering output 136 may include information about the time or time window that contains a given image (face), where the time window may be defined based on consecutive frames / images falling within a particular cluster or using other criteria. In some cases, rules may be applied to help define the time window. That is, a given face corresponding to a particular individual (such as an actor or actress) may appear multiple times within a given time window, but may not appear in all frames of the time window. Criteria may be used to determine when a window should be opened or closed, such as defining a new window if a face within a given cluster does not appear within a specific time of a previous instance of a face. It should be noted that a frame / image may have multiple faces, or a frame / image within a specific time period may include multiple faces. Therefore, a given "window" need not be associated with a single face, and windows may optionally be configured to overlap.

[0043] although Figure 1 Not shown, but clustering output 136 may include additional information, such as data used to define clusters, including data defining the centroids of clusters and information used to calculate the centroids, such as information for various samples.

[0044] Example 3 - Example Clustering Technique Using Observation Summary Information

[0045] Figure 2A and Figure 2B An example process 200 for clustering data and particularly video data (as a video or as a series of extracted images) according to the present disclosure is shown. The law of large numbers indicates that the mean value of many independent samples has a high probability of being close to the mean value of the underlying distribution from which the sample is obtained. Similarly, the density histograms of many independent samples have a high probability of being close to the density histogram of the underlying distribution from which the sample is obtained. Typically, as the number of independent samples increases, the closer the sample mean and density histogram are to the underlying distribution, it is a value that can be calculated using all underlying data rather than just a sample set. In many cases, even for samples of moderate size (such as a set with about 30 samples), the law of large numbers also provides good results.

[0046] The disclosed technology utilizes the implications of the law of large numbers to collect observations (images / video frames) in samples of (usually fixed or static) size, such that the average of the samples is a good substitute for the individual values ​​of the observations (frames / images) in the collection. In some cases, the law of large numbers can be applied at multiple levels, such as to define the size of the sample, and to define the number of observations to be analyzed in a larger pool of observations. For example, a sample can be defined as having 30 samples, and 30 samples can be extracted from a larger pool of samples. In the case of a video running at 30 frames per second, only 30 frames may be sampled in a five second period.

[0047] For ease of presentation, this Example 3 continues to assume that it is used with the previously discussed use case of clustering video frames / images into clusters representing specific actors or actresses, but these techniques can be used in other applications. Mathematically, the set of clusters C at time t=0 can be defined as in is the cluster mean vector of cluster i, and where the cluster mean vector describes the centroid of the cluster. In the specific example, represents the embedding of the ith actor or actress, which can be defined as in is the sample mean vector of the jth cluster of a given sample set S for a given cluster i.

[0048] As discussed, each sample is based on a particular number of samples up to a maximum value n, where the value of n can be adjusted as described, but is typically selected using the implications of the law of large numbers as described above.

[0049] The sample mean vector of a given sample in the sample set S can be calculated as where N is the number of observations o (and N is less than or equal to n), and where at time t=0, |N|=0, and where o k is the facial recognition data (embedding) for a particular observation, typically provided in the form of a vector. Note that the sample mean vector also includes the embedding, where each element of the sample mean vector corresponds to the mean of a particular embedding computed from the corresponding embedding elements of the observations in the sample. Similarly, the cluster mean vector has elements that are embeddings computed from the corresponding elements / embeddings of the sample mean vector.

[0050] Now, turning to process 200, and specifically to Figure 2A The process begins at 204 where an initial observation is received. In the specific example described above, this observation is in the form of a vector where each element of the vector provides a value corresponding to an embedding at a corresponding vector position. Common facial recognition techniques provide 128 different embeddings that can be represented by a vector having 128 elements, one element per embedding.

[0051] Since the observations received at 204 are initial observations, no clusters have yet been defined. Therefore, initial clusters are defined at 206. Figure 2A Not shown in , but at this point, the initial observation is assigned to the first sample of the initial cluster. The centroid / mean vector of clusters and samples can be calculated, but at this point it is equal to the initial observation.

[0052] At 208, a determination is made as to whether any additional observations have been provided or are available as input. If not, process 200 may end at 210. Otherwise, process 200 proceeds to 212 where the next observation is received. At this point, since clusters exist, a determination needs to be made as to whether the next observation should be placed in an existing cluster or in a new cluster. The distance between the observation and the current set of clusters is calculated at 214, although there is only a single cluster for the second observation, there may be multiple clusters when the "next observation" is an observation being analyzed after many other observations have been processed. The distance may be calculated as the Euclidean distance where x, y, z, ... are the elements / embeddings of the corresponding vectors.

[0053] At 216, the cluster with the shortest distance between the center of the cluster and the observation is identified. At 218, it is determined whether the distance is less than a threshold distance, where the threshold distance can be adjusted to provide the desired results. Larger values ​​for the distance threshold can increase the chance that different individuals will be classified in the same cluster, while smaller values ​​can increase the chance that multiple clusters will be created for a single individual. If it is determined at 218 that the distance is not less than the threshold distance, process 200 can proceed to Figure 2B 250 , otherwise, the observation is assigned to the nearest cluster and the process proceeds to 220 .

[0054] At 220, it is determined whether the current sample of the cluster has a number of observations that is less than the maximum number of observation sets for the cluster samples. If the number of observations is less than a threshold, process 200 proceeds to 222, where the observation is added to the current sample. An updated sample mean vector is calculated at 224 (using the new observations and existing observations in the sample), and the recorded sample mean of the sample is updated at 226. Since the center of a cluster is calculated from the set of sample mean vectors for the samples of the cluster, an updated cluster mean vector (representing the centroid of the cluster) is calculated at 228, which is used as the new cluster centroid at 230. At 232, the frame (or other identifier associated with the observation) is added to the list of frames associated with the cluster. Process 200 then returns to 208.

[0055] If it is determined at 220 that the current sample of the cluster has reached the maximum number of observations, the current sample is deleted at 234. That is, the summary information of the sample (its mean vector) is maintained, but the observation of the sample is deleted. Typically, sample mean information is maintained on a sample-by-sample basis, so it may be helpful to distinguish one sample (sample vector) from another. In some cases, this may include numbering the samples, in which case the number of samples may optionally be incremented at 236. However, in other cases, the sample mean vector may be maintained in a different manner, and 236 may simply represent continuing with an empty / new sample. At 238, the empty sample is filled with the current observation. Process 200 may then proceed to 222 (where the updated sample mean vector is simply equal to the vector used for the observation).

[0056] Now return to 216, where it is determined that the distance between the current observation and the nearest cluster is not less than a threshold, go to Figure 2B At 250, a new cluster is created, where the observation vector is set to the center (centroid) mean vector of the cluster. A new sample is created, and the observation is added to the sample (as the first element in the sample's observation set), where the observation vector is also set to the sample mean vector at 252. At 254, the frame (or other observation identifier) ​​is added to the frame list of the new cluster. At 256, the process returns to 208.

[0057] After process 200 ends at 210, the set of sample mean vectors may be deleted if desired. Similarly, the cluster center mean vectors may also be deleted so that the clusters are not used to process further image data. That is, in some cases, the fact that there are different clusters and the identifiers of the frames / images associated with such clusters is the primary information of interest.

[0058] Example 4 - Example Merge of Clusters

[0059] As described in Example 1, it may be beneficial to evaluate the clusters to consider whether any of them should be merged together. That is, as training / data evaluation proceeds, clusters that were initially separate enough to be considered separate clusters may have their centers shifted so that they become close enough together that they were initially considered part of a common cluster. Thus, it may be the case that two clusters may have observations belonging to the same feature (i.e., face, person, etc.). The criteria for merging clusters may be the same distance used to determine whether an observation should be classified in an existing cluster or a new cluster that is created, or a different criteria may be used.

[0060] In at least some cases, it may be beneficial to perform the merge analysis after data processing is complete because clusters may not be split. The disadvantage is the use of additional memory / resources in maintaining samples of clusters that may have been merged with another cluster, but the number of clusters is usually low enough that this is not a major consideration. Cluster merging operations can be performed during processing of the input data set if desired, but more stringent criteria may be used to determine whether clusters should be merged. For example, one would not expect the amount of data left to be processed to cause a merged cluster to later fail to meet the merge criteria.

[0061] Figure 3 An example merge analysis process 300 according to the present disclosure is shown. At 310, a set of cluster centers is analyzed, such as determining distances between pairs of clusters in the set (where distances may be calculated as described in Example 3). At 320, it is determined whether any pairs of clusters are within a threshold distance of each other. If not, the process 300 may end at 370, or it may loop at 310 (such as if the process is part of an ongoing process, such as process 200 of FIG. 2 ).

[0062] If it is determined at 320 that two or more clusters are within a threshold distance of each other, such clusters may be merged at 330. At 340, the center of the merged cluster may be calculated. In some cases, the center of the merged cluster is calculated as the average of the centers of the merged clusters. In other cases, the clusters may be weighted, such as based on the number of frames in the cluster or the number of samples / observations in the cluster. If sample data is available, the sample mean vector of the merged clusters may be used to calculate the center of the merged cluster.

[0063] The frames / lists (or other indicators of observations falling within the merged clusters) are merged at 350. The original clusters being merged are deleted and the new merged cluster is added at 360. Process 300 may then loop to 310, including determining whether the cluster created by the merge operation may be further merged with additional clusters.

[0064] Example 5 - Example Calculations in Clustering Using Observation Summary Information

[0065] Figure 4 Specific examples of how process 200 may be implemented are provided. Figure 4 , two clusters have been identified, with the first cluster having a center as defined at row 404, and having two sample mean vectors 408, 410 for respective first and second samples. The first sample is "full", while the second sample has a single observation 414. The second cluster has a center defined at row 418.

[0066] The next observation 422 is received. The distances are calculated between observation 422 and the mean vector 404 of the first cluster and the mean vector 418 of the second cluster, and the results are shown at rows 426, 428. It can be seen that observation 422 is closer to the first cluster 404 than the second cluster 418. Assume that a distance has been set, where an observation will be added to its closest cluster as long as its distance is less than six. In this case, the distance between observation 422 and the first cluster 404 is 4.8, which meets the threshold. Therefore, observation 422 is added to the first cluster, and in particular, is added to the second sample 410 (assuming that a sample can have a maximum of thirty observations, the second sample with only a single observation can accept an additional observation).

[0067] After adding observation 422 to second sample 410, the mean vector for the second sample is recomputed by taking the average of the vector elements of the first observation and the second observation on an element-by-element basis to provide an updated sample mean vector shown at row 432. The updated center of the first cluster can then be recomputed by taking the average of the sample mean vector of the first sample and the updated sample mean vector of the second sample to provide an updated center (centroid) mean vector of the first cluster 436. It can be seen that the centroid of the first cluster has moved slightly as a result of adding observation 422 to the cluster.

[0068] Example 6 - Example cluster information

[0069] Figure 5 Cluster information that may be generated using the disclosed techniques (such as using process 200 of FIG. 2 ) is shown. Figure 5 Three clusters 505, 510, 515 are included, which may correspond to different actors or actresses in the movie. Each cluster 505, 510, 515 includes an identifier 520, a list of frames 525 identified as belonging to the cluster (and used to define the cluster), a vector 530 defining the centroid of the cluster, and a set of sample mean vectors 535.

[0070] Example 7 - Example relationships between observations, samples, and clusters

[0071] Figure 6 A diagram further illustrating the relationship between observations, samples, and clusters is provided. It can be seen that a plurality of observation vectors 610 are included in the corresponding samples (S1, S m ) 615 and used to form sample mean vector 620. As already described, once a sample reaches a determined number of observations, the data of observation vector 610 can be discarded and sample mean vector 620 can be used for operations of their respective clusters.

[0072] Figure 6Also shown is how sample mean vectors 620 may be used to define clusters 625, including mean vectors 630 defining the centers (centroids) of the clusters.

[0073] Example 8 - Example code implementation

[0074] Fig. 7A and 7B Example code 700 is provided that can be used to implement the disclosed technology.

[0075] Example 9 - Example Clustering Operation

[0076] Figure 8 800 is a flow chart of a clustering method 800 according to the present disclosure. The method includes an operation to reduce memory consumption during feature clustering. At 805, a first observation for an input data set is received. The first observation instance includes a first plurality of values ​​of a corresponding plurality of elements of the input data set. The first plurality of values ​​of the plurality of elements describe a feature of interest in the input data set.

[0077] A distance between a first observation and a first cluster of samples is determined at 810, wherein the sample has a location determined using the observation classified within the first cluster. At 815, it is determined that the distance satisfies a threshold. Based on determining that the distance satisfies the threshold, at 820, the first observation is added to a first sample associated with the first cluster.

[0078] At 825, for the first sample, a first set of means for the plurality of elements is calculated using a first plurality of values ​​of the first observation and at least a second plurality of values ​​of the plurality of elements of at least a second observation in the first sample. At 830, a second set of means for the plurality of elements of the cluster is calculated using the first set of means for the first cluster and at least a third set of means for at least a second sample associated with the first cluster, wherein using the samples as summary information for individual observations classified within the first cluster reduces memory consumption compared to calculating the second set of means using data for discrete observations classified within the first cluster. Based on determining that the distance satisfies the first threshold, at 835, an identifier of the first observation is added to a set of observation identifiers classified within the first cluster.

[0079] Example 10 - Additional Example

[0080] Example 1 is a computing system comprising at least one memory and at least one hardware processor coupled to the at least one memory. The computing system also includes one or more computer-readable storage media storing computer-executable instructions, which when executed cause the computing system to perform various operations, wherein the operations reduce memory consumption during feature clustering. The operations include receiving a first observation for an input data set. The first observation includes a first plurality of values ​​for a corresponding plurality of elements of the input data set. The first plurality of values ​​for the plurality of elements describe a feature of interest in the input data set.

[0081] A distance between the first observation and a first cluster of samples having a location determined using the observations classified within the first cluster is determined. The distance is determined to satisfy a first threshold. Based on determining that the distance satisfies the first threshold, the first observation is added to a first sample associated with the first cluster.

[0082] For the first sample, a first set of means for the plurality of elements is calculated using a first plurality of values ​​of the first observation and at least a second plurality of values ​​of the plurality of elements of at least a second observation in the first sample. At 830, a second set of means for the plurality of elements of the cluster is calculated using the first set of means for the first cluster and at least a third set of means for at least a second sample associated with the first cluster, wherein using the samples as summary information for individual observations classified within the first cluster reduces memory consumption compared to calculating the second set of means using data for discrete observations classified within the first cluster. Based on determining that the distance satisfies the first threshold, an identifier of the first observation is added to a set of observation identifiers classified within the first cluster.

[0083] Example 2 includes the subject matter of Example 1, and further specifies that observation location data is not maintained for observations associated with at least the second sample.

[0084] Example 3 includes the subject matter of Example 1 or Example 2, and further specifies additional operations. The additional operations include: receiving a third observation including a third plurality of values ​​of a plurality of elements. Determining a distance between the third observation and the first cluster. Determining that the distance between the third observation and the first cluster satisfies a first threshold. Determining that the first mean value set includes the number of observations that satisfy a second threshold. Based on determining that the first mean value set includes the number of observations that satisfy the second threshold, creating a third mean value set for a third sample associated with the first cluster. The mean value set for the plurality of elements of the third sample is set to a third plurality of values. Using the first mean value set, the second mean value set, and the third mean value set, an updated sample mean value set is calculated for the plurality of elements of the first cluster. Adding an identifier of the third observation to a set of observation identifiers classified in the first cluster.

[0085] Example 4 includes the subject matter of Example 3, and further specifies that the second threshold is selected to provide a desired probability that the first set of means computed from the observations in the first sample represents the observations in the first sample.

[0086] Example 5 includes the subject matter of Example 3 or Example 4, and further specifies that the second threshold is at least thirty observations.

[0087] Example 6 includes the subject matter of either Example 1 or Example 2, and further specifies: determining that adding the first observation to the first sample causes the first sample to meet a threshold number of observations. After calculating the first set of means, deleting data for the observation of the first sample.

[0088] Example 7 includes the subject matter of Example 6, and further specifies: selecting the threshold number of observations to provide a desired probability that a first set of means computed from the observations in the first sample represents the observations in the first sample.

[0089] Example 8 includes the subject matter of Example 7, and further specifies that the threshold number of observations is at least thirty observations.

[0090] Example 9 includes the subject matter of any of Examples 1-8, and further specifies: determining a distance between the first observation and the second cluster. Determining that the distance between the first observation and the first cluster is less than a distance between the first observation and the second cluster.

[0091] Example 10 includes the subject matter of any of Examples 1-9, and further specifies: receiving an image data set. Detecting a face in the image data set. Generating data describing the face from the image data, wherein at least a portion of the data describing the face, or data derived at least in part therefrom, is included in the first observation.

[0092] Example 11 includes the subject matter of Example 10, and further specifies that the set of image data corresponds to samples of a streaming video file.

[0093] Example 12 includes the subject matter of Example 11, and further specifies: extracting the image data set from the streaming video file.

[0094] Example 13 includes the subject matter of any of Examples 1-12, and further specifies: outputting an identifier of, or data associated with, an observation classified in the first cluster.

[0095] Example 14 includes the subject matter of any of Examples 1-13, and further specifies that: the observation classified in the first cluster corresponds to a portion of the video file. A timestamp associated with the observation of the first cluster is output.

[0096] Example 15 includes the subject matter of any of Examples 1-14, and further specifies: determining a distance between the first cluster and the second cluster. Determining that the distance between the first cluster and the second cluster satisfies a threshold. Merging the first cluster and the second cluster.

[0097] Example 16 includes the subject matter of any of Examples 1-15, and further specifies: receiving a third observation including a third plurality of values ​​of the plurality of elements. determining a distance between the third observation and the first cluster. determining that the distance between the third observation and the first cluster does not satisfy a threshold. creating a second cluster. adding the third observation to the second cluster.

[0098] Example 17 includes the subject matter of any of Examples 1-16, and further specifies: the first set of averages includes a vector, wherein a given element of the vector corresponds to an average calculated at least in part based on a corresponding element of the vector for the first observation and a corresponding element of the vector for at least a second observation.

[0099] Example 18 includes the subject matter of any of Examples 1-17, and further specifies: calculating the distance as a Euclidean distance of a vector representing the first observation and a vector representing a center of the first cluster.

[0100] Example 19 includes the subject matter of Example 14, and further specifies: generating at least one time window from output timestamps associated with observations of the first cluster. Receiving a request to watch a portion of a streaming video file corresponding to the at least one time window. In response to the request, sending the portion of the streaming video file to the client.

[0101] Example 20 is one or more computer-readable media storing computer-executable instructions that, when executed, cause a computing system to perform various operations to reduce memory consumption during feature clustering. The operations include: receiving a first observation instance for an input data set. The first observation instance includes a first plurality of values ​​for a corresponding plurality of elements of the input data set. The first plurality of values ​​for the plurality of elements describe a feature of interest in the input data set.

[0102] A distance between the first observation and a first cluster of samples having a location determined using the observations classified within the first cluster is determined. The distance is determined to satisfy a first threshold. Based on determining that the distance satisfies the first threshold, the first observation is added to a first sample associated with the first cluster.

[0103] For a first sample, a first set of means for a plurality of elements is calculated using a first plurality of values ​​of the first observation and at least a second plurality of values ​​of a plurality of elements of at least a second observation in the first sample. At 830, a second set of means for a plurality of elements of a cluster is calculated using the first set of means for a first cluster and at least a third set of means for at least a second sample associated with the first cluster, wherein using samples that are summary information of individual observations classified within the first cluster reduces memory consumption compared to calculating the second set of means using data of discrete observations classified within the first cluster. Based on determining that the distance satisfies a first threshold, an identifier of the first observation is added to a set of observation identifiers classified within the first cluster. Additional examples include the subject matter of Example 20 and the subject matter of any of Examples 2-19 in the form of computer-executable instructions.

[0104] Example 21 is a method that can be implemented in hardware, software, or a combination thereof. The method includes an operation to reduce memory consumption during feature clustering. A first observation instance for an input data set is received. The first observation instance includes a first plurality of values ​​for a corresponding plurality of elements of the input data set. The first plurality of values ​​for the plurality of elements describe a feature of interest in the input data set.

[0105] A distance between the first observation and a first cluster of samples having a location determined using the observations classified within the first cluster is determined. The distance is determined to satisfy a first threshold. Based on determining that the distance satisfies the first threshold, the first observation is added to a first sample associated with the first cluster.

[0106] For a first sample, a first set of means for a plurality of elements is calculated using a first plurality of values ​​of the first observation and at least a second plurality of values ​​of a plurality of elements of at least a second observation in the first sample. At 830, a second set of means for a plurality of elements of the cluster is calculated using the first set of means for the first cluster and at least a third set of means for at least a second sample associated with the first cluster. Based on determining that the distance satisfies a first threshold, an identifier of the first observation is added to a set of observation identifiers classified in the first cluster, wherein using samples as summary information for individual observations classified in the first cluster reduces memory consumption compared to calculating the second set of means using data for discrete observations classified in the first cluster. Additional examples include the subject matter of Example 21 and the subject matter of any of Examples 2-19 in the form of additional elements of the method.

[0107] Example 11 - Calculation System

[0108] Fig. 9 A generalized example of a suitable computing system 900 is depicted in which the described innovations may be implemented. Computing system 900 is not intended to suggest any limitation as to the scope of use or functionality of the present disclosure, as the innovations may be implemented in various general-purpose or special-purpose computing systems.

[0109] refer to Fig. 9 , the computing system 900 includes one or more processing units 910, 915 and memories 920, 925. Fig. 9 , the basic configuration 930 is included within the dashed line. Processing units 910, 915 execute computer executable instructions, such as for implementing the features described in Examples 1-10. The processing unit may be a general-purpose central processing unit (CPU), a processor in an application-specific integrated circuit (ASIC), or any other type of processor. In a multi-processing system, multiple processing units execute computer executable instructions to increase processing power. For example, Fig. 9 A central processing unit 910 and a graphics processing unit or co-processing unit 915 are shown. Tangible memory 920, 925 may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or some combination of both accessible by the processing units 910, 915. The memory 920, 925 stores software 980 implementing one or more innovations described herein in the form of computer-executable instructions suitable for execution by the processing units 910, 915.

[0110] The computing system 900 may have additional features. For example, the computing system 900 includes storage 940, one or more input devices 950, one or more output devices 960, and one or more communication connections 970, including input devices, output devices, and communication connections for interacting with a user. An interconnection mechanism (not shown) such as a bus, controller, or network interconnects the components of the computing system 900. Typically, operating system software (not shown) provides an operating environment for other software executed in the computing system 900 and coordinates the activities of the components of the computing system 900.

[0111] Tangible storage 940 may be removable or non-removable and include disks, magnetic tapes or cartridges, CD-ROMs, DVDs, or any other medium that can be used to store information in a non-transitory manner and that can be accessed within computing system 900. Storage 940 stores instructions for software 980 that implement one or more innovations described herein, including those described in Examples 1-10.

[0112] Input device 950 may be a touch input device such as a keyboard, a mouse, a pen or trackball, a voice input device, a scanning device, or another device that provides input to computing system 900. Output device 960 may be a display, a printer, a speaker, a CD burner, or another device that provides output from computing system 900.

[0113] Communication connection 970 enables communication with another computing entity via a communication medium. The communication medium conveys information such as computer executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal that has one or more of its characteristics set or changed in a manner that encodes the information in the signal. By way of example and not limitation, the communication medium may use electrical, optical, RF or other carrier waves.

[0114] The innovations may be described in the general context of computer executable instructions, such as those included in program modules, which are executed in a computing system on a target real or virtual processor. Typically, program modules or components include routines, programs, libraries, objects, classes, components, data structures, etc. that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules may be combined or split between program modules as desired. Computer executable instructions for program modules may be executed within a local or distributed computing system.

[0115] The terms "system" and "device" are used interchangeably herein. Unless the context clearly indicates otherwise, neither term implies any limitation on the type of computing system or computing device. In general, a computing system or computing device may be local or distributed, and may include any combination of special-purpose hardware and / or general-purpose hardware with software that implements the functionality described herein.

[0116] In various examples described herein, a module (e.g., component or engine) may be "coded" to perform certain operations or provide certain functionality, computer executable instructions indicating the module may be executed to perform such operations, cause such operations to be performed, or otherwise provide such functionality. Although functionality described with respect to a software component, module, or engine may be performed as a discrete software unit (e.g., a procedure, function, class method), it need not be implemented as a discrete unit. That is, functionality may be incorporated into a larger or more general program, such as one or more lines of code in a larger or general program.

[0117] For purposes of presentation, the detailed description uses terms such as "determine" and "use" to describe computer operations in a computing system. These terms are high-level abstractions for operations performed by a computer and should not be confused with actions performed by a human. The actual computer operations corresponding to these terms vary depending on the implementation.

[0118] Example 12 - Cloud computing environment

[0119] Fig.10An example cloud computing environment 1000 is depicted in which the described techniques may be implemented. Cloud computing environment 1000 includes cloud computing services 1010. Cloud computing services 1010 may include various types of cloud computing resources, such as computer servers, data repositories, networking resources, etc. Cloud computing services 1010 may be centrally located (e.g., provided by a data center of an enterprise or organization) or distributed (e.g., provided by various computing resources located in different locations, such as different data centers and / or located in different cities or countries).

[0120] The cloud computing service 1010 is utilized by various types of computing devices (e.g., client computing devices), such as computing devices 1020, 1022, and 1024. For example, the computing devices (e.g., 1020, 1022, and 1024) may be computers (e.g., desktop or laptop computers), mobile devices (e.g., tablet computers or smart phones), or other types of computing devices. For example, the computing devices (e.g., 1020, 1022, and 1024) may utilize the cloud computing service 1010 to perform computing operations (e.g., data processing, data storage, etc.).

[0121] Example 13 - Implementation

[0122] Although the operations of some disclosed methods are described in a particular order for ease of presentation, it should be understood that this description encompasses rearrangement unless the specific language set forth herein requires a particular ordering. For example, in some cases, the operations described in sequence may be rearranged or performed simultaneously. In addition, for simplicity, the accompanying drawings may not illustrate the various ways in which the disclosed methods may be used in conjunction with other methods.

[0123] Any disclosed method may be implemented as computer executable instructions or computer program products stored on one or more computer readable storage media and executed on a computing device (e.g., any available computing device, including a smartphone or other mobile device that includes computing hardware). Tangible computer readable storage media are any available tangible media that can be accessed within a computing environment (e.g., one or more optical media disks such as DVDs or CDs, volatile memory components such as DRAM or SRAM, or non-volatile memory components such as flash memory or hard drives). As an example and with reference to Fig. 9 , computer-readable storage media include memories 920 and 925 and storage device 940. The term computer-readable storage medium does not include signals and carrier waves. In addition, the term computer-readable storage medium does not include communication connections (eg, 970).

[0124] Any of the computer executable instructions for implementing the disclosed technology and any data created and used during the implementation of the disclosed embodiments can be stored on one or more computer readable storage media. The computer executable instructions can be, for example, a dedicated software application or a part of a software application that is accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software can be executed, for example, on a single local computer (e.g., any suitable commercially available computer) or in a network environment (e.g., via the Internet, a wide area network, a local area network, a client-server network (such as a cloud computing network or other such network)) using one or more network computers.

[0125] For the sake of clarity, only certain selected aspects of software-based implementations are described. It should be understood that the disclosed technology is not limited to any particular computer language or program. For example, the disclosed technology can be implemented by software written in C++, Java, Perl, JavaScript, Python, Ruby, ABAP, SQL, Adobe Flash, or any other suitable programming language, or in some examples, by a markup language such as html or XML, or a combination of suitable programming languages ​​and markup languages. Likewise, the disclosed technology is not limited to any particular computer or particular type of hardware.

[0126] In addition, any software-based embodiment (including, for example, computer-executable instructions for causing a computer to perform any disclosed method) can be uploaded, downloaded, or remotely accessed via suitable communication means. Such suitable communication means include, for example, the Internet, the World Wide Web, an intranet, a software application, cables (including fiber optic cables), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such communication means.

[0127] The disclosed methods, apparatus, and systems should not be construed as limiting in any way. Instead, the present disclosure is directed to all novel and non-obvious features and aspects of the various disclosed embodiments, individually and in various combinations and sub-combinations with each other. The disclosed methods, apparatus, and systems are not limited to any particular aspect or feature or combination thereof, nor do the disclosed embodiments require the presence of any one or more specific advantages or solutions to problems.

[0128] The technology from any example can be combined with the technology described in any one or more of the other examples. In view of the many possible embodiments to which the principles of the disclosed technology can be applied, it should be recognized that the illustrated embodiments are examples of the disclosed technology and should not be considered as limiting the scope of the disclosed technology. On the contrary, the scope of the disclosed technology includes what is covered by the scope and spirit of the appended claims.

Claims

1. A computing system comprising: at least one hardware processor; at least one memory coupled to the at least one hardware processor; as well as One or more computer-readable storage media including computer-executable instructions that, when executed, cause the computing system to perform operations for reducing memory consumption during feature clustering, the operations comprising: receiving a first observation for an input data set, the first observation comprising a first plurality of values ​​for a respective plurality of elements of the input data set, the first plurality of values ​​for the plurality of elements describing features of interest in the input data set; determining a distance between the first observation and a first cluster of samples having locations determined using observations classified within the first cluster; Determining that the distance satisfies a first threshold; adding the first observation to a first sample associated with the first cluster based on the determining that the distance satisfies the first threshold; for the first sample, calculating a first set of means for the plurality of elements using the first plurality of values ​​for the first observation and at least a second plurality of values ​​for the plurality of elements for at least a second observation in the first sample; calculating a second set of means for the plurality of elements of the first cluster using the first set of means for the first samples and at least a third set of means for at least second samples associated with the first cluster, wherein using samples that are summary information of individual observations classified within the first cluster reduces memory consumption compared to calculating the second set of means using data of discrete observations classified within the first cluster; and Based on the determining that the distance satisfies the first threshold, an identifier of the first observation is added to a set of observation identifiers classified in the first cluster.

2. The computing system of claim 1, wherein: No observation position data is maintained for observations associated with the second set of mean values.

3. The computing system of claim 1 or claim 2, wherein the operations further comprise: receiving a third observation comprising a third plurality of values ​​of the plurality of elements; determining a distance between the third observation and the first cluster; determining that the distance between the third observation and the first cluster satisfies the first threshold; determining that the first set of means includes a number of observations that satisfy a second threshold; creating a third set of means for a third sample associated with the first cluster based on the determining that the first set of means includes a number of observations that satisfy the second threshold; setting a set of average values ​​of the plurality of elements of the third sample to include the third plurality of values; computing an updated set of means for the plurality of elements of the first cluster using the first set of means, the second set of means, and the third set of means; as well as An identifier of the third observation is added to a set of observation identifiers classified in the first cluster.

4. The computing system of claim 3, wherein: The second threshold is selected to provide a desired probability that the first set of means computed from observations in the first sample represents the observations in the first sample.

5. The computing system of claim 1 or claim 2, wherein the operations further comprise: determining that adding the first observation to the first sample causes the first sample to satisfy a threshold number of observations; as well as After calculating the first set of means, the observed data of the first sample is deleted.

6. The computing system of claim 5, wherein: The threshold number of observations is selected to provide a desired probability that the first set of means computed from observations in the first sample represents the observations in the first sample.

7. The computing system of any one of claims 1 to 6, wherein the operations further comprise: determining a distance between the first observation and a second cluster; as well as It is determined that a distance between the first observation and the first cluster is less than a distance between the first observation and the second cluster.

8. The computing system of any one of claims 1-7, wherein the operations further comprise: receiving an image data set; detecting faces in the image dataset; as well as Data describing the face is generated from the image data, wherein at least part of the data describing the face, or data at least partly derived therefrom, is comprised in the first observation.

9. The computing system of claim 8, wherein: The image data set corresponds to samples of a streaming video file.

10. The computing system of claim 9, the operations further comprising: The image data set is extracted from the streaming video file.

11. The computing system of any one of claims 1-10, wherein the operations further comprise: Output is an identifier of or data associated with an observation classified in the first cluster.

12. The computing system according to any one of claims 1 to 11, wherein: The observations classified in the first cluster correspond to a portion of a video file, the operations further comprising: Output the timestamp associated with the observations of the first cluster.

13. The computing system of claim 12, the operations further comprising: generating at least one time window from output timestamps associated with observations of the first cluster; receiving a request to view a portion of the streaming video file corresponding to the at least one time window; as well as In response to the request, the portion of the streaming video file is sent to the client.

14. The computing system of any one of claims 1-13, the operations further comprising: determining a distance between the first cluster and the second cluster; as well as Determining that the distance between the first cluster and the second cluster satisfies a threshold; as well as The first cluster and the second cluster are merged.

15. The computing system of any one of claims 1-14, the operations further comprising: receiving a third observation comprising a third plurality of values ​​of the plurality of elements; determining a distance between the third observation and the first cluster; determining that the distance between the third observation and the first cluster does not satisfy a threshold; Create a second cluster; as well as The third observation is added to the second cluster.

Images