Systems and methods for identifying recurring objects of interest from a plurality of images

Abstract

A multisensor processing platform includes, in at least some embodiments, a face detector and embedding network for analyzing unstructured data to detect, identify and track any combination of objects (including people) or activities through computer vision algorithms and machine learning. In some embodiments, the unstructured data is compressed by identifying the appearance of an object across a series of frames of the data, aggregating those appearances and effectively summarizing those appearances of the object by one or more representative images and associated embeddings. Further, given a collection of visual data, an aspect of the present system and methods enables specifying a pattern of conduct or activity and having the system automatically identify objects which match the specified pattern even though those objects may not be of interest but for their match to the pattern.

Description

Systems and Methods for Identifying Recurring Objects of Interest From A Plurality of DetectionsSPECIFICATIONRELATED APPLICATIONS

[0001] This application is a PCT conversion and claims the benefit of U.S. Patent Application Serial Number 63 / 502,068, filed May 12, 2023, incorporated herein by reference. The following applications and patents are also incorporated by reference:U.S. Patent Application SN 17 / 866,396 filed 8 / 16 / 2022,U.S. Patent Application S.N. 16 / 120,128 filed August 31 , 2018,U.S. Patent Application S.N. 62 / 553,725 filed September 1 , 2017.FIELD OF THE INVENTION

[0002] The present invention relates generally to computer vision systems configured for object recognition and more particularly relates to computer vision systems capable of identifying an object from a volume of multisensory unstructured data such as audio, video, still frame imagery, or other identifying data, either in near real time or from stored data, in order to assist in contextualizing and exploiting elements of the data relevant to further analysis, such as by compressing identifying data to facilitate a system or user to rapidly distinguish an object of interest from other similar and dissimilar objects. An embodiment relates generally to an object recognition system and, more specifically, to identifying faces of one or more individuals from a volume of video footage or other sensory data collected at different times from either a single monitored location or from a plurality of disparate locations, while other embodiments relate to identification of animate and / or inanimate objects from similar types of data and locations. Additionally, in some embodiments, a pattern filter is applied to aid in the recognition of objects of interest that appear at a given location recurrently, or appear at multiple monitored locations.BACKGROUND OF THE INVENTION

[0003] Conventional computer vision and machine learning systems are configured to identify objects, including people, cars, trucks, etc., by providing to those systems a quantity of training images that are evaluated in a neural network, for example by a convolutional neural network such as shown in Figure 1 . In the absence of such training images, these conventional systems are typically unable to identify the object of interest. In many situations it remains desirable to identify an individual or other object even if there is no picture or similar training image that enables the computer vision system to distinguish the object of interest from other objects having somewhat similar characteristics or features. For example, an observer who has seen an event, such as a person shoplifting, can probably identify the shoplifter if shown a picture, but the shoplifter’s face is just one of many images contained in the video footage of the store’s security system and there is no conventional way to extract the shoplifter’s image from those hundreds or thousands of faces. Conventionally, given the dearth of better data, a sketch artist or a modern digital equivalent would be asked to create a composite image that resembles the suspect. However, this process is time consuming and, often, far from accurate.

[0004] Many, if not most, conventional object identification systems that employ computer vision attempt facial recognition where the objects of interest are people. Most such conventional systems have attempted to identify faces of people in the video feed by clustering images of the object, such that each face or individual in a sequence of video footage is represented by selecting a single picture from that footage. While conventional systems implement various embedding approaches, the approach of selecting a single picture typically results in systems that are highly inaccurate because they are typically incapable of selecting an optimal image when the face or individual appears multiple times throughout the video data, but with slight variations in head or body angle, position, lighting, shadowing, etc. Further, such conventional systems typically require significant time to process the volume of images of faces or other objects that may appear in a block of video footage, such as when those faces number in the thousands.

[0005] Another challenge faced by conventional facial recognition systems using conventional embedding techniques is the difficulty of mapping all images of the same person or face to exactly the same point in a multidimensional space by conventional facial recognition systems. Additionally, conventional systems operate under the assumption that embeddings from images of the same person are closer to each other than to any embedding of a different person. In reality, there exists a small chance that embeddings of two different people are much closer than two embeddings of the same person, which conventional facial recognition systems fail to account for. In such instances, conventional systems can generate false positives that lead to erroneous conclusions.

[0006] The foregoing issues become even more challenging for conventional objects when the objective is to detect recurrences of objects where those recurrences are disparate either temporally or spatially.

[0007] The result is that there has been a long felt need for a system that can synthesize accurately a representation of a face or other object by extracting relevant data from video footage, still frame imagery, or other data feed.SUMMARY OF THE INVENTION

[0008] The present invention is a multisensor processing platform for detecting, identifying and tracking any of entities, objects and activities, or combinations thereof through computer vision algorithms and machine learning. The multisensor data can comprise various types of unstructured data, for example, full motion video, still frame imagery, InfraRed sensor data, communication signals, geospatial imagery data, etc. Entities can include faces and their identities, as well as various types of objects such as vehicles, backpacks, weapons, etc. Activities can include correlation of objects, persons and activities, such as packages being exchanged, two people meeting, presence of weapons, vehicles and their operators, etc. In some embodiments, the invention allows human analysts to contextualize their understanding of the multisensor data. For multisensor data flowing in real time, the invention permits such analysis at near real time speed and scale and allows the exploitation of elements of the data that are relevant to their analysis. Embodimentsof the system are designed to strengthen the perception of an operator through supervised, semi-supervised and unsupervised learning in an integrated intuitive workflow that is constantly learning and improving the precision and recall of the system.

[0009] In at least some embodiments, the multisensor processing platform comprises a face detector and an embedding network. In an embodiment, the face detector generates cropped bounding boxes around detected faces. The platform comprises in part one or more neural networks configured to perform various of the functions of the platform. Depending upon the implementation and the particular function to be performed, the associated neural network can be fully connected, convolutional or other forms as described in the referenced patent applications.

[0010] As is characteristic of neural networks, in some embodiments a training process precedes a recognition process. The training step typically involves the use of a training dataset to estimate parameters of a neural network to extract a feature embedding vector for any given image. The resulting universe of embeddings describes a multidimensional space that serves as a reference for comparison during the recognition process.

[0011] The present invention comprises a plurality of different major aspects, each of which implements the multisensor processing platform albeit with somewhat different functionality. Each has as one of its aspects the ability to provide a user with a representative image that effectively summarizes the appearance of a person or object of interest in a plurality of frames of imagery and thus enables a user to make an “at a glance” assessment of the result of a search. In the first major aspect of the invention, the objective is to identify appearances of a known person or persons of interest within unstructured data such as video footage, where the user generating the query has one or more images of the person or persons. In an embodiment, the neural network of the multisensor processing platform has been trained on a high volume of faces using a conventional training dataset. The facial images within each frame of video are input to the embedding network to produce a feature vector, for example a 128-dimensional vector of unit length. In an embodiment, the embedding network is trained to map facial images of the same individual to a common individual map, for example the same 128-dimension vector. The embedding can beimplemented using deep neural networks, among other techniques. Through the use of deep neural networks trained with gradient descent, such an embedding network is continuous and implements differentiable mapping from image space (e.g. 160x160x3 tensors) to, in this case, S127, i.e. the unit sphere embedded in 128-dimensional space.

[0012] To elaborate, the recognition phase is, in an embodiment, implemented based on one shot or low shot learning depending upon whether the user has a single image of a person of interest such as a driver’s license photo or, for greater accuracy, a collection of images of face of the person of interest as a probe image or images. The embedding resulting from processing that image or collection of images enables the system to identify faces that match the person of interest from the gallery of faces in the video footage or other data source. The user’s query can be expressed as a Boolean equation or other logical expression, and seeks detection and identification of a specified combination of objects, entities and activities as described above. The query is thus framed in terms of fixed identities, essentially “Find Person A” or “Find Persons A and B” or “Find Persons A and B performing activity C”. On a frame-by-frame basis, each face in the frame is evaluated to determine the likelihood that it is one of the identities in the set {Person A, Person B}. A confidence histogram analysis of pair-wise joint detections of identities can be employed in some embodiments to evaluate the likelihood of any pair of identities being connected. In an embodiment, a linear assignment is used to match the face most likely to be Person A and the face most likely to be Person B.

[0013] In a second major aspect of the invention, there is no prior image of the person of interest, who may be known only from an observer’s recollection or some other general description, and the objective is to permit a large volume of video footage to be rapidly and accurately summarized, or compressed, in a way that permits either automatic or human browsing of the detected faces so as to identify those detected faces that meet the general description without requiring review of each and every frame. The resulting time savings has the added benefit of increased accuracy, in part due to the fatigue that typically besets a human reviewer after extensive maual review of images.

[0014] In this second major aspect faces are identified in a first frame of a data sequence such as video footage, and those images serve as the reference fordetecting in a second frame the same faces found in the first frame. The first and second images together serve as references for detecting faces in the third frame, and so on until either the sequence of footage ends or the individual exits the video footage. The collection of images, represented in the platform by their embeddings and sometimes referred to as a tracklet herein, permits the selection of an image for each detected face that is the most representative of their nonvariant features in that entire sequence. Thus, instead of being required to review each and every frame of an entire video sequence, an operator or automated system needs only to scan the representative embeddings. Thus the unstructured data of a video feed that captures a multitude of faces can be compressed into a readily manageable set of thumbnails with substantial savings in time and, potentially, storage space.

[0015] Because of the variation in appearance that can occur when an individual travels through the field of view of a camera or other data collector, it is possible that in some embodiments the same person will not be perceived as identical across a series of frames. Thus, one person’s face might result in a plurality of tracklets, each with its own representative image. Some of these different representative images are labeled as “key faces” and are grouped together for further processing and resolution. Such a grouping approach is particularly helpful in embodiments where avoiding false positives is a higher priority than avoiding false negatives. The selection of specific representative images as key faces depends at least in part upon the thresholds or tolerances chosen for clustering, and can vary with the specific embodiment or application.

[0016] As with the first major aspect of the invention, linear assignment techniques are implemented to determine levels of confidence that a face in a first frame is the same as the face in a second frame, and so on. Further, conditional probability distribution functions of embedding distance can be used to validate the identity of a face detected in a second or later frame as being the same (or different) as a face detected in an earlier frame. Even with multiple key faces, the present invention provides an effective compression of a large volume of unstructured video data into a series of representative images that can be reviewed and analyzed far more quickly and efficiently than possible with the prior art approaches.

[0017] In some applications, reducing the data to a more easily manageable volume - i.e., greater data compression - is more useful than ensuring accuracy, while in other applications greater accuracy is more important than reduced volume. The tradeoff between accuracy and compression can be represented as probability distributions, and the desired balance between the two represented as a line as described in greater detail hereinafter.

[0018] In some embodiments, color is also important. In such cases, a color histogram in a convenient color space such as Cl ELAB is extracted from the image. If better generalization is desired, the histogram is blurred which in turn permits matching to nearby colors as well. A Gaussian distribution around the query color can also be used to better achieve a match.

[0019] In a further aspect of the invention, instances of objects appearing either at the same location at different times, or at different locations, can be detected and identified as patterns that permit further analysis and action even though those objects may not be of interest except for their being surfaced in response to a supplied pattern.

[0020] In some embodiments, reporting results to an operator in a curated manner can greatly simplify an operator’s review of the data generated by the facial recognition aspects of the present invention. In such embodiments, localized clustering, layout optimization, highlighting, dimming, or blurring, and other similar techniques can all be used to facilitate more rapid assessments without unduly sacrificing accuracy.

[0021] It is one object of the present invention to provide a system, method and device by which large volumes of unstructured data can be sorted and inspected, and animate or inanimate objects can be found and tabulated.

[0022] It is a further object of the present invention to develop an assessment of objects based on invariant features.

[0023] It is another object of the present invention to identify matches to a probe image through the use of per-frame analysis together with Boolean or similar querying.

[0024] It is a further object of the present invention to detect faces within each frame of a block of video footage or other sensor data collected over time,

[0025] A still further object of the present invention is to assign a representative image to a face detected in a sequence of frames where the representative image is either one of the face captures of an individual or a composite of a plurality of face captures of that individual.

[0026] Yet a further object of the present invention is to group faces identified as the same person in a plurality of frames, choose a single image from those faces, and present that single image as representative of that person in that plurality of frames.

[0027] Another object of the present invention is to facilitate easy analysis of a video stream by representing as a tracklet the locations of an individual in a series of video frames.

[0028] Still another object of the invention is to provide to a user a representative image of each of at least a plurality of the individuals captured in a sequence of images whereby the user can identify persons of interest by browsing the representative images.

[0029] A still further object of the present invention is to provide a summary search report to a user comprising a plurality of representative images arranged by level of confidence in the accuracy of the search results.

[0030] Yet another object of the invention is to provide search results where certain search results are emphasized relative to other search results by selective highlighting, blurring or dimming.

[0031] Another object of the invention is to detect occurrences of one or more objects at different times at a location.

[0032] A further object of the invention is to detect occurrences of one or more objects at different locations and assign similarity scores to the various detections.

[0033] Yet another object of the invention is to assign confidence values to various detections based at least in part on similarity scores.

[0034] A still further object of the invention is to detect patterns in the occurrence of one or more objects at different locations.

[0035] Another object of the invention is to detect patterns in the recurrence of one or more objects at substantially the same location at different times.

[0036] A still further object of the invention is to provide a discovery-by- pattern capability by which identities which would otherwise remain unknown are surfaced as the result of their pattern either spatially or temporally.

[0037] These and other objects of the invention can be better appreciated from the following Detailed Description of the Invention, taken together with the appended Figures briefly described below.THE FIGURES

[0038] Figure 1 [Prior Art] describes a convolutional neural network typical of the prior art.

[0039] Figure 2A shows in generalized block diagram form an embodiment of the overall system as a whole comprising the various inventions disclosed herein.

[0040] Figure 2B illustrates in circuit block diagram form an embodiment of a system suited to host a neural network and perform the various processes of the inventions described herein.

[0041] Figure 2C illustrates in generalized flow diagram form the processes comprising an embodiment of the invention.

[0042] Figure 2D illustrates an approach for distinguishing a face from background imagery in accordance with an aspect of the invention.

[0043] Figure 3A illustrates a single frame of a video sequence comprising multiple frames, and the division of that frame into segments where a face snippet is formed by placing a bounding box placed around the face of an individual appearing in a segment of a frame.

[0044] Figure 3B illustrates in flow diagram form the overall process of retrieving a video sequence, dividing the sequence into frames and segmenting each frame of the video sequence.

[0045] Figure 4 illustrates in generalized flow diagram form the process of analyzing a face snippet in a first neural network to develop an embedding, followed by further processing and classification.

[0046] Figure 5A illustrates a process for evaluating a query in accordance with an embodiment of an aspect of the invention.

[0047] Figure 5B illustrates an example of a query expressed in Boolean logic.

[0048] Figure 6 illustrates a process in accordance with an embodiment of the invention for detecting faces or other objects in response to a query.

[0049] Figure 7 A illustrates a process in accordance with an embodiment of the invention for creating tracklets for summarizing detection of a person of interest in a sequence of frames of unstructured data such as video footage.

[0050] Figure 7B illustrates how the process of Figure 7A can result in grouping tracklets according to confidence level.

[0051] Figure 8 is a graph of two probability distribution curves that depict how a balance between accuracy and data compression can be selected based on embedding distances, where the balance, and thus the confidence level associated with a detection or a series of detections, can be varied depending upon the application or the implementation.

[0052] Figure 9 illustrates a process in accordance with an aspect of the invention for determining a confidence metric that two or more individuals are acting together.

[0053] Figure 10 illustrates the detection of a combination of faces and objects in accordance with an embodiment of an aspect of the invention.

[0054] Figure 11 illustrates in generalized flow diagram form an embodiment of the second aspect of the invention.

[0055] Figure 12 illustrates a process in accordance with an embodiment of an aspect of the invention for developing tracklets representing a record of an individual or object throughout a sequence of video frames, where an embedding is developing for each frame in which the individual or object of interest is detected.

[0056] Figure 13 illustrates a process for determining a representative embedding from the tracklets various embeddings.

[0057] Figures 14A-14B illustrate a layout optimization technique for organizing tracklets on a grid in accordance with an embodiment of the invention.

[0058] Figure 15A illustrates a simplified view of clustering in accordance with an aspect of the invention.

[0059] Figure 15B illustrates in flowchart form an exemplary embodiment for localized clustering of tracklets in accordance with an embodiment of the invention.

[0060] Figure 15C illustrates a visualization of the clustering process of Figure of Figure 15B.

[0061] Figure 15D illustrates the result of the clustering process depicted in the embodiment of Figures 15B and 15C.

[0062] Figure 16A illustrates a technique for highlighting similar tracklets in accordance with an embodiment of the invention.

[0063] Figures 16B-16C illustrate techniques for using highlighting and dimming as a way of emphasizing tracklets of greater interest in accordance with an embodiment of the invention.

[0064] Figure 17 illustrates a curation and optional feedback technique in accordance with an embodiment of the invention.

[0065] Figure 18A-18C illustrate techniques for incorporating detection of color through the use of histograms derived from a defined color space.

[0066] Figures 19 illustrates a report and feedback interface for providing a system output either to an operator or an automated process for performing further analysis.

[0067] Figures 20A-20D illustrate an embodiment of the invention configured to apply a pattern definition to a volume of data representative of images of one or more monitored spaces.

[0068] Figure 21 illustrates a context-based face orientation detector in accordance with an embodiment of the invention.

[0069] Figure 22 illustrates an embeddings-based recognition method for distinguishing objects from non-objects in accordance with an embodiment of the invention.

[0070] Figures 23A-23B illustrate a tracklet embedding space and the selection of one or more representative embeddings for a given tracklet.

[0071] Figure 24 illustrates the development of clusters from batches of tracklets related to one another in time, spatial area, or related spatial areas in accordance with an embodiment of the invention.

[0072] Figure 25 illustrates a comparison process for calculating similarity scores between a single cluster’s key tracklets and the full corpus of associated tracklets in accordance with an embodiment of the invention.

[0073] Figure 26 illustrates a process for comparing clusters to tracklets in a batch process in accordance with an embodiment of the invention.

[0074] Figure 27 illustrates the development of similarity thresholds in accordance with an embodiment of the invention.

[0075] Figure 28 illustrates a process for comparing sightings with a pattern for extraction of candidates in accordance with an embodiment of the invention.

[0076] Figure 29 illustrates in greater detail an exemplary pattern comparison in accordance with the embodiment of Figure 28.

[0077] Figure 30 illustrates in flow diagram form a de-duplication process in accordance with an embodiment of the invention.

[0078] Figure 31 illustrates an embodiment of a display for review by an operator including one or more candidates, the sightings associated with each, and a link to the associated video, images, or other data source, all in accordance with an embodiment of the invention.DETAILED DESCRIPTION OF THE INVENTION

[0079] As discussed briefly above, the present invention comprises a platform for quickly analyzing the content of a large amount of unstructured data, as well as executing queries directed to the content regarding the presence and location of various types of entities, inanimate objects, and activities captured in the content. For example, in full motion video, an analyst might want to know if a particular individual is captured in the data and if so the relationship to others that may also be present. An aspect of the invention is the ability to detect and recognize persons, objects and activities of interest using multisensor data in the same model substantially in real time with intuitive learning.

[0080] Viewed from a high level, the platform of the present invention comprises an object detection system which in turn comprises an object detector andan embedding network. The object detector is trainable to detect any class of objects, such as faces as well as inanimate objects such cars, backpacks, and so on.

[0081] Drilling down, an embodiment of the platform comprises the following major components: a chain of processing units, a data saver, data storage, a reasoning engine, web services, report generation, and a User Interface. The processing units comprise a face detector, an object detector, an embedding extractor, clustering, an encoder, and person network discovery. In an embodiment, the face detector generates cropped bounding boxes around faces in an image such as a frame, or a segment of a frame, of video. In some such embodiments, video data supplemented with the generated bounding boxes may be presented for review to an operator or a processor-based algorithm for further review, such as to remove random or false positive bounding boxes, add bounding boxes around missed faces, or a combination thereof. It will be appreciated by those skilled in the art that the term “segment” is used herein in two different contexts, with a different meaning depending upon the context. As noted above, a frame can be divided into multiple pieces, or segments. However, as discussed in connection with Figures 6A-6B et seq., a sequence of video data is sometimes described as a segment.

[0082] As noted above, in an embodiment the facial images within each frame are inputted to the embedding network to produce a feature vectorforeach such facial image, for example a 128-dimensional vector of unit length. The embedding network is trained to map facial images of the same individual to a common individual map, for example the same 128-dimension vector. Because of how deep neural networks are trained if the training involves the use of gradient descent, such an embedding network is a continuous and differentiable mapping from image space (e.g. 160x160x3 tensors) to, in this case, S127, i.e. the unit sphere embedded in 128- dimensional space. Accordingly, the difficulty of mapping all images of the same person to exactly the same point is a significant challenge experienced by conventional facial recognition systems.

[0083] Although there are several major aspects to the present invention, these aspects share a common origin in the multisensor processing system and many of the functionalities extant in that system. Thus, the platform and its functionalitiesare discussed first hereinafter, followed by a discussion of the various major aspects as described briefly in the Summary of the Invention, above.

[0084] Referring first to Figure 2A, shown therein is a generalized view of an embodiment of a system 100 and its processes comprising the various inventions as described hereinafter. The system 100 can be appreciated in the whole. The system 100 comprises a user device 105 having a user interface 110. A user of the system communicates with a multisensor processor 115 either directly or through a network connection which can be a local network, the internet, a private cloud or any other suitable network. The multisensory processor, described in greater detail in connection with Figure 2B, receives input from and communicates instructions to a sensor assembly 125 which further comprises sensors 125A-125n. The sensor assembly can also provide sensor input to a data store 130, and in some embodiments can communicate bidirectionally with the data store 130.

[0085] Next with reference to Figure 2B, shown therein in block diagram form is an embodiment of the multisensor processor system or machine 115 suitable for executing the processes and methods of the present invention. In particular, the processor 115 of Figure 2B is a computer system that can read instructions 135 from a machine-readable medium or storage unit 140 into main memory 145 and execute them in one or more processors 150. Instructions 135, which comprise program code or software, cause the machine 115 to perform any one or more of the methodologies discussed herein. In alternative embodiments, the machine 115 operates as a standalone device or may be connected to other machines via a network or other suitable architecture. In a networked deployment, the machine may operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. In some embodiments, system 100 is architected to run on a network, for example, a cloud network (e.g., AWS) or an on-premise data center network.

[0086] The multisensor processor 115 can be a server computer such as maintained on premises or in a cloud network, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a cellular telephone, a smartphone, a web appliance, a network router, switch or bridge, or any machine capable of executing instructions 135 (sequential or otherwise) that specifyactions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute instructions 135 to perform any one or more of the methods or processes discussed herein.

[0087] In at least some embodiments, the multisensor processor 115 comprises one or more processors 150. Each processor of the one or more processors 150 can comprise a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a controller, one or more application specific integrated circuits (ASICs), one or more radio-frequency integrated circuits (RFICs), or any combination of these. In an embodiment, the machine 115 further comprises static memory 155 together with main memory 145, which are configured to communicate with each other via bus 160. The machine 115 can further include one or more visual displays as well as associated interfaces, all indicated at 165, for displaying messages or data. The visual displays may be of any suitable type, such as monitors, head-up displays, windows, projectors, touch enabled devices, and so on. At least some embodiments further comprise an alphanumeric input device 170 such as a keyboard, touchpad or touchscreen or similar, together with a pointing or other cursor control device 175 such as a mouse, a trackball, a joystick, a motion sensor, a touchpad, a tablet, and so on), a storage unit or machine-readable medium 140 wherein the machine-readable instructions 135 are stored, a signal generation device 180 such as a speaker, and a network interface device 185. A user device interface 190 communicates bidirectionally with user devices 120 (Figure 2A). In an embodiment, all of the foregoing are configured to communicate via the bus 160, which can further comprise a plurality of buses, including specialized buses, depending upon the particular implementation.

[0088] Although shown in Figure 2B as residing in storage unit or machine- readable medium 140, instructions 135 (e.g., software) for causing the execution of any of the one or more of the methodologies, processes or functions described herein can also reside, completely or at least partially, within the main memory 145 or within the processor 150 (e.g., within a processor’s cache memory) during execution thereof by the multisensor processor 115. In at least some embodiments, main memory 145 and processor 150 also can comprise, in part, machine-readable media. Theinstructions 135 (e.g., software) can also be transmitted or received over a network 120 via the network interface device 185.

[0089] While machine-readable medium or storage device 140 is shown in an example embodiment to be a single medium, the term “machine-readable medium” should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) able to store instructions (e.g., instructions 135). The term “machine-readable medium” includes any medium that is capable of storing instructions (e.g., instructions 135) for execution by the machine and that cause the machine to perform any one or more of the methodologies disclosed herein. The term “machine-readable medium” includes, but is not limited to, data repositories in the form of solid-state memories, optical media, and magnetic media. The storage device 140 can be the same device as data store 130 (Figure 2A) or can be a separate device which communicates with data store 130.

[0090] Figure 2C illustrates, at a high level, an embodiment of the software functionalities implemented in an exemplary system 100 shown generally in Figure 2A, including an embodiment of those functionalities operating in the multisensor processor 115 shown in Figure 2B. Thus, inputs 200A-200n can be video or other sensory input from a drone 200A, from a security camera 200B, a video camera 200C, or any of a wide variety of other input device 200n capable of providing data sufficient to at least assist in identifying an animate or inanimate object. It will be appreciated that combinations of different types of data can be used together for the analysis performed by the system. For example, in some embodiments, still frame imagery can be used in combination with video footage. In other embodiments, a series of still frame images can serve as the gallery. Still further, while organizing the input feed chronologically is perhaps the most common, arranging the input data either by lat / long or landmarks or relative position to other data sources, or numerous other methods, can also be used in the present invention. Further, the multisensor data can comprise live feed or previously recorded data. The data from the sensors 200A- 200n is ingested by the processor 115 through a media analysis module 205. In addition to the software functionalities operating within the multisensor processor 115, described in more detail below, the system of Figure 2C comprises encoders 210 that receive entities (such as faces and / or objects) and activities from the multisensorprocessor 115. Further, a data saver 215 receives raw sensor data from processor115, although in some embodiments raw video data can be compressed using video encoding techniques such as H.264 or H.265. Both the encoders and the data saver provide their respective data to the data store 130 in the form of raw sensor data from data saver 210 and faces, objects, and activities from encoders 205. Where the sensor data is video, the raw sensor data can be compressed in either the encoders or the data saver using video encoding techniques, for example, H.264 & H.265 encoding.

[0091] Where the multisensor data from inputs 200A-200n includes full motion video from terrestrial or other sensors, the processor 115 can, in an embodiment, comprise a face detector 220 chained with a recognition module 225 which comprises an embedding extractor, and an object detector 230. In an embodiment, the face detector 220 and object detector 230 can employ a single shot multibox detector (SSD) network, which is a form of convolutional neural network. SSD’s characteristically perform the tasks of object localization and classification in a single forward pass of the network, using a technique for bounding box regression such that the network both detects objects and also classifies those detected objects. Using, for example, the FaceNet neural network architecture, the face recognition module 225 represents each face with an “embedding”, which is a 128 dimensional vector designed to capture the identity of the face, and to be invariant to nuisance factors such as viewing conditions, the person’s age, glasses, hairstyle, etc. Alternatively, various other architectures, of which SphereFace is one example, can also be used. In embodiments having other types of sensors, other appropriate detectors and recognizers may be used. Machine learning algorithms may be applied to combine results from the various sensor types to improve detection and classification of the objects, e.g., faces or inanimate objects. In an embodiment, the embeddings of the faces and objects comprise at least part of the data saved by the data saver 210 and encoders 205 to the data store 130. The embedding and entities detections, as well as the raw data, can then be made available for querying, which can be performed in near real time or at some later time.

[0092] Queries to the data are initiated by analysts or other users through a user interface 235 which connects bidirectionally to a reasoning engine 240, typicallythrough network 120 (Figure 2A) via a web services interface 245. In an embodiment, the web services interface 245 can also communicate with the modules of the processor 115, typically through a web services external system interface 250. The web services comprise the interface into the back-end system to allow users to interact with the system. In an embodiment, the web services use the Apache web services framework to host services that the user interface can call, although numerous other frameworks are known to those skilled in the art and are acceptable alternatives.

[0093] Queries are processed in the processor 115 by a query process 255. The user interface 235 allows querying of the multisensor data for faces and objects (collectively, entities) and activities. One exemplary query can be “Find all images in the data from multiple sensors where the person in a given photograph appears". Another example might be, “Did John Doe drive into the parking lot in a red car, meet Jane Doe, who handed him a bag”. Alternatively, in an embodiment, a visual GUI can be helpful for constructing queries. The reasoning engine 240, which typically executes in processor 115, takes queries from the user interface via web services and quickly reasons through, or examines, the entity data in data store 130 to determine if there are entities or activities that match the analysis query. In an embodiment, the system geo-correlates the multisensor data to provide a comprehensive visualization of all relevant data in a single model. Once that visualization of the relevant data is complete, a report generator module 260 in the processor 115 saves the results of various queries and generates a report through the report generation step 265. In an embodiment, the report can also include any related analysis or other data that the user has input into the system.

[0094] The data saver 215 receives output from the processing system and saves the data on the data store 130, although in some embodiments the functions may be integrated. In an embodiment, the data from processing is stored in a columnar data storage format such as Parquet that can be loaded by the search backend and searched for specific embeddings or object types quickly. The search data can be stored in the cloud (e.g. AWS S3), on premise using HDFS (Hadoop Distributed File System), NFS, or some other scalable storage. In some embodiments, web services 245 together with user interface (Ul) 235 provide users such as analysts with access to the platform of the invention through a web-based interface. The webbased interface provides a REST API to the Ul. The web based interface, in turn, communicates with the various components with remote procedure calls implemented using Apache Thrift. This allows various components to be written in different languages.

[0095] In an embodiment, the Ul is implemented using React and node.js, and is a fully featured client side application. The Ul retrieves content from the various back-end components via REST calls to web service. The User Interface supports upload and processing of recorded or live data. The User Interface supports generation of query data by examining the recorded or live data. For example, in the case of video, it supports generation of face snippets from uploaded photograph or from live video, to be used for querying. Upon receiving results from the Reasoning Engine via the Web Service, the Ul displays results on a webpage.

[0096] In some embodiments, the Ul allows a human to inspect and confirm results. When confirmed the results can be augmented with the query data as additional examples, which improves accuracy of the system. The Ul augments the raw sensor data with query results. In the case of video, results include keyframe information which indicates - as fractions of the total frame dimensions - the bounding boxes of the detections in each frame that yielded the result. When the corresponding result is selected in the Ul, the video is overlaid by the Ul with visualizations indicating why the algorithms believe the query matches this portion of the video. An important benefit of this aspect of at least some embodiments is that such summary visualizations support “at a glance” verification of the correctness of the result. This ease of verification becomes more important when the query is more complex. Thus, if the query is “Did John drive a red car to meet Jane, who handed him a bag”, a desirable result would be a thumbnail, viewable by the user, that shows John in a red car and receiving an object from Jane. One way of achieving this is to display confidence measures as reported by the Reasoning Engine. Using fractions instead of actual coordinates makes the data independent of the actual video resolution, which makes it easy to provide encodings of the video at various resolutions.

[0097] Continuing the use of video data as an example, in an embodiment the Ul displays a bounding box around each face, creating a face snippet. As the video plays back, the overlay is interpolated from key-frame to key-frame, so that boundingbox information does not need to be transmitted for every frame. This decouples the video (which needs high bandwidth) from the augmentation data (which only needs low bandwidth). This also allows caching the actual video content closer to the client. While the augmentations are query and context specific and subject to change during analysts’ workflow, the video remains the same.

[0098] In some embodiments, certain pre-filtering of face snippets may be performed before face embeddings are extracted. For example, the face snippet can be scaled to a fixed size, typically but not necessarily square, of 160 x 160 pixels. In many instances, the snippet with the individual’s face will also include some pixels from the background, which are not helpful to the embedding extraction. Likewise, it is desirable for the embeddings to be as invariant as possible to rotation or tilting of the face. This is best achieved by emphasizing the true face of the individual, and deemphasizing the background. Since an individual’s face typically occupies a central portion of the face snippet, one approach is to identify, during training, an average best radius which can then be used during run time, or recognition. An alternative approach is to detect landmarks, such as eyes, nose, mouth, ears, using any of the face landmark detection algorithms known to those skilled in the art. Knowledge of the eyes, for example, will allow us to define a more precise radius based upon the eye locations. For example, we might set the radius as R = s * d_e, where d_e is the average distance of each eye from the center of the scaled snippet, and s is a predetermined scaling factor.

[0099] Regardless of the method used to identify background from the actual face, once that is complete, the background is preferably eliminated or at least deemphasized. Referring to Figure 2D, a vignetting or filtering technique used in connection with the aforementioned bounding boxes and face snippets can be better appreciated. In most segments of a video frame, the bounding box that surrounds a detected face includes aspects of the background that are not relevant to the detection. Through a vignetting or filtering technique, that irrelevant data is excised. Thus, bounding box 280A includes a face 285 and background pixels 290A. By applying a vignetting filter or other suitable algorithmic filter, the background pixels 290A are annulled, or “zeroed out”, and bounding box 290A becomes box 290A where face 285 is surrounded by 290B. A separation layer 295, comprising a few pixels forexample, can be provided between the face 285 and the annulled pixels 290A to help ensure that no relevant pixels are lost through the filtering step. The annulled pixels can be the result of any suitable technique, for example being darkened, blurred, or converted to a color easily distinguished from the features of the face or other object. More details of the sequence for isolating the face will be discussed hereinafter in connection with Figure 4.[000100] The video processing platform for recognition of objects within video data provides functionality for analysts to more quickly, accurately, and efficiently assess large amounts of video data than historically possible and thus to enable the analysts to generate reports 265 that permit top decision-makers to have actionable information more promptly. The video processing platform for recognition within video data enables the agent to build a story with his notes and a collection of scenes or video snippets. Each of these along with the notes provided can be organized in any order or time order. The report automatically provides a timeline view or geographical view on a map.[000101] To better understand the operation of the system of the first major aspect of the invention, where the objective is to identify appearances of a known person in unstructured data, and where at least one image of the person of interest is available, consider the example of an instantiation of the multisensor processor system where the multisensor data includes full motion video. In such an instance, the relevant processing modules include the face detector 220, the recognition module 225, the object detector 230, a clustering module 270 and a person network discovery module 275. The instantiation also includes the encoders 210, the data saver 215, the data store 130, the reasoning engine 240, web services 245, and the user interface 235.[000102] In this example, face detection of faces in the full motion video is performed as follows, where the video comprises a sequence of frames and each frame is essentially a still, or static, image or photograph. An object recognition algorithm, for example an SSD detection algorithm as discussed above, is trained on a wide variety of challenging samples for face detection. Using this approach, and with reference to Figures 3A-3C, an embodiment of the face detection method of the present invention processes a frame 300 and detects one or more unidentifiedindividuals 310. The process thereupon produces a list of bounding boxes 320 surrounding faces 330. In an embodiment, the process also develops a detection confidence, and notes the temporal location in the video identifying the frame where each face was found. The spatial location within a given frame can also be noted.[000103] To account for the potential presence of faces that appear small in the context of the entire frame, frames can be cropped into n images, or segments 340, and the face recognition algorithm is then run on each segment 340. The process is broadly defined by Figure 3B, where a video is received at step 345, for example as a live feed from sensor 200C, and then divided into frames as shown at step 350. The frames are then segmented at step 355 into any convenient number of segments, where, for example, the number of segments can be selected based in part on the anticipated size of a face.[000104] In some instances, the face detection algorithm may fail to detect a face because of small size or other inhibiting factors, but the object detector (discussed in greater detail below) identifies the entire person. In such an instance the object detector applies a bounding box around the entire body of that individual, as shown at 360 in Figure 2A. For greater accuracy in such an instance, portions of a segment may be further isolated by selecting a snippet 365, comprising only the face. The face detection algorithm is then run on those snippets.[000105] Again with reference to the system of Figure 2C, in an embodiment object detection is performed using an SSD algorithm in a manner similar to that described above for faces. The object detector 230 can be trained on synthetic data generated by game engines. As with faces, the object detector produces a list of bounding boxes, the class of objects, a detection confidence metric, and a temporal location identifying the frame of video where the detected object was found.[000106] In an embodiment, face recognition as performed by the recognition module 225, or the FRC module, uses a facial recognition algorithm, for example, the FaceNet algorithm, to convert a face snippet into an embedding which essentially captures the true identity of the face while remaining invariant to perturbations of the face arising from variables such as eye-glasses, facial hair, headwear, pose, illumination, facial expression, etc. The output of the face recognizer is, for example, a 128 dimension vector, given a face snippet as input. In at least some embodiments,during training the neural network is trained to classify all training identities. The ground truth classification has a “1” in the ithcoordinate for the ithand 0 in all other coordinates. Other embodiments can use triplet loss or other techniques to train the neural network.[000107] Training from face snippets can be performed by any of a number of different deep convolutional networks, for example Inception-Resnet V1 or similar, where residual connections are used in combination with an Inception network to improve accuracy and computational efficiency. Such an alternative process is shown in Figure 4 where a face snippet 400 is processed using lnception-ResNet-V1 , shown at 405, to develop an embedding vector 410. For detection and classification during training, the embedding 410 is then processed through a convolutional neural network having a fully connected layer, shown at 415, to develop a classification or feature vector 420. Rectangular bounding boxes containing a detected face are expanded along one axis to a square to avoid disproportionate faces and then scaled to the fixed size as discussed above. During recognition, only steps 400-405-410 are used. In an embodiment, classification performance is improved during training by generating several snippets of the same face.[000108] The reasoning engine 240 (Figure 2C) is, in an embodiment, configured to query the detection data produced by the face and object detectors and return results very fast. To this end, the reasoning engine employs a distributed processing system such as Apache Spark in a horizontally scalable way that enables rapid searching through large volumes, e.g. millions, of face embeddings and object detections. In an embodiment, queries involving identities and objects can be structured using Boolean expression. For specific identities, the cohort database is queried for sample embeddings matching the specified name. A designator such as a colon allows identification of a class of objects rather than a person. Class terms, in the example “:car”, do not carry embeddings but instead are generic terms: any car matches “:car”. Similarly, any face in the data store will match “:face”. Specific examples of an item in a class can be identified if the network is trained to produce suitable embeddings for a given class of objects. As one example, a specific car (as identified e.g. by license plate), bag or phone could be part of the query if a network is trained to produce suitable embeddings for a given class.[000109] As noted above, in an embodiment the search data contains, in addition to the query string, the definitions of every literal appearing in the query. [It will be appreciated by those skilled in the art that a “literal” in this context means a value assigned to a constant variable.] Each token level detection, that is, each element in the query, is processed through a parse-tree of the query. For example, and as illustrated in Figure 5A, the query “(Alice & Bob) | (Dave & !:car)”, shown at 500, will first be received by the REST API back-end 505, and will be split into operators to extract literals. Responsive embeddings in the data store or other memory location are identified at 515 and the response returned to the REST API. Embeddings set to null indicate that any car detection is of interest. Response to the class portion of the query is then added, resulting in the output seen at 520. The result is then forwarded to the SPARK-based search back-end 525.[000110] The process of Figure 5A is illustrated in Boolean form in Figure 5B, where detections for each frame are evaluated against the literals in parse tree order, from bottom to top: Alice, Bob, Dave and :car. The query is first evaluated for instances in which both Alice (550) and (“&”, 555) Bob (560) are present, and also Dave (565) and (“&”, 570) any (“!”, 575) car (“:car”, 580) are present. The Boolean intersection of those results is determined at 585 for the final result. Detections can only match if they represent the same class.[000111] If embeddings for the specific entities are provided, then a level of confidence in the accuracy of the match is determined by the shortest distance between the embedding for the detection in the video frame to any of the samples provided for the literal. It will be appreciated by those skilled in the art that ‘distance’ in context means vector distance, where both the embedding for the detected face and the embedding of the training sample are characterized as vectors, for example 128-bit vectors as discussed above. In an embodiment, an empirically derived formula can be used to map the distance into a confidence range of 0 to 1 or other suitable range. This empirical formula is typically tuned / trained so that the confidence metric is statistically meaningful for a given context. For example, the formula may be configured such that a set of matches with confidence 0.5 is expected to have 50% true matches. In other implementations, perhaps requiring that a more rigorous standard be met for a match to be deemed reliable, a confidence of 0.5 may indicatea higher percentage of true matches. Less stringent standards may also be implemented by adjusting the formula. It will be appreciated by those skilled in the art that the level of acceptable error varies with the application. In some cases it is possible to map the confidence to a probability that a given face matches a person of interest by the use of Bayes rule. In such cases the prior probability of the person of interest being present in the camera view may be known, for example, via news, or some other data. In such cases, the prior probability and the likelihood of a match can be used in Bayes rule to determine the probability that the given face matches the person of interest.[000112] In an embodiment, for literals not carrying sample embeddings, the match confidence is simply the detection confidence. This should represent the likelihood that the detection actually represents the indicated class and again should be tuned to be statistically meaningful. As noted above, detections can only match if they are of the same class, so the confidence value for detections in different classes is zero. For all detections in the same class, there is a non-zero likelihood that any detection matches any identity. In other embodiments, such as those using geospatial imagery, objects may be detected in a superclass, such as “Vehicle”, but then classified in various subclasses, e.g, “Sedan”, “Convertible”, “Truck”, “Bus”, etc. In such cases, a probability / confidence metric might be associated with specific subclasses instead of the binary class assignment discussed above.[000113] Referring to Figure 6, an embodiment of a query process is shown from the expression of the query that begins the search until a final search result is achieved. The embodiment illustrated assumes that raw detections with embeddings have previously been accumulated, such as in Data Store 130 (Figure 2B). Alternatively, the development of raw detections and embeddings can occur concurrently with the evaluation of the query. For purposes of simplicity and clarity, it is assumed that each identity can appear only once in any given frame. This is not always true, for example a single frame could include faces of identical siblings could, or a reflection in a mirror. Similarly, there can be numerous identical objects, such as “blue sedan”, in a single frame. However, in most instances, especially involving faces, the assumption will be true and, at least for many embodiments, the final truth value of the expression of the query is derived from the best possible instance. Thispermits the expression to be solved as a linear assignment problem where standard solvers, for example the Hungarian algorithm, can be used to yield a solution.[000114] Thus, for Figure 6, at step 600 a collection of raw detections (e.g., faces, objects, activities) with embeddings is made available for evaluation in accordance with a query 620 and query parse tree 625. Identity definitions, such as by class or set of embedding, are defined at step 605, and the raw detections are evaluated accordingly at step 610. The result is solved with any suitable linear assignment solver as discussed above, where detections are assigned unique identity with a confidence value, shown at 615. In some embodiments, for example those where it might be desirable to rigorously avoid false positives, a solution is a one-to- one assignment of literals to detections in the frame, which requires there to be exactly the same number of literals and detections in the frame. In other embodiments, a more relaxed implementation of the algorithm can yield better results. For example, if the query is (Alice & blue sedan) | (purple truck), it is useful to match “blue sedan” and “purple truck” literals to a single vehicle detection in the frame rather than forcing a linear assignment that prevents one or the other from matching at all. This enables a more considered evaluation of the truthfulness of (Alice & blue sedan | (purple truck). If, in the example, the probability of Alice is low, then even though the vehicle might be more blue than purple, and more sedan than truck, the evaluation of the final query would get a higher truth value as matching “purple truck”. Depending upon the nature of the literal, matching multiple literals to the same detection can be either allowed or disallowed. As one example, an embodiment can have all face detections matched one-to-one to named persons in the query, while all other detections allow many-to-many matching.[000115] When this is not the case a priori, either dummy detections or literals can be introduced. These represent “not in frame” and “unknown detection”, respectively. A fixed confidence value, for example -1 , can be assigned to any such detections. The linear assignment problem maximizes the sum of confidences of the assignments, constrained to one-to-one matches. In this case, it gives the maximum sum of confidences. Since there must be |#detections - #literals| assignments to dummy entries, there will be a fixed term in the cost, but the solution still yields the strongest possible assignment of the literals.[000116] As noted above, steps 600 to 610 can occur well in advance of the remaining steps, such as by recording the data at one time, and performing the searches defined by the queries at some later time.[000117] The total frame confidence is then evaluated through the query parse tree, step 630, using fuzzy-logic rules: a & b => min(a,b), a | b => max(a,b), la => 1 - a. Additionally, a specific detection box is associated to each literal. These boxes are propagated through the parse tree. Each internal node of the parse tree will represent a set of detection boxes. Forit is the union of the detection boxes of the two children. it is the set on the side that yields the maximum confidence. For “!” (not), it is an empty set, and may always be an empty set. In the end, this process yields, for each frame, a confidence value for the expression to match and a set of detection boxes that has triggered that confidence, 635.[000118] For example, assume that the query asks “Are both Alice and Bob in a scene” in the gallery of images. The analysis returns a 90% confidence that Alice is in the scene, but only a 75% confidence that Bob is in the scene. Therefore, the confidence that both Bob and Alice are in the scene is the lesser of the confidence that either is in the scene - in this case, the 75% confidence that Bob is in the scene. Similarly, if the query asks “Is either Alice or Bob in the scene”, the confidence is the maximum of the confidence for either Alice or Bob, or 90% because there is a 90% confidence that Alice is in the scene. If the query asks “Is Alice not in the scene”, then the confidence is 100% minus the confidence that Alice is in the scene, or 10%.[000119] The per-frame matches are pooled into segments of similar confidence and similar appearance of literals. Typically the same identities, e.g., “Alice & Bob”, will be seen in multiple consecutive frames, step 640. At some point, this might switch and while the expression still has a high confidence of being true, it is true because Dave appears in the frame, without any cars. When this happens, the first segment produces a separate search result from the second. Also, if there is empty space where the query is true with a much lower confidence, in an embodiment that result is left out or moved into a separate search result, and in either case may be discarded due to a low confidence value (e.g., score). As noted hereinabove, the term “segment” in this context refers to a sequence of video data, rather than parts of a single frame as used in Figures 3A-3B.[000120] Finally, for each segment, the highest confidence frame is selected and the detection boxes for that frame are used to select a summary picture for the search result, 645. The segments are sorted by the highest confidence to produce a sorted search response of the analyzed video segments with thumbnails indicating why the expression is true, 650.[000121] The foregoing discussion has addressed detecting movement through multiple frames based on a per-frame analysis together with a query evaluated using a parse tree. In an alternative embodiment, tracking movement through multiple frames can be achieved by clustering detections across a sequence of frames. The detection and location of a person of interest in a sequence of frames creates a tracklet (sometimes called a “streak” or a “track”) for that person (or object) through that sequence of data, in this example a sequence of frames of video footage. In such an embodiment, clusters of face identities can be discovered algorithmically as discussed below, and as illustrated in Figures 7 A and 7B.[000122] In an embodiment, the process can begin by retrieving raw face detections with embeddings, shown at 700, such as developed by the techniques discussed previously herein, or by the techniques described in the patent applications referred to in the first paragraph above, all of which are incorporated by reference in full. In some embodiments, and as shown at 705, tracklets are created by joining consecutive frames where the embeddings assigned to those frames are very close (i.e., the “distance” between the embeddings is within a predetermined threshold appropriate for the application) and the detections in those frames overlap. Next, at 710 a representative embedding is selected for each tracklet developed as a result of step 705. The criteria for selecting the representative embedding can be anything suitable to the application, for example, the embedding closest to the mean, or an embedding having a high confidence level, or one which detects an unusual characteristic of the person or object, or an embedding that captures particular invariant characteristics of the person or object, and so on.[000123] Next, as shown at 715, a threshold is selected for determining that two tracklets can be considered the same person. As discussed previously, and discussed further in connection with Figure 8, the threshold for such a determination can be set differently for different applications of the invention. In general, everyimplementation has some probability of error, either due to misidentifying someone as a person of interest, or due to failing to identify the occurrence of a person of interest in a frame, The threshold set at step 715 reflects the balance that either a user or an automated system has assigned. Moreover, multiple iterations of the process can be performed, each at a different threshold such that groupings at different confidence levels can be presented to the user, as shown better in Figure 7B. Then at step 720, each tracklet is considered to be in a set of tracklets of size one (that is, the tracklet by itself) and at 725 a determination is made whether the distance between the embeddings of two tracklet sets is less than the threshold for being considered the same person. If yes, the two tracklet sets are unioned as shown at 730 and the process loops to step 725 to consider further tracklets. If the result at 725 is no, then at 735 the group of sets of tracklets at a given threshold setting is complete and a determination is made whether additional groupings, for example at different thresholds, remain to be completed. If so, the process loops to step 715 and another threshold is retrieved or set and the process repeats. Eventually, the result at step 735 is “yes”, all groupings at all desired thresholds have been completed, at which time the process returns the resulting groups of sets of tracklets as shown at 740.[000124] The result of the process of Figure 7 A can be better appreciated from Figure 7B. In Figure 7B, three groups 750, 755, 760 are shown, each representative of a different confidence level of detection. Thus, group 750 represents sets of tracklets where each set comprises one or more tracklets of an associated person or object. Figure 7B shows sets 765A-765n of tracklets 770A-7770m for Person 1 through Person N to which the system has assigned a high level of confidence that each tracklet in the set is in fact the person identified. As illustrated, there is one set of tracklets per person, but, since the number of tracklets in any set can be more than one, sets 765A-765n can comprise, in total, tracklets 770A-770m.[000125] Then, at 755 is shown a group of tracklets that have been assigned only a midlevel confidence value; that is, in sets 775A-775n, it is likely but not certain that each of the tracklets 780A-780p corresponds to the identified person or object. Finally, at 760 is a group of sets 785A-785n of tracklets 790A-790q where detection and filtering has been done only to a low confidence level, such as where only gross characteristics are important. Thus, while the tracklets 790A-790q are probablyprimarily associated with the person or object of interest, e.g., Person 1 - PersonN, they are likely to include other persons of similar appearance or, in the case of objects, other objects of similar appearance. It will be appreciated that, in at least some embodiments, when the tracklets are displayed to a user, each tracklet will be depicted by the representative image for that tracklet, such that what the user sees is a set of representative images by means of which the user can quickly make further assessments.[000126] Referring next to Figure 8, an important aspect of some embodiments of the invention can be better appreciated. As noted previously, in some applications of the present invention, greater accuracy or greater granularity is preferred at the expense of less compression of the data, whereas in other applications, greater compression of the data is preferred at the expense of reduced accuracy and reduced granularity. Stated differently, in some applications permitting missed recognitions of an object or person of interest may be preferred over false matches, i.e., wrongly identifying a match. In other applications, the opposite can be preferred. The probability distribution curves 880 and 885 of Figure 8 illustrate this trade-off, in terms of choosing an optimal embedding distance that balances missed recognitions on the one hand and false matches on the other. In Figure 8, curve 880 (the left, flatter curve) depicts “in class” embedding distances, while the curve 885 (the right curve with the higher peak) depicts cross class embedding distances. The vertical line D depicts the embedding distance threshold for a given application. The placement of vertical line D along the horizontal axis depicts the balance selected for a particular application. As an example, for the vertical line D indicated at 790, the area of curve 780 to the right of the line D represents the missed recognition probability while the area under the curve 785 to the left of the line D, 790, represents the false recognition probability. It will be appreciated by those skilled in the art that selection of that threshold or balance point can be implemented in a number of different ways within the systems of the present invention, including during training, at selection of thresholds as shown in Figure 7A, or during clustering as discussed hereinafter in connection with Figures 15A-15D, or at other convenient steps in processing the data.[000127] Referring next to Figure 9, an aspect of the invention related to assigning a confidence value to a detection can be better appreciated. Morespecifically, Figure 9 illustrates a novel capability to discover the strength of relationships between actors around a person of interest through analysis of the multisensor data. Assuming this is proportional to the amount of time people appear together in the same frame in the videos, the strength of the relationship between two detected faces or bodies can be automatically computed for every individual defined by sample embeddings.[000128] Starting with retrieving raw detections with embeddings, shown at 900, and identity definitions, 905, every frame of the video is evaluated for presence of individuals in the same way as if searching for (A | B | ...) - e.g. the appearance of any identity as discussed above. Every frame then produces a set of key value pairs, where the key is a pair of names, and the value is confidence, shown at 910 and 915. For example, if a frame is deemed to have detections of A, B and C, with confidences c_a, c_b, c_c, respectively, then three pairs exist: ((A,B),min(c_a,c_b)), ((A,C), min(c_a,c_c), ((B,C), min(c_b, c_c)) as shown at 920.[000129] These tuples are then reduced (for example, in Spark, taking advantage of distributed computing) according to the associated key into histograms of confidences, shown at 925, with some bin size, e.g. 0.1 (producing 10 bins). In other words, for any pair of people seen together, the count of frames where they appear together at a given confidence range can be readily determined.[000130] From this, the likelihood or strength of connection between the individuals can be inferred. Lots of high confidence appearances together indicate a high likelihood that the individuals are connected. However, this leaves an uncertainty: are ten detections at confidence 0.1 as strong a single detection at confidence 1.0? This can be resolved from the histogram data, by providing the result to an artificial intelligence algorithm or to an operator by means of an interactive tool and receiving as a further input the operator’s assessment of the connections derived with different settings. As noted above, the level of acceptable error can vary with the particular application, as will the value / need for user involvement in the overall process. For example, one application of at least some aspects of the present invention relate to customer loyalty programs, for which no human review or intervention may be necessary.[000131] For some detected individuals, the objective of searching for companions may be to find any possible connection, such as looking for unlikely accomplices. For example, certain shoplifting rings travel in groups but the individuals appear to operate independently. In such a case, a weaker signal based on lower confidence matches can be acceptable. For others, with many strong matches, higher confidence can be required to reduce noise. Such filtering can easily be done at interactive speeds, again using the histogram data.[000132] Other aspects of the strength of a connection between two detected individuals are discussed in U.S. Patent Application S.N. 16 / 120,128 filed 8 / 31 / 2018 and incorporated herein by reference. In addition, it may be the case that individuals within a network do not appear in the same video footage, but rather within a close time proximity of one another in the video. Other forms of connection, such as geospatial, with reference to a landmark, and so on, can also be used as a basis for evaluating connection. In such cases, same-footage co-incidence can be replaced with time proximity or other relevant co-incidence. Using time proximity as an example, if two persons are very close to each other in time proximity, their relationship strength would have a greater weight than two persons who are far apart in time proximity. In an embodiment, a threshold can be set beyond which the connection algorithm of this aspect of the present invention would conclude that the given two persons are too far apart in time proximity to be considered related.[000133] As noted earlier in the discussion of Figures 5A-5B et seq., in some embodiments the present invention can identify an entity, i.e. , a person, in combination with a specific object. Figure 10 shows an example flowchart describing the process for detecting matches between targets received from a query and individuals identified within a selected portion of video footage, according to an example embodiment. As described above, the techniques used to match target individuals to unidentified individuals within a sequence of video footage may also be applied to match target objects to unidentified objects within a sequence of video footage. At 1005 a search query is received from a user device and at 1010 each target object and each target individual within the query is identified. For each target object, at step 1015 the query processor extracts a feature vector from the query describing the physical properties of each object. The process then iteratively moves through frames of the digital fileand the groupings derived therefrom to compare the feature vector of each target object to the feature vector of each unidentified object. Before comparing physical properties between the two feature vectors, the classes of the two objects are compared at step 1020. If the object classes do not match the process branches to step 1025 and the process advances to analyze the next unidentified object within the file. If the objects do match, the process advances to step 1030 where the feature distance is calculated between the query object and the object from the digital file. Finally, each match is labeled at step 1035 with a confidence score based on the determined distance of the feature vectors. The process then loops to examine any objects remaining for analysis.[000134] Simultaneously following step 1010, embeddings are extracted at step 1050 for each face from the query. The embeddings of each individual in the query are then compared at step 1055 to the unidentified individuals in the data file. At step 1060 a feature distance is determined between the individuals in the query and the individuals identified from the digital file to identify matches. At step 1065 each match is labeled with a confidence based on the determined feature distance. Finally, the recognition module aggregates at step 1080 the matches detected for objects and faces in each grouping into pools pertaining to individual or combinations of search terms and organizes each of the aggregated groupings by confidence scores.[000135] Referring next to Figure 11 , details of the second major aspect of the present invention can be better appreciated from the following. As summarized above, the second major aspect differs from the first in that the detections are made without the use of a probe or reference image, although both rely on the same basic multisensor processing platform. Fundamentally, the objective of this aspect of the invention is to simplify and accelerate the review of a large volume of sequential data such as video footage by an operator or appropriate algorithm, with the goal of identifying a person or persons of interest where the likeness of the those individuals is known only in a general way, without a photo. As will be appreciated from the following discussion, this goal is achieved by compressing the large volume of unstructured data into representative subsets of that data. In addition, in some embodiments, frames that reflect no movement relative to a prior frame are notprocessed and, in other embodiments, portions of a frame that show no movement relative to a prior frame are not processed.[000136] This is accomplished by dividing the footage into a plurality of sequences of video frames, and then identifying all or at least some of the persons detected in a sequence of video frames. The facial detection system comprises a face detector and an embedding network. The face detector generates cropped bounding boxes around faces in any image. In some implementations, video data supplemented with the generated bounding boxes may be presented for review to an operator. As needed, the operator may review, remove random or false positive bounding boxes, add bounding boxes around missed faces, or a combination thereof. In an embodiment, the operator comprises an artificial intelligence algorithm rather than a human operator.[000137] The facial images within each network are input to the embedding network to produce some feature vector, for example a 128-dimensional vector of unit length. The embedding network is trained to map facial images of the same individual to a common individual map, for example the same 128-dimension vector. Because of how deep neural networks are trained in embodiments where the training uses gradient descent, such an embedding network is a continuous and differentiable mapping from image space (e.g. 160x160x3 tensors) to, in this case, S127, i.e. the unit sphere embedded in 128 dimensional space. Accordingly, the difficulty of mapping all images of the same person to exactly the same point is a significant challenge experienced by conventional facial recognition systems. Additionally, conventional systems operate under the assumption that embeddings from images of the same person are closer to each other than to any embedding of a different person. However, in reality, there exists a small chance that embeddings of two different people are much closer than two embeddings of the same person, which conventional facial recognition systems fail to account for.[000138] To overcome those limitations of conventional systems, the facial recognition system interprets images of the same person in consecutive frames as differing from each other much less than two random images of that person. Accordingly, given the continuity of the embedding mapping, the facial recognition system can reasonably expect embeddings to be assigned much stronger facedetections between consecutive frames compared to the values assigned to two arbitrary pictures of the same person.[000139] Still referring to Figure 11 , the overall process of an embodiment of this aspect of the invention starts at 1100 where face detections are performed for each frame of a selected set of frames, typically a continuous sequence although this aspect of the present invention can yield useful data from any sequence. The process advances to 1105 where tracklets are developed as discussed hereinabove. Then, at 1110 and 1 115, a representative embedding and representative picture is developed. The process advances to laying out the images developed in the prior step, 1120, after which localized clustering is performed at step 1125 and highlighting and dimming is performed substantially concurrently at step 1130. Curation is then performed at step 1135, and the process loops back to step 1120 with the results of the newly curated data. Each of these general steps can be better appreciated from the following discussion. It will be appreciated that, in at least some embodiments, when the tracklets are displayed to a user, such as at the layout step, each tracklet will be depicted by the representative image or picture for that tracklet, such that what the user sees is a set of representative images by means of which the user can quickly make further assessments.[000140] As touched on hereinabove, in at least some embodiments the system of the present invention can join face detections in video frames recorded over time using the assumption that each face detection in the current frame must match at most one detection in the preceding frame. As noted previously, a tracklet refers to a representation or record of an individual or object throughout a sequence of video frames. The system may additionally assign a combination of priors I weights describing a likelihood that a given detection will not appear in the previous frame, for example based on the position of a face in the current frame. For example, in some implementations new faces may only appear from the edges of the frame. The facial recognition system may additionally account for missed detections and situations in which one or more faces may be briefly occluded by other moving objects I persons in the scene.[000141] For each face detected in a video frame, the facial recognition system determines a confidence measure describing a likelihood that an individual ina current frame is an individual in a previous frame and a likelihood that the individual was not in the previous frame. For the sake of illustration, the description below describes a simplified scenario. However, it should be understood that the techniques described herein may be applied to video frames with much larger amounts of detections, for example detections on the order of tens, hundreds or thousands. In a current video frame, individuals X, Y, and Z are detected. In a previous frame, individuals A and B are detected. Given the increase in detections from the previous frame to the current frame, the system recognizes that at least one of X, Y, and Z were not in the previous frame at all, or at least were not detected in the previous frame. Accordingly, in one implementation, the facial recognition system approaches the assignment of detection A and B to two of detections X, Y, and Z using linear assignment techniques, for example the process illustrated below.[000142] An objective function may be defined in terms of match confidences. In one embodiment, the objective function may be designed using the embedding distances given that smaller embedding distances correlate with a likelihood of being the same person. For example, if an embedding distance between detection X and detection A is less than an embedding distance between detection Y and detection A, the system recognizes that, in general, the individual in detection A is more likely to be the same individual as in detection X than the individual in detection Y. To maintain the embedding network, the system may be trained using additional training data, a calibration function, or a combination thereof.[000143] In another embodiment, the probability distributions that define the embedding strength areP(d(x,y) | ld(x) = ld(y)) andP(d(x,y) | ld(x) ld(y)),where d(x,y) is the embedding distance between two samples x,y and ld(x) is the identity (person) associated with sample x. These conditional probability distribution functions of the embedding distance are independent of the prior probability P(ld(x) = I d(y)), which is a critical feature of the validation data that would be reflected in typical Receiver Operating Characteristic (ROC) curves used to evaluate machine learning (ML) systems. However, these conditional probabilities can also be estimated using validation data, for example using validation data that represents sequences of faces from videos to be most representative of the actual scenario[000144] Given the prior probability distribution PT= P(ld(x) = ld(y)), the following can be defined:where the Bayes theorem is used to obtain the last equality. Further, it is natural to expect that k = 1-p-r.[000145] Continuing from the example scenario described above, the facial recognition system can estimate the probability distribution (p-r) from the number of detections in the current frame and the previous frame. If there are N detections (e.g., 3) in current frame and M (e.g., 2) in the previous frame, then the probability distribution may be modeled as min(M,N) pTX - E r ' MN where £ represents the adjustment made based on missed or incorrect detections.[000146] In an embodiment, initially the active tracklets T are represented as an empty feature vector []. In one embodiment, tracklet IDs are assigned to detections D in a new frame using the following process:Define N = max(len(T), len(D))Define PT= (min(len(T), len(D)) / (len(T) * len(D)) - £Generate an NxN matrix D such that D(i,j) = s(D(i), T(j)) if i < len(D) and j < len(T) and 1 - pi otherwiseBased on the generated matrix, compute a one-to-one mapping f(i): [1 ,N] -> [1 ,N] such that X" i D(i,f(ty) is maximized.For i e [1 , N], assign a tracklet Id of T(f(i))to detection i, if f(i) < len(T). Otherwise generate a new tracklet ID for the detection.Replace T with the detections from D with the assigned tracklet IDs.[000147] Referring next to Figure 12, a technique for extracting tracklets in accordance with an embodiment of the invention can be better appreciated. Beginning at 1200, detections and embeddings at time T are retrieved. The embedding distance matrix D(l J) is computed from the embedding distance between detection I and tracklet j, shown at 1205. Matrix D is then expanded into square matrix A, step 1210, where A is as shown at 1215 and discussed below, after which the linear assignment problem on A is solved, step 1220, to determine matches. For detections that were matched, an identity tracklet ID is either assigned or carried over from the matching detection in the preceding frame and the embedding of the matched tracklet is updated, 1225. New tracklets are created for detections that were not matched, 1230, with a new unique ID assigned to the detection and to the new tracklet. Finally, at step 1235, remove tracklets that were not assigned a detection. The process then loops back to step 1205 for the next computation.[000148] As will be appreciated by those skilled in the art, for N detections and M active tracks, D is an NxM matrix. The matrix A will be (N+M)x(N+M) square matrix. The linear assignment problem is understood to produce a permutation P of [1 , ... , N+M] such that the sum over A[i, P(i)] for i=1 ..N+M is minimized. The padded regions simply represent detections that represent identities that appear, identities that disappeared or are simply computational overhead as depicted on the right. Constant values are used for these regions and they represent the minimum distance required for a match. The linear assignment problem can be solved using standard, well known algorithms such as the Hungarian Algorithm.[000149] To improve run time, a greedy algorithm can be used to find a “good enough” solution, which for the purposes of tracking is often just as good as the optimal. The greedy algorithm simply matches the pair (i,j) corresponding to minimum A(i,j) and removes row i and column j from consideration and repeats until every row is matched with something.[000150] Tracklets will have their representative embedding taken from the detection upon creation. A number of update rules can be used to match embeddings to tracks, including using an average of the embeddings assigned to the track. Alternatives include storing multiple samples for each tracklet, or using a form of k- nearest distance to produce a meaningful sample-based machine learning solution. RANSAC or other form of outlier detection can be used in the update logic.[000151] For each tracklet, the facial recognition system constructs a single embedding vector to represent the entire tracklet, hereafter referred to as a representative embedding. In one embodiment, the representative embedding is generated by averaging the embeddings associated with every detection in the tracklet. In another implementation, the facial recognition system determines a weighted average of the embeddings from every detection in the tracklet, where each of the weights represent an estimate of the quality and usefulness of the sample for constructing an embedding which may be used for recognition. The weight may be determined using any one or more combination of applicable techniques, for example using a Long Short-term Recurrent Memory (LSTM) network trained to estimate weights that produce optimized aggregates.[000152] In another embodiment, the facial recognition system generates a model by defining a distance threshold in the embedding space and selecting a single embedding for the tracklet that has the largest number of embeddings within the threshold. In other embodiments, for example those in which multiple embeddings are within the distance threshold, the system generates a final representative embedding by averaging all embeddings within the threshold.[000153] For purposes of illustration, in an embodiment a representative embedding is determined using the following process:Define max_count = 0For e embeddings for the trackletDefine ent = count ( d(e,x) < th for x in embeddings ) - 1If ent > max_count:Define max_count = entDefine center = eDetermine the output as:avg(x for x in embeddings if d(x, center) < th)[000154] With reference to Figure 13, a process for selecting a representative embedding is illustrated in flow diagram form. Beginning at step 1300, the process initiates by selecting N random embeddings. Then, at 1305, for each embedding, a count is made of the number of other embeddings within a predetermined threshold distance. The embedding with the highest count is selected, 1310, and at 1315 an average is calculated of the embeddings within the threshold. The result is normalized to unit length and selected as the representative embedding, 1320.[000155] Selection of a representative picture, or thumbnail, for each tracklet can be made in a number of ways. One exemplary approach is to select the thumbnail based on the embedding that is closest to the representative embedding, although other approaches can include using weighted values, identification of a unique characteristic, or any other suitable technique.[000156] Once a representative picture and representative embedding have been selected, an optimized layout can be developed, per step 1120 of Figure 11. In an embodiment, for each face detected in a sequence of video frames, the facial recognition system generates a tracklet with a thumbnail image of the individual’s face, a representative embedding, and a time range during which the tracklet was recorded. In such an embodiment, the facial recognition system thereafter generates an interface for presentation to a user or Al system by organizing the group of tracklets based on the time during which the tracklet was recorded and the similarity of the tracklet embedding to the representative embedding.[000157] The results of such an approach can be appreciated from Figures 14A-14B. In the embodiment illustrated there, the vertical axis of the interface is designated as the time axis. Accordingly, scrolling down and up is equivalent to moving forward and back in time, respectively. By vertically scrolling through an entire interface of tracklets, shown as T1 to T10 arranged on grid 1400, a user can inspect the entirety of the footage of video data. Reviewing the tracklets by scrolling through the interface vertically may provide a user with a sense of progress as the user scrolls down the grid.[000158] Additionally, each tracklet is positioned on the interface such that a first occurrence of a person may never be earlier to any appearing tracklet positioned higher on the interface.[000159] Based on a fixed width of the display, a number of tracklets W can be displayed along the horizontal rows of the interface where the number W is defined as W = window_width I (thumbnail_width + padding). Images on the same row may be displayed in arbitrary order. Accordingly, in an embodiment designed to facilitate quick visual scanning, images can be ordered based on similarity using the following algorithm.[000160] Given a list of tracklets T, sorted by their start time: let P = [] let S = T[:W], and T = T[W:], i.e. S is the first W tracklets taken out of TIf P is not empty, set N[0] to the tracklet in S closest to P[0] in embedding, otherwise N[0] = S[0]Remove N[0] from SFor i in range(1 , W):Find the element) in S such that d(S[j], N[i-1 ]) + d(S[j], P[j]) is minimized where the latter term is zero if there is no element P[j] available.N[i] = SO]Remove element ) from S add row N to the grid.P = N if T is not empty, goto 2[000161] The foregoing algorithm attempts to minimize embedding distance between adjacent face pictures, such as shown at 1405 and 1410 of Figure 14B. Accordingly, individuals with similar facial features, for example glasses or a beard, may be clustered together. In another implementation, the system may generate a globally optimal arrangement.[000162] It may be the case that the same face appears multiple times within a layout such as shown in Figures 14A-14B, where tracklets T1-T14 represent achronology of captured images intended for layout in a grid 1400. Even within a small section of video, the same face / object may appear in multiple distinct tracklets. This could be due to a number of reasons, such as occlusions that interrupted the continuity of the face / object from one frame to the next, the face / object exiting then re-entering the frame, or the inner workings of neural networks whereby two faces / objects which are the same to the human observer are not recognized as such by the system based on their embeddings. Because many people’s faces look somewhat different depending upon the camera angle at which a person’s image is captured, or the lighting conditions, or other physical or environmental factors, it is possible for images of the face of a single person to be categorized by the present invention as several different faces, and to have tracklets developed for each of those faces. I n the present invention, those different perspectives of the same person are referred to as “key faces”. In an embodiment, tracklets with similar embeddings, e.g. 1405, can be arranged near one another while those that are dissimilar, e.g. 1410, are placed at the outer portions of the layout. As noted above, while the tracklets shown are depicted as shaded squares, in some embodiments each tracklet presented for review by a user will display the representative image or picture for that tracklet.[000163] Combining tracklets that are of the same person effectively reduces, or compresses, the volume of data a user must go through when seeking to identify one or more persons from the throng of people whose images can be captured in even just a few minutes of video taken at a busy location. To aid in identifying cases where two or more tracklets are in fact the same face / object and thus enable further compression of the number of distinct data points that the user must review, the system may employ clustering, and particularly agglomerative clustering.[000164] In simplified terms, agglomerative clustering begins with each tracklet being a separate cluster. The two closest clusters are iteratively merged, until the smallest distance between clusters reaches some threshold. Such clustering may take several forms, one of which can be a layer of chronologically localized clustering. One algorithm to achieve such clustering is as follows:Given a list of tracklets T for a small section of footage (e.g. 5-10 minutes) ordered by confidence descending: let C = []For i in range(0, T): let t = T[i]Calculate the distance D between t and c for each cluster c in C as follows:For k in c where k is a “key face” tracklet which is part of the cluster:Calculate the distance between t and k Return the minimum distanceIf D < same-cluster tolerance:Add tracklet t to cluster c and re-compute “key faces” (see below)Otherwise:Create a new cluster c, add tracklet t to it as a key face, and add c to CKey face algorithm for tracklets t in cluster c let K = []Fort in range(1 , c): (2)For key face tracklet k in K:Calculate the distance D between t and kIf D < same-key face tolerance, goto (2) Otherwise add t to K[000165] The narrower the band of time, the more performant such a clustering algorithm will be. This can be tuned depending on how many faces are displayed in the grid at any given time such that the faces within the current frame of view are covered by the clustering algorithm. The results of such a clustering algorithm are embodied visually in the grid 1400. As shown there, in an embodiment, when one of the faces is selected (either by clicking or by hovering), all faces within the same cluster are highlighted within the grid. There is no guarantee that all faces within the cluster are indeed the same person, so this is an aid to the user and not a substitute for their own review and discretion.[000166] To elaborate on the foregoing, it will be appreciated by those skilled in the art that a distance between two clusters can be defined in various ways, suchas Manhattan, Euclidean, and so on, which may give somewhat different results. The choice of how distance is defined for a particular embodiment depends primarily on the nature of the embedding. One common choice for distance is set distance. In at least some embodiments of the present invention, averaging the embedding works well and is recognized in the literature. Further, various methods of outlier removal can be used to select a subset of embeddings to include in computing the average. One approach, used in some embodiments is to exhaustively test, or randomly (RASNAC-like) select points and find how many other points are within some threshold of that point. The point that has the largest number of neighbors by this rule is selected as the “pivot” (see Figure 16) and all the points within threshold of the pivot are then averaged, with points beyond the threshold being discarded as outliers.[000167] Figure 15A illustrates a simplified representation of localized clustering. Thus, at 1500, a single point cluster is created from all tracklets under consideration. Then, at 1505, using a similarity metric, a search is made for the two clusters that are the most similar. At 1510, the similarity of the two clusters is compared to a predetermined threshold. If the similarity is sufficiently high that it exceeds the threshold value, the two clusters are merged (agglomerated) at 1520. Conversely, if similarity between the two clusters is less than the threshold, the process is done and the current set of clusters is returned. It will be understood that, because the threshold can be varied, in accordance with the probability distribution curves discussed at Figure 8, more or less merging of clusters will occur depending upon the balance between the level of granularity of result and the level of data compression desired for a particular embodiment, and a particular application.[000168] Referring next to Figures 15B-15D, a more detailed exposition of clustering in accordance with some embodiments of the present invention can be appreciated. As discussed above, clustering could be for the entire video orfor a small section. For greater performance, it might be applied only to a narrow band of time in the video corresponding to what the system is currently reporting to the user in the aforementioned grid. If the goal is to more comprehensively analyze the entire video, then clustering could be applied to all tracklets or at least larger sections of the video.[000169] Further, clustering can be hierarchical. Outer tiers in the hierarchy yield the most compression and least accuracy, i.e., the highest likelihood that twotracklets that represent different underlying faces / objects are erroneously grouped together in the same cluster. Inner tiers yield the least compression but the most accuracy. One such hierarchical embodiment comprises three tiers as follows, and as depicted in Figures 15C and 15D:[000170] Outer Tier (Cluster), 1580A-1580n: Each cluster C contains multiple key groups K. Key groups within a cluster are probably the same face / object. Different clusters C are almost surely different faces / objects.[000171] Middle Tier (Key Group), 1585A (in Cluster 0), 1587A-1587B (in Cluster 1 ), 1589A (in Cluster 2), and 1591 A (in Cluster N): A key group is simply a group of tracklets where the group itself has a representative embedding. In its simplest form, the group’s representative embedding is the same as the representative embedding of the first tracklet added to the group. Tracklets within the key group are almost surely the same face / object. In an embodiment, when a key group is presented to a user, the key face is displayed as representative of that key group.[000172] Inner Tier (Tracklet), T1-Tm: Each tracklet T is as described previously. Detections within a tracklet are substantially certain to be the same face / object.[000173] One algorithm to generate such a hierarchical set of clusters is shown in flow chart form in Figure 15B, and is further described as follows with reference numerals as indicated, with the first four steps below being collectively designated at 1525 on Figure 15B:Let C[] be an empty set of clusters representing the outermost tierLet Tolerancecluster be the tolerance threshold for determining when two key groups belong in the same clusterLet ToleranceKey be the tolerance threshold for determining when two tracklets belong in the same key group Given a list of tracklets T[]For each tracklet Ti: (1530)For each cluster Ci in C[]:For each key group Ki in Ci:(1540)Calculate the vector distance Di between the representative embedding of Ti and the representative embedding of the key tracklet in KiIf Di < ToleranceKey then add Ti to the key group Ki and continue with the next tracklet T in step (1530) (1545)If min(Di-n) < Tolerancecluster:Create a new key group K with tracklet Ti as the key tracklet and add K to Ci then continue with the next tracklet T in step (1530) (1560)T was not within tolerance of any given cluster C, so create a new key group K with T as the key tracklet, add to a new cluster C, and add C to the list of all outer clusters C[] and continue with next tracklet T in step (1530) (1565 -1575)[000174] To assist in understanding, the foregoing process can be visualized with reference to Figure 15C. A group of tracklets T1-Tn, indicated collectively at 1578, is available for clustering. Each cluster, indicated at 1581A-n and captioned Cluster 0 through Cluster n, comprises one or more key groups, indicated at 1580A-n and captioned Key Group 0 through Key Group n. Through the process discussed above and shown in Figure 15B, each tracklet is assigned to a Key Group, such as key group 1583A of Cluster 1580A. Each Cluster may have more than one Key Group, and the first tracklet assigned to each Key Group is the key tracklet for that group, as indicated at 1585A in Cluster 0. Each Key Group can have more than one tracklet. Embedding distance, calculated by any approach suitable to the application, is used to determine which key group a particular tracklet is assigned to.[000175] In the example shown, the first tracklet, selected randomly or by any other convenient criteria and in this case T10, is assigned to Cluster 0, indicated at 1580A, and more specifically is assigned as the key tracklet 1585A in Cluster 0’s Key Group 0, indicated at 1583A. The embedding of a second tracklet, T3, is distant from Cluster 0’s key (i.e., the embedding of T10), and so T3 is assigned to Cluster 1 , indicated at 1580B. As with tracklet T10, T3 is the first tracklet assigned to Cluster 1and so becomes the key of Cluster 1’s key group 0, indicated at 1587A. A third tracklet, T6, has an embedding very near to the embedding of T10 - i.e. , the key for key group 0 of Cluster 0 - and so joins T10 in key group 0 of Cluster 0. A fourth tracklet, T7, has an embedding distance that is far from the key of either Cluster 0 or Cluster 1. As a result, T7 is assigned to be the key for Key Group 0 of Cluster 2, indicated at 1589A and 1580C, respectively. A fifth tracklet, T9, has an embedding distance near enough to Cluster Ts key, T3, that it is assigned to the same Cluster, or 1580B, but is also sufficiently different from T3’s embedding that it is assigned to be the key for a new key group in Cluster 1’s Key Group 1 indicated at 1587B. Successive tracklets are assigned as determined by their embeddings, such that eventually all tracklets, ending with tracklet Tn, shown assigned to Key Group N, indicated at 1591 A of Cluster N, indicated at 1580n, are assigned to a cluster and key group. At that time, spaces 1595, allocated for tracklets, are either filled or no longer needed.[000176] The end result of the processes discussed above and shown in Figures 15B and 15C can be seen in Figure 15D, where each tier - Group of Clusters, Cluster, Key Group - can involve a different levels of granularity or certainty. Thus, each cluster typically has collected images of someone different from each other cluster. For example, Cluster 0 may have collected images that are probably of Bob but almost certainly not of either Mike or Charles, while Cluster 1 may have collected images of Mike but almost certainly not of either Bob or Charles, and Cluster N may have collected images of Charles but almost certainly not of either Bob or Mike. That’s the first tier.[000177] Then, within a given cluster, for example Cluster 0, while all of the images are probably of Bob, it remains possible that one or more key groups in Cluster 0 has instead collected images of Bob’s doppelganger, Bob2, such that Key Group 1 of Cluster 0 has collected images of Bob2. That is the second tier of granularity.[000178] The key group is the third level of granularity. Within a key group, for example Key Group 0 in Cluster 0, every tracklet within that Key Group 0 almost surely comprises images of Bob and not Bob2 nor anyone else. In this manner, each cluster represents a general area of the embedding space with several centers of mass inside that area. Using keys within each cluster reduces computational cost since it allows the system to compare a given tracklet with only the keys in a cluster rather than everytracklet in that cluster. It also produces the helpful side-effect of a few representative tracklets for each cluster. Note that, while three tiers of granularity have been used for purposes of example, the approach can be extended to N tiers, with different decisions and actions taken at each different tier. This flexibility is achieved in at least some embodiments through the configuration of various tolerances.[000179] More specifically, and referring to steps 1540 and 1550 of Figure 15B, the settings of Tolerancekey and Tolerancecluster are used in at least some embodiments to configure the system to achieve a balance between data compression and search accuracy at each tier. This approach is an efficient variant of agglomerative clustering based on the use of preset fixed distance thresholds to determine whether a tracklet belongs in a given cluster and, further, whether a tracklet constitutes a new key within that cluster. As discussed above, each unassigned tracklet is compared to every key within a given existing cluster. The minimum distance of those comparisons is compared against Tolerancekey. If that minimum distance is less than or equal to Tolerancekey, then that tracklet is assigned to that key group within that cluster. If the minimum distance is greater than Tolerancekey for every key group within a cluster, but smaller than or equal to Tolerancecluster for that cluster, then the unassigned tracklet is designated a key for a new key group within that cluster. If, however, the minimum distance for that unassigned tracklet is greater than Tolerancecluster then the unassigned tracklet is not assigned to that cluster and instead is compared to the keys in the next cluster, and so on. If that unassigned tracklet remains unassigned after being compared with all existing clusters, either a new cluster (cluster N, step 1575 of Figure 15B) is defined or, in some embodiments, the unassigned cluster is rejected as an outlier.[000180] Such a hierarchy allows for different degrees of automated decision making by the system depending on how trustworthy and accurate the clustering is at each tier. It also allows the system to report varying degrees of compressed data to the user. At outer tiers, the data is more highly compressed and thus a user can review larger sections of data more quickly. The trade off, of course, is the chance that the system has erroneously grouped two different persons / objects into the same cluster and thus has obfuscated from the user unique persons / objects. The desired balance between compression of data, allowing more rapid review of large amounts of data,versus the potential loss of granularity is different for different applications and implementations and the present invention permits adjustment based on the requirements of each application.[000181] As noted initially, there are two main aspects to the present invention. In some applications, an embodiment which combines both aspects can be desirable. Those skilled in the art will recognize that the first aspect, discussed above, uses a per-frame analysis followed by aggregation into groups. The per-frame approach to performing a search has the advantage that it naturally segments to a query in the sense that a complex query, particularly those with OR terms, can match in multiple ways simultaneously. As objects and identities enter and leave the scene - or their confidences change due to view point - the "best" reason to think the frame matched the query may change. It can be beneficial to split results so that these alternative interpretations of the data can be shown. The second main aspect of the invention, involving the use of tracklets, allows for more pre-processing of the data. This has advantages where no probe image exists although this also means that detections of objects are effectively collapsed in time up front.[000182] In at least some embodiments of the invention, the system can combine clustering with the aforementioned optimized layout of tracklets as an overlay layer or other highlighting or dimming approach, as illustrated in Figures 16A-16C. As before, for some embodiments, while the tracklets in the grid 1400 are shown in the figures as shaded squares, when displayed to a user the tracklets will display the representative image forthat tracklet. This can be appreciated from Figure 16C, which shows how data in accordance with the invention may actually be displayed to a user, including giving a better sense of how many representative images might be displayed at one time in some embodiments.[000183] Thus, to provide a visual aid to the user, all tracklets within a given cluster, e.g. tracklets 1600 can be highlighted or outlined together and differently than tracklets of other clusters, e.g. tracklets 1605, to serve to allow a human user to easily associate groups of representative faces / objects and thus more quickly review the data presented to them. Alternatively, the less interesting tracklets can be dimmed or blanked. The system in this sense would emphasize its most accurate data at the mostgranular tier (tracklets) while using the outermost tier (clusters) in an indicative manner to expedite the user’s manual review.[000184] Referring particularly to Figure 16B, a process for selecting tracklets for dimming or highlighting can be better appreciated. At 1615, a “pivot” tracklet 1620 with its representative image is selected from a group of tracklets 1625 in the grid 1400. At 1630, embedding distances are calculated between the pivot tracklet and the other tracklets in the grid. Then, at 1635, tracklets determined to have an embedding distance less than a threshold, indicated at 1640, are maintained while tracklets determined to have an embedding distance greater than a threshold, indicated at 1645, are dimmed.[000185] To further aid the visualization and readability of the generated interface, the facial recognition system may dim certain faces on the interface based on anticipated features of the suspect, as shown in Figure 16C. When only an embedding is available, selecting (by clicking on them) similar looking faces may yield a set of close matches. For example, other samples in the grid that are close to this set can be highlighted making it easier to visually spot more similar faces. This implementation is illustrated in the following illustrated interface.[000186] As a further aid to the user, a curation and feedback process can be provided, as shown in Figure 17. Using the aforementioned visual aids, a human operator 1700 can identify sets of faces within the grid 1400 which they can confirm are the same person, e.g. as shown at 1705. Selecting a set of faces (e.g., by clicking) enables extraction of those faces from the grid as a curated person of interest, whereupon the grid re-adjusts as shown at 1710. In an embodiment, rows in the grid where faces were extracted are reduced in size, or eliminated altogether. In an alternative embodiment the grid is recalculated based on the operator’s action. In this way, the grid becomes interactive and decreases in noisiness as the operator engages with the data.[000187] In an embodiment, curated persons of interest appear in a separate panel adjacent to the grid. Drilling into one of the curated persons (by clicking) will update the grid such that only faces deemed similar to that person (within a threshold) are displayed. Faces in the drilled-down version of the grid have a visual indicator of how similar they are to the curated person. One implementation is highlighting anddimming as described above. Another implementation is an additional visual annotation demarcating “high”, “medium”, and “low” confidence matches.[000188] It will be appreciated that, in some embodiments, no human operator 1700 is involved and the foregoing steps where a human might be involved are instead fully automated. This can be particularly true for applications which tolerate lower thresholds of accuracy, such as fruit packing, customer loyalty programs, and so on. Referring next to Figures 18A-18C, a still further aspect of some embodiments of the present invention can be better appreciated. Figures 18A-18C illustrate the use of color as an element of a query. In many searches for objects, color is a fundamental requirement for returning useful search results. Color is usually defined in the context of a “color space”, which is a specific organization of colors. The usual reference standard for defining a color space is the CIELAB or CIEXYZ color space, often simply referred to as Lab. In the Lab color space, “L” stands for perceptual lightness, from black to white, while “a” denotes colors from green to red and “b” denotes colors from blue to yellow. Representation of color in the Lab color space can thus be thought of as a point in three-dimensional Space, where “L” is one axis, “a” another, and “b” the third.[000189] In an embodiment, a 144-dimensional histogram in Lab color space is used to perform a color search. Lab histograms use four bins in the L axis, and six bins in each of the “a” and “b” axes. For queries seeking an object where the query includes color as a factor, such as a search for an orange car of the sort depicted at 1800 in Figure 18A, a patch having the color of interest is selected and a color histogram is extracted, again using Lab color space. For convenience of illustration, in Figure 18A Lab color space is depicted on a single axis by concatenating the values. This appears as a plurality of different peaks as shown at 1810.[000190] Because colors from patches will have natural variance due to the variety of lighting conditions under which the image was captured, whereas a query color typically is a point in Lab color space with zero variance, artificial variance is added to the query to allow matching with colors that are close to the query color. This is achieved by using Gaussian blurring on the query color, 1815, which results in the variety of peaks shown at 1820 in Figure 18A.[000191] The query color, essentially a single point in Lab color space, is plotted at 1830. Again Gaussian blurring is applied, such that the variety of peaks shown at 1840 result. Then, at Figure 18C, the Gaussian plot of the patch histogram is overlaid on the Gaussian plot of the query color, with the result that a comparison of the query color and patch color can be made. Matching between the two 144- dim histograms hi and h2 is performed as:Ti 0.5 * min(h1 i], h2 i])2 / (hi i] + h2[i] ) ]Depending upon how a threshold for comparison is selected, the object that provided the patch - e.g., the car 1800 - is either determined to be a match to the query color or not.[000192] Referring next to Figure 19, a report and feedback interface to a user can be better appreciated. A query 1900 is generated either automatically by the system, such as in response to an external event, or at a preset time, or some other basis, or by human operator 1915. The query is fed to the multisensor processor 1905 discussed at length herein, in response to which a search result is returned for display on the device 1910. The display of the search results can take numerous different forms depending upon the search query and the type of data being searched. In some embodiments as discussed herein, the search results will typically be a selection of faces or objects 1915 that are highly similar to a known image, and in such instances the display 1910 may have the source image 1920 displayed for comparison to the images selected by the search. In other embodiments, the presentation of the search results on the display may be a layout of images such as depicted in Figures 16A-16C, including highlighting, dimming or other audio or visual aids to assist the user. In any case, system confidence in the result can be displayed as a percentage, 1925. If operator feedback is permitted in the particular embodiment, the operator 1930 can then confirm system-proposed query matches, or can create new identities, or can provide additional information. Depending upon the embodiment and the information provided as feedback, one or more of the processes described herein may iterate, 1935, and yield further search results.[000193] In a further aspect of the invention, the detection of objects, including the compression afforded by developing tracklets and clusters as described above, can be enhanced to enable the system to recognize identities that conform to aspecified pattern. In simplified terms, this aspect of the invention takes as its input a set of detections and / or tracklets together with a pattern definition and outputs candidate identities which match that pattern. For simplicity, hereinafter this aspect of the invention will sometimes be referred to as Discovery-by-pattern. Discovery-by- pattern may be applied with very specific patterns like “objects that appear at the coffee shop on Monday, Wednesday, and Friday”, or “Present at the coffee shop on 7th St as well as at the department store on Main”, as well as countless other examples.[000194] For example, let’s say that a private parking garage operator has observed fresh oil spots on the ramps inside of the garage on three separate days in the last month, where the security system monitors the relevant location such that an image set is available. The operator would like to know which cars showed up on those days in order to narrow down identification of the offending vehicle. As another example, a stolen credit card has been used at five convenience stores throughout town, each with CCTV footage available. This aspect of the invention can be used to identify persons who appeared at a selected number of those locations - one, some, or all - to narrow down the suspect list. Still further, a tour bus company is developing new tour routes for its tour buses, and wants to ensure that the new routes go to the places tourists want to see, again with video footage or similar image set available for each location. By defining a pattern of “persons who were at places A, B, ... n within one week of each other”, the tour company can ascertain what locations are the most popular with tourists who are on vacation for only a few days. Equally viable with this aspect of the invention is use by a bank security officer, who wants to know whether the various branches of his bank are being surveyed by malefactors where CCTV footage is available, can identify persons, vehicles, or other objects, alone or in combination, that show up either in or outside enough of the various branches to raise suspicion.[000195] Discovery-by-pattern may also be applied with more generic patterns like “multiple days in the last month” or anti-patterns like “present for longer than typical”. An example is a coffee shop owner who wants to identify repeat customers fora rewards program. They would be interested in identifying any person who comes to the shop multiple days over the span of several weeks or months. Alternatively, imagine a system that monitors cars parked on city streets in a dense urban area. Theoperator might be interested in vehicles that stay parked in the same spot for an unusually long period of time. That period of time may be specified by an operator as a pre-configured threshold or may be automatically derived from the data based on how long a typical car is parked on the street.[000196] Further examples of generic patterns are “present in at least two locations” or “NOT present in location A, but present in at least one of locations B, C, or D”. Perhaps there is an unknown person of interest suspected of being linked to particular events that happened or did not happen at these locations. This aspect of the invention can be used to develop a short list of persons matching the defined pattern. The term “pattern” as used herein can be thought of in some respects as a template or rubric against which to compare data. Also note that this aspect of the invention involves matching a pattern based on the providence of the available data, i.e. the locations from which it was gathered. Alternatively, in an embodiment, the devices from which the data was gathered could also be part of the pattern definition. As an example, a company has reasonable suspicions that a trade secret theft is occurring, and monitors firm computers to ascertain who had access at the time of the theft, or, alternatively, who had access at the time a device capable of downloading data was plugged into the computer.[000197] Whether matching against a specific pattern or a generic pattern, this aspect of the invention provides a methodology to start with the collection of identities resulting from the object detection processes described hereinabove, then apply a defined pattern and automatically find identities in the set of visual data who match that pattern. Those identities along with their pertinent sightings are then presented to an operator or downstream system with a confidence-based ranking for further analysis or decision making.[000198] To avoid requiring the reader to again go through the steps described above by which the relevant visual data is developed, and then a patten is applied, the following summary is provided and can be better appreciated in combination with Figures 20A-20C where Figure 20A illustrates in greater granularity the process steps by which candidates responsive to a supplied pattern are surfaced, while Figure 20B shows the use of similarity and confidence thresholds for the development of clusters and sightings and Figure 20C shows the capture of instancesof an identity to enable detections and the formation of tracklets. First, an object 2000 that exists in the world, such as a person, a vehicle, a ball, a purse, a hat, etc., can be an "identity”, shown at 2005, and its appearance in the real world can be thought of an “instance”, shown at 2010. When an instance of an identity is captured as an image in a camera 2015, whether video, still frame, point cloud, or other data format, that instance can be detected, shown at 2020, and, in aspects of the present invention, the characteristics of the object in that instance are used to create an embedding 2025 associated with that detection where that embedding is a N-term numeric vector that attempts to uniquely identify the object as seen in that instance. Using the characteristics captured by the associated embeddings, detections can be compared against one another to determine, within some level of confidence, whether those detections represent the same identity.[000199] If a series of detections within a data set such as a video stream are determined to be of the same instance of the object, which may be a person or a face, that series can be compressed into a tracklet 2030. In a tracklet, one or more detections from frames are selected algorithmically to represent all of the detections in the tracklet, and those representative detections are used for future comparisons rather than using each and every detection in the tracklet. To further compress the data, it is useful to compare instances of an object, such as comparing two tracklets, or an identity and a tracklet, a detection and a tracklet, or other similar combination to see whether one or more images and embeddings can faithfully represent that combination of instances. In an embodiment, the comparison is performed by determining a similarity score based on calculating the similarities of the associated embeddings. For example, if each instance of an object or person is represented by an embedding, a numeric vector, then one could calculate the cosine similarity or take the inverse of euclidean distance between two embeddings to derive a single numeric similarity score. A higher score means the two instances are more alike. In some instances, the instance may be represented by a collection of embeddings rather than a single embedding. In some embodiments, pre-filtering is used to improve tracklet quality and computational performance, as discussed in greater detail below[000200] In an embodiment, where the similarity scores developed from comparing a group of tracklets match closely enough to one another that thosetracklets are considered to represent the same identity, those tracklets are grouped into clusters 2040, as discussed at length hereinabove. Depending upon the embodiment, clustering can be performed on batches of tracklets as discussed in greater detail below.[000201] In an embodiment, similarity scores are representations of how alike one instance of a given object, e.g. a person or vehicle or other thing, is to another of that same type of object. For example, a similarity score can be determined for how alike gray sedan XA from the parking garage recording yesterday is to gray sedan XB from the parking garage recording today. The similarity score for that comparison is specific to car objects. If the object is a face instead of a gray car, the same numerical score may not imply the same level of alikeness because faces and cars might be recognized by two separate models with two separate notions of what it means to be “close” in the embedding space. To be able to reason about various types of objects (faces, cars, etc) within the same overarching framework without having to custom tune thresholds for each object type, it is useful to normalize the similarity scores based on the characteristics of the underlying embedding spaces to develop a confidence value 2045. In an embodiment, confidence values are between 0 and 1 where 1 is exactly alike and 0 is nothing alike. The use of confidence values is optional in at least some embodiments, for example where it is computationally less expensive to use similarity scores rather than confidence values. In at least such embodiments, confidence values can be backward converted to similarity values on a per-object-type basis.[000202] In an embodiment, if a series of detections in the frames of a video sequence are grouped into a single tracklet and that tracklets characteristics match the characteristics of an identity such as “gray car with dented driver’s door” within a certain threshold of similarity, which is typically application dependent as discussed above, then that is referred to as tracklet-based sighting 2050 of that identity. Sightings of faces or persons are determined in the same manner. Sightings can be determined from detections, tracklets or clusters. Because machine recognition of objects and persons is imprecise, associated with each sighting is a confidence value 2045 indicating the likelihood that it accurately identifies the specific identity it is perceived to match.[000203] In an embodiment of this aspect of the invention, one or more sightings comprise a data set which can be compared against the defined pattern, shown at 2055. The result of applying the pattern is to extract a collection of candidates with their representative images and associated embeddings as shown at 2060, with the final result being a collection of candidate identities for presentation to an operator. The operator may be a human or an automated system, and can confirm or reject the candidate images as discussed both above and hereinafter.[000204] The process described above can be represented in somewhat simplified form as shown in Figure 20B, where this aspect of the invention can be thought of broadly as three sequential stages, inputs, processes, and output, with options for pre-filtering of the data flow as discussed below. Thus, in Figure 20B, one or more tracklets 2030 (which could also be detections or any combination of the two) serve as an input and are filtered, potentially including pre-filtering, into clusters 2040. The clusters are then compared to determine sightings, indicated at search 2070, after which the pattern is applied to perform candidate extraction 2060 and provide one or more candidate identities for further review and action by an operator.[000205] As shown in Figure 20B, in some embodiments the option of prefiltering the tracklets and detections that are fed into the Clustering and Compare stages is provided. Such pre-filtering serves two purposes: computational performance and quality improvements. Less data upstream results in fewer calculations downstream, which translates directly into a lower computational cost. Perhaps more importantly, if some prior assumptions can be made to eliminate poor quality tracklets or eliminate tracklets matching certain criteria, then the ensuing stages can operate on data that is less prone to false positive results.[000206] For example, a face orientation model might generate an orientation score for each detection and tracklet to determine if the person is facing the camera or facing away. Faces pointed at the camera are easier to distinguish from one another and generally result in less confusion with other faces than faces that are pointed away from the camera where the back of the head is all that can be seen. An embodiment of a face orientation model as well as a similar face quality model are shown in Figures 21 and 22, respectively, and described below. Pre-filtering can also be of value where color is involved. For example, assume an upstream model assignsa color to each object. If the objective of the analysis of the data pertains only to colorful backpacks, then all monochrome backpacks can be eliminated by pre-filtering, resulting in higher quality matching downstream in the process.[000207] Pre-filtering can be applied using any number of criteria such as sharpness, size of the detection or tracklet in the scene, position of the detection or tracklet in the scene, duration of the tracklet, or any other characteristic of the detection or tracklet that can be extracted by a prior model or process.[000208] Referring to Figure 21 , a face orientation model can be better appreciated. DNN based face detector uses context to detect face disregarding actual face quality. It will detect a face even if the face is heavily occluded or rotated in the opposite to the camera direction. Such face detections are harder and sometimes impossible to recognize. The face rotation model of Figure 21 detects face orientation and helps filter out low quality and non-faces. Frontal faces correspond to the rotation angle 0 while backs of the head and non-faces correspond to the rotation angle 180. Faces looking to the left (with respect to the camera direction) correspond to the rotation angle -90 degrees and to the right correspond to 90 degrees.[000209] In an embodiment, part of training a face orientation model includes all backs of heads and non-faces (like hubcaps or trees that otherwise get detected as faces) in the set of examples of 180-degree orientated faces. This way the model learns to detect similar examples at run time as 180-degrees, and those detections can be filtered out as such.[000210] Image quality of objects, which for faces can be regarded as face quality, is also important and can be appreciated from Figure 22. In an embodiment, the empirical hypothesis is that in the embeddings space 2200, embeddings of nonobjects (i.e., non-faces) 2205 are localized in a non-objects region 2210 whereas embeddings of objects 2215 occur elsewhere in the space 2200. In such embodiments, embedding-based recognition of such objects or faces can be performed and is defined by the set of embeddings of non-objects / non-faces where the distance to the closest non face is mapped to the confidence that the detection is actually a face.[000211] As discussed above, tracklets provide compression of the data because they comprise a plurality of detections, with one or more images andembeddings selected for downstream representation of all of the detections in a given tracklet. In some embodiments, it is adequate to select on a single embedding from a single detection as representative of the tracklet. However, in other embodiments, it is desirable to select one or more embeddings as representative of the tracklet’s detections. For example, a tracklet of a face may involves frames where the faces enters the frame in a video, moves around, turns left and right, goes from foreground to background, etc. The cumulative detections for that face would tend to cover a fair amount of surface area in the embedding space. The embeddings of those detections can be visualized two-dimensionally in Figure 23A, where in an embedding space 2200 the black dots 2300 depict embeddings for a detection in a given tracklet A, while hollow dots 2305 depict embeddings for another detection and together the dots form a point cloud. If a single embedding is chosen to represent this tracklet, for example one of the embeddings in the middle of the black point cloud, then it would potentially have a low similarity with embeddings on the periphery of the black point cloud as well as low similarity with embeddings just outside of the black point cloud. As such, that choice of representative would potentially fail to match other embeddings (from other tracklets) that are indeed detections of the same object as the black point cloud.[000212] In an embodiment, this risk is minimized by instead choosing a set of embeddings for a tracklet such that the entire relevant area of the embedding space is reasonably covered. One approach for achieving this result is to define a distance D and for each embedding within the tracklet count the number of other embeddings within the tracklet which are within distance D and can thus be regarded as “inliers.” Each embedding can then be ranked by the number of inliers (high to low). The first one can be taken as a representative, followed by going down the list of embeddings and only adding a given embedding as a new representative if it is not an inlier of one of the previously chosen representatives. An alternative approach is to rank by embedding quality or face rotation (e.g. front-facing higher rank than side-facing). This would favor objectively “good” embeddings as representatives 2310.[000213] As shown in Figures 20A-20B, after representative embeddings are selected for the various tracklets, clustering is performed to further compress the data. As noted above, clusters are formed from groups of tracklets where those tracklets are determined as matching with one another closely enough to be considered part ofthe same identity. In an embodiment, clustering is performed on relatively small batches of tracklets that are near to one another in time and space. For example, in a video with 10,000 tracklets and a clustering batch size of 500, clusters are initially formed from the first 500 tracklets and then set aside, after which clusters are formed independently from the next 500 tracklets, again set aside, and so on. If there exists a second video or other image set, clustering of tracklets from within that second video is performed independently of clustering of tracklets from within the first video.[000214] Thus, with reference to Figure 24, in an embodiment tracklets T1 through T100 are close to each other in time, space, or both, and form a first batch 2410. From that batch, clusters are built, 2415, where the resulting number of clusters is less than or equal to the number of tracklets in the batch, for example clusters R1 through R34 as shown in Figure 24. The next batch 2420 comprises tracklets T101 through T200, with build process 2425 resulting in clusters R35 through R93, and so on with batch 2430 resulting in clusters Rk through Rk+ from build process 2435. While the example of Figure 24 illustrates a fixed length batch size, batches can be of any size and could alternatively be delineated based on a period of time or a spatial area in which the tracklets were identified, such as a cell in a geospatial grid.[000215] Clustering in batches reduces the likelihood of identity confusion i.e. grouping tracklets of two or more different identities into the same cluster. As an example, consider the case where an analysis is performed on people in a video stream and assume that in the general population there is a pair of doppelgangers (from the perspective of computer vision) in every 1000 people on average. If the chosen batch size is 500 tracklets, then there is a 50% chance that a cluster will have falsely combined a pair of doppelgangers (i.e. two different people) into the same cluster. As the batch size grows, that probability increases. Different objects will have different probabilities of identity confusion. Oranges, for example, look very similar to one another and would have a vastly higher probability of confusion. Paintings by prominent artists would presumably have vastly lower probabilities. As such, batch size should be adjusted per the type of objects being analyzed.[000216] Clustering in batches also provides an upper bound to the amount of data that must be held in memory at a given time and provides an avenue for distributed computing. Because the batches are orthogonal, they can be processedconcurrently. Further, the resulting clusters can be written to disk as an intermediate hand-off such that they need not all be retained in RAM memory space in order to be passed to the subsequent Compare stage.[000217] Clustering within each batch, i.e. processes 2415, 2425 and 2435, is performed in the same manner as described in connection with Figure 15C and associated materials, above. Note that the clustering algorithm relies on similarity scores between many pairs of tracklets within the batch. In the worst case scenario performance-wise, each tracklet becomes a new key in a new cluster. T 1 is chosen as the first key in the first cluster with no similarity score needed. T2 is compared to T 1 with a similarity score. It is added as a new key in a second cluster. T3 is compared to T 1 and T2 and so on. Thus for N tracklets in a batch, the max number of similarity scores needed is:[000218] The Compare stage compares clusters to the full corpus of tracklets from which the clusters were formed. As illustrated in the clustering algorithm above, clusters are a form of compression and thus each cluster is represented by only a subset of the tracklets which are considered members of the cluster. By comparing clusters to the full corpus of tracklets rather than simply comparing clusters to other clusters, we avoid magnifying the loss from that compression.[000219] For each cluster, we calculate similarity scores between the cluster’s key tracklets and the full corpus of tracklets in the analysis. See the section below for similarity score calculation. If a given similarity score is above a predetermined threshold, then we register it as a Sighting for the cluster.[000220] In the best case scenario performance-wise, every tracklet is part of the same cluster. T1 is chosen as the first key in the first cluster with no similarity score needed. T2 is compared to T1 with a similarity score and determined to be similar enough that it is included in the cluster as a non-key. T3 is then compared only to T2 and also found to be similar enough to be included in the cluster as a non-key,and so on. Rather than being O(N2) complexity as in the worst case, the max number of similarity scores in the best case becomes O(N).[000221] In realistic scenarios, it is usually advantageous in at least some embodiments to employ optimization techniques to the calculation of similarities between tracklets. Let’s say that all tracklets are represented by one or more embeddings. A simple optimization on the brute force method of calculating pairwise similarities would be to put all embeddings from the tracklets in the batch into a list of length N and employ an N x N matrix-based similarity function on specialized hardware such as a GPU. Since it is known that only similarities above a certain threshold are of interest, many of the resulting similarities can be thrown out and those pairs can be considered dissimilar such that they belong in different clusters. The remaining similarities can be used as a simple lookup for the remainder of the clustering algorithm.[000222] Depending upon the embodiment, further optimizations can involve indexing techniques which are designed to determine very quickly which other tracklets are within a similarity threshold of any given tracklet Tn. This and other optimization techniques around similarity scores are discussed in greater detail in the Compare stage description, below.[000223] The resulting clusters from this stage can be thought of as proto candidates. Given perfectly tuned thresholds in the algorithm, each cluster will contain multiple tracklets of the same identity and be unique amongst clusters in the given batch. Clusters are not necessarily unique across batches, however. In an embodiment, cross-batch uniqueness is achieved in the Candidate Extraction stage after the population of these proto-candidates has shrunk significantly to only those which match the given pattern definition.[000224] Next in the process is the Compare step, with reference to Figures 25 and 26. In an embodiment, the Compare stage compares clusters to the full corpus of tracklets from which the clusters were formed. As illustrated in the clustering algorithm above, clusters are a form of compression and thus each cluster R1 in Figure 25 is represented by only a subset of the member tracklets, shown at T1 , T3 and T4 with only T 1 and T4 as key tracklets. By comparing each of the cluster’s key trackletsto the full corpus of tracklets T1 through TN rather than simply comparing clusters to other clusters, the system avoids magnifying the loss from that compression.[000225] For each cluster, we calculate similarity scores between the cluster’s key tracklets and the full corpus of tracklets in the analysis, where the calculation of similarity score is discussed below. If a given similarity score is above a predetermined threshold, then a Sighting is registered for the cluster. Still referring to Figure 25, the key tracklets within Cluster R1 are compared with all N tracklets T1 through TN in the full corpus of data. R1 has two key tracklets T 1 and T4. T1 satisfies the similarity threshold with T1 (same tracklet), T3, T4, and T8. T4 satisfies the similarity threshold with T1 , T4 (same tracklet), and T5. However, T4 does not satisfy the similarity threshold with T3 even though T3 is part of the same cluster. Due to the triangle inequality, this is possible and it means that T3 is ONLY similar to T1 . In this example, sightings are registered forT1 , T3, T4, T5, and T8 with Cluster R1 . Because T3 is now registered as a sighting, there is no need to otherwise keep track of T3 as part of Cluster R1 . The connection to T1 and T4 are retained as keys for the sake of presenting a condensed representation of the cluster to downstream processes or visualizations.[000226] Referring next to Figure 26, as with the Clustering stage, cluster-to- tracklet comparisons are done in batches: batches of clusters compared to batches of tracklets. In Figure 26, Clusters R1 , R2, and R3 are the first batch. For the first batch comprising R1-R3, similarity comparisons of RTs key tracklets are made with all N tracklets, the same for R2, and the same for R3. This is repeated for the second batch containing R4, R5, and R6 and so on. In at least some embodiments, each batch of clusters is orthogonal to other batches of clusters. As such, they can be processed concurrently. Optionally, the resulting Sightings for a batch of clusters can be written to disk as an intermediate step such that they need not all be retained in RAM memory.[000227] At the end of the Compare stage, we have all sightings for each cluster that satisfy our similarity threshold. There is still duplication of identities (i.e. one identity represented by multiple clusters), where each duplicate cluster has different keys. De-duplication will be handled in the Candidate Extraction stage.[000228] As described above, each cluster is compared to each tracklet in the corpus of data to obtain a similarity score. Scores above a predeterminedthreshold indicate a match and the tracklet is noted as a Sighting for the cluster. With that goal in mind, there are a number of techniques to obtain these similarity scores ranging from lossless yet inefficient brute force to potentially lossy but efficient indexing. The choice of technique can depend upon the particular application or specific implementation.[000229] For the following methodologies, consider a set of M clusters and N tracklets. Each cluster m, has a set of keys. Each key is represented by one or more embeddings. Thus, each cluster is represented by a set of embeddings e(mi). Each tracklet nj is also represented by a set of one or more embeddings e(nj).[000230] A naive brute force solution would compare all embeddings from all clusters with all embeddings from all tracklets. Imagine a function f that compares embeddings from a cluster with embeddings from a tracklet and outputs a singular similarity score. Such a function would be applied to all pairwise combinations of cluster to tracklet:[000231] The function f would need to consider several pairwise embedding comparisons to arrive at a singular number. It could take the min or max of those comparisons or use some other aggregation operation. If the exact similarity score were not needed downstream, then the function f could potentially take shortcuts and abort calculations as soon as it identified a comparison that satisfied the predetermined similarity threshold or (the opposite) precluded it from satisfying said threshold. This would save some computational cost, but not fundamentally change the order of complexity of the problem.[000232] Note that the set of embeddings contained within the keys of the clusters is a subset of the embeddings contained among the tracklets they are being compared against. As such, each pair of embeddings will be compared twice and the score could potentially be retained and re-used for the second comparison. This would take the computational cost from M x N to (M x N) / 2 plus the added expense of consuming additional memory space and lookup time.[000233] For certain pattern definitions, entire swaths of similarity calculations could be eliminated with the foreknowledge that they have no bearing on whether or not a candidate matches the pattern. For example, imagine a dataset withtwo videos V1 and V2 and a pattern of “appears in multiple videos”. Any cluster derived from V1 need not be compared with any other tracklets in V1 and vice versa for clusters derived from V2. In this scenario, assuming the videos have equal numbers of clusters and tracklets, this optimization decreases computational cost to 2 x M / 2 x N / 2 = 2 x (M x N) / 4 =(M x N) / 2 . One major tradeoff of this second optimization is that it limits the applicability of the Compare stage results to very specific patterns. If subsequent analyses were executed with different pattern definitions, these results would have to be entirely recalculated whereas without this optimization, they could be reused.[000234] Combining these two optimizations (if circumstances allow) would reduce cost to (M x N) / 4. Regardless, the big-0 complexity remains O(MN) with the brute force approach.[000235] In an alternative embodiment, an indexing scheme can be used which first organizes the embeddings from all tracklets (and thereby all keys from all clusters) in memory such that subsequent neighborhood searches are very quick. In this case, the problem is mapped to one of an efficient k nearest neighbor search problem. Various approximate and exact k nearest neighbor search algorithms are known to those skilled in the art which return the set of k neighbors of a point in sorted distance order. One such algorithm is the DCI-KNN algorithm. Other alternatives include a k-d tree, vantage point tree, etc. In an embodiment which is implemented with such an algorithm, an index is built for all of the N tracklets. Memory constraints will determine how large N can be, or more specifically, the largest number N such that N * [number of embeddings per tracklet] * [memory footprint per embedding] * [indexing overhead per embedding] < [available memory],[000236] Rather than comparing each cluster to all tracklets as in the brute force approach, this technique involves looking up the k nearest neighbors for each embedding in a given cluster. Given a cluster m, with embeddings e(rni), an index of all tracklet embeddings E={e(nj), 0<j<N} and a predetermined minimum similarity threshold S, an e exemplary algorithm is:[000237] Function ClusterSimilarEmbeddingsWithlndexing[000238] / / We assume that for each embedding in E there is a back- reference to the tracklet it is associated with[000239] / / The function similarity(e1 , e2) yields the similarity between two embeddings[000240] Let T = {} / / This will contain the set of tracklets within similarity threshold S of the cluster[000241] Let TS = {} / / This contains the corresponding best similarity scores for each tracklet in T[000242] For each embedding e, in e(mi):[000243] K = set of k nearest embeddings of e, in E[000244] Siast = similarity(Klast, e,)[000245] If s^st >- S, then K - 2*k nearest embeddings of e, in E; Repeat, each time doubling k, until we reach a set K such that s / asf< S[000246] For each embedding Ki in K:[000247] s = similarity(Ki, ei)[000248] If s >= S:[000249] Add the tracklet associated with K, to T and updateTS with similarity score s if s is better than the existing similarity score for this tracklet in TS[000250] For each tracklet in T, add a sighting to cluster m, with the corresponding similarity score from TS[000251] If k is chosen such that there is generally only one nearest neighbor lookup needed per embedding in order to reach deep enough into the neighbors to dip below the similarity threshold, then we can write the simplify the complexity of this algorithm as follows:[000252] O(xMk) where x is the average number of embeddings per cluster, M is the number of clusters and k is much smaller than xM. If each tracklet resulted in a single cluster, we could alternatively write as O(Nk) where N is the number of tracklets.[000253] Depending upon the embodiment, other considerations can also play a role. Lack of merging duplicate clusters at this stage may in some implementations be a potential disadvantage. For example, a pattern definition is “appears on three separate days’’ and there are four tracklets of the same identity: T- Monday, T-Tuesday-1 , T-Tuesday-2, and T-Wednesday. Each of the Tuesday tracklets is the key for a separate cluster, R1 and R2 respectively. As measured by apredetermined similarity threshold S, assume for exemplary purposes that one of them is similar to the T-Monday tracklet, but the other is similar to the T-Wednesday tracklet, as shown in Figure 27. Without a merge, each cluster will only match with one other day in the Compare stage and thus neither will be deemed to match the “appears on three separate days” pattern. However, keeping clusters separate becomes an advantage when T-Tuesday-1 and T-Tuesday-2 are in fact NOT the same identity. In such a case, a false positive result is avoided. Depending upon the application, it can be more important to avoid a false positive than to miss a positive; the converse may also be true for other applications. In a still further alternative, an embodiment can include a Cluster Merge stage that merges clusters across batches, with appropriate caution to ensure that a given cluster does not forever expand to include other clusters near to it, then more clusters near to those clusters, and so on.[000254] The output of the clustering and comparison stages comprises structured clusters with sightings derived from the original unstructured visual data, as shown at 2800 in Figure 28. Each sighting has an associated numerical confidence match value and thus can be treated as part of the signal for its associated cluster. In the third stage shown in Figure 20B and shown at 2805 in Figure 28 and in greater detail in Figure 29 discussed below, an attempt is made to match that signal with a given pattern definition P. Clusters that satisfy that definition are retained as candidates, given a rank and sorted according to rank, 2810, deduplicated 2815, and then denominated candidates ready for review, 2820, and sent to a downstream process or user for review and curation. It will be appreciated that the steps involving checking whether any of the clusters satisfy the pattern and ranking can be performed on a cluster-by-cluster basis in an orthogonal manner.[000255] Referring to Figure 29, a comparison is made at 2900 between each cluster R, indicated at 2905, and the given pattern P, 2910, to determine whether the cluster’s sightings satisfy the pattern. Clusters that do not satisfy the pattern are discarded, 2915, and the next cluster is evaluated. Clusters that do satisfy the pattern P advance to having their rank calculated, 2920, and are added as a candidate at 2925.[000256] The manner in which the result of step 2900 is determined will vary depending on the nature of the pattern, and in some cases, additional statistics 2930 from the entire set of clusters will be n^<"iad as a prerequisite input. *£,*‘Monday-Wednesday-Friday pattern as an example. If a given cluster has at least one sighting on each of the three days, it at least some embodiments it can could be said that the cluster satisfies the pattern and is designated a candidate to retain. In an embodiment, it would be reasonable to rank it by taking the most-similar sighting on each of the three days respectively then choosing the minimum similarity among those.[000257] Next, consider as an example the pattern of a person of interest who “shows up at two or more locations throughout town” where there is data for a total of five locations. In this case, a person who shows up at three, four, or all five locations is no better a candidate than a person who shows up at only two. As such, in an embodiment each cluster can be ranked based on the highest similarity between sightings at any two different locations.[000258] Finally, in an embodiment consider the exemplary pattern “more frequent than normal”. To evaluate clusters in this case, there must be some definition of “normal”. Here, each cluster’s sightings can be considered a two-dimensional timebased signal where the similarity score of each sighting is the amplitude. That signal could be converted to the frequency domain and all frequencies above a certain amplitude A extracted with the median frequency F arbitrarily defined as “normal”. Any cluster which exhibits a frequency higher than F with amplitude greater than A would be considered a candidate. It would be reasonable to take the highest amplitude A of each candidate’s frequencies above F as the rank in this case.[000259] At this point in the process, a plurality of clusters have each been identified as candidates because they have satisfied the given pattern in an appropriate manner. Each has a rank and can be sorted from “best” to “worst” candidate based on that rank. However, since the earlier clustering stage made no effort to de-duplicate clusters, it is likely that at least some candidate clusters have one or more duplicates in this set. A process for removing such duplicates in shown in Figure 30. To deduplicate, each candidate C is taken in ranked order at 3005 and compared against a running set of deduplicated candidates D, shown at 3010. If a duplicate, the process loops to evaluate the next candidate C. If a given candidate C is not determined to be a duplicate of the current set D, shown at 3015, then at 3020 a check is made to see if there are more candidates D against which candidate C isto be checked. If there are, the process loops for the next check. If there no more, candidate C is added to that set of deduplicated candidates D, shown at 3025. The specific methodology for comparing two candidates to check for duplication can vary with embodiment. One simple approach is to do a pairwise comparison of the embeddings from candidate CTs tracklets with the embeddings from candidate C2’s tracklets and take the pair with the max similarity. If that max similarity is above a predetermined duplication threshold, then C1 and C2 are considered duplicates. An alternative embodiment is to consider the two candidates duplicates if all of the pairwise comparisons are above a given threshold. Upon the completion of deduplication, the result is a mostly unique list of candidate clusters ranked in order of how likely they are to have matched the given pattern. At this point, they are ready for presentation to either a human operator or a downstream process for further review and curation.[000260] It will be appreciated that the candidates which result from the foregoing steps are suggestions rather than absolute truth, but represent identities in the data set that are likely to satisfy a given pattern. To facilitate further review and curation, particularly by a human operator, the candidates will preferably be presented in a way that permits rapid access to enough relevant data that an operator can quickly assess each candidate and decide whether to reject or confirm each suggestion.[000261] One embodiment of such a view is illustrated in Figure 31. In a display 3100, the leftmost column 3105 is a list of candidates in ranked order (highest to lowest rank). In this embodiment, each candidate is represented in column 3100 by a single representative thumbnail image 3110A-n which corresponds to a key tracklet, sometimes referred to as a reference tracklet, from which the candidate was formed, In alternative embodiments, multiple views of each candidate can be presented.[000262] By selecting the thumbnail image of a candidate, the middle portion 3115 of the display 3100 populates with a representative image 3120A-n from each of the sightings for that candidate. Each sighting is represented by a thumbnail image in a similar manner to the images in column 3105, and the sightings can be arranged by their compliance with the parameters of the pattern, e.g. “Monday” in a first row, “Wednesday” in a second row and “Friday” in a third row represent a pattern of“appears on Monday, Wednesday and Friday.” By selecting a specific sighting from among the images 3120A-n, additional contextual information is displayed in one or more panels 3125A-n that comprise the right column 3130. That contextual information can comprise still frame images, links to the points in videos to which the sighting corresponds, maps showing the geographical location, or other information helpful to the operator or a downstream process. One goal in the presentation of the images in Figure 31 is to provide the operator or downstream process with sufficient information that a given sighting can be confirmed or rejected as being the same identity as shown in the associated thumbnail image in column 3105.[000263] Confirming means that the sighting is, in the perception of the operator or other downstream process, the same identity as the reference tracklet for the candidate. Rejecting means that it is not, in the perception of the operator or downstream process, not the same identity. The view 3100 can adapt to feedback and highlight or hide candidates per the user’s actions. For example, if the pattern matched in the diagram above was “Monday, Wednesday, & Friday” and the user rejects both sightings in the “Friday” bucket, then this candidate no longer satisfies the pattern and can be hidden. Alternatively, if the user confirms a single sighting from each of the “Monday”, “Wednesday”, and “Friday” buckets respectively, then the candidate definitely satisfies the pattern and can be highlighted and / or pinned on the view.[000264] As a user or downstream process curates the results of the analysis, they may reach the point at which the candidate list is exhausted. In some scenarios, all of which are optional, they may want to dive deeper into the dataset at this point. As such, they can adjust the various thresholds like minimum tracklet similarity that were used in the three stages of analysis and cause a re-analysis of the underlying data set. In at least some embodiments the iterated analysis can take into account any confirmations and rejections that were made and count those as 100% and 0% similarity in the Clustering and Compare stages.[000265] In a still further option, available at any point upon selection by the operator, the pattern can be altered and applied to the same data, i.e., keeping the results of the clustering and compare stages but applying different pattern parameters. The clusters and sightings from the first two stages will still be valid as they weregenerated in a manner agnostic to the pattern being matched. It will thus be appreciated that the system and processes described herein can discover identities from unstructured visual data based on a supplied pattern definition. The various embodiments of the present invention afford re-use of processed data, concurrent execution on a distributed system, and interactions from a user or downstream process which can be applied to the results for further analysis or curation purposes.[000266] From the foregoing it can be appreciated by those skilled in the art that, in one aspect, the present invention comprises a method for recognizing objects of interest that conform to a specified pattern comprising the steps of providing a pattern wherein an object of interest is detected at one of a group comprising either a location of interest, a specified period of time, or a specified device, performing detections of a class of objects on images from one or more data sources, combining into a first tracklet a first set of detections of the class of objects that meet a first predetermined threshold of spatial or temporal proximity or device, selecting a key image as representative of the first tracklet, repeating the combining and selecting steps to develop n tracklets each of which meets the first predetermined threshold of spatial and temporal proximity but differs in space or time from the previous tracklets and is represented by a key image, applying the pattern to the key images to identify candidates matching the pattern to at least a predetermined pattern threshold, and providing to a user the identified candidates. It can further be appreciated that, in some embodiments, the data sources comprise images, which can be video data among other forms of media. In at least some such embodiments, each key image has associated therewith an embedding.[000267] Further, in some embodiments a key image may comprise plural images, each associated with an embedding, where the distance between the embeddings for the key images is less than a second predetermined threshold that is smaller than the first predetermined threshold. In some embodiments, the method can include assigning a confidence level to the identified candidates based at least in part on embedding distance. The confidence level can in some embodiments be assigned to the identified candidates based at least in part on linear assignment techniques, or can be assigned to the identified candidates based at least in part on conditionalprobability distribution functions of embedding distance. Further, the confidence level can be varied to balance accuracy versus data compression.[000268] A related aspect of the invention can comprise further compression of the data by augmenting the method of combining detections into tracklets with the additional steps of aggregating into a first cluster a group of tracklets wherein each of the key images is characterized by an embedding, comparing the embedding for a key image representative of a first tracklet to the embedding for a key image of a second tracklet, combining the tracklets into the first cluster if the comparison meets a predetermined criteria, repeating the comparing and combining steps for a third through nth tracklet, selecting a cluster key image as representative of the cluster, repeating each of the comparing, combining and selecting steps until a plurality of clusters results, each having a representative cluster key image associated therewith, and providing the cluster key images as the key images for applying the pattern.[000269] A related aspect of the invention comprises a system for recognizing objects of interest that conform to a specified pattern comprising at least one processor and associated memory, an input device for receiving input from a user and a display, wherein a pattern for detecting an object of interest at one of a group comprising either a location of interest, a specified period of time, or a specified device is stored in memory, the at least one processor executing instructions to perform detections of a class of objects on images from one or more data sources, the at least one processor further executing instructions to combine into a first tracklet a first set of detections of the class of objects that meet a first predetermined threshold of spatial or temporal proximity or device, the at least one processor further executing instructions for selecting a key image as representative of the first tracklet, the at least one processor further executing instructions to repeat the combining and selecting in order to develop n tracklets each of which meets the first predetermined threshold of spatial and temporal proximity but differs in space or time from the previous tracklets and is represented by a key image, the at least one processor further executing instructions to apply the pattern to the key images to identify candidates matching the pattern to at least a predetermined pattern threshold, and the processor further executing instructions to provide to a user the identified candidates.It will be appreciated by those skilled in the art that the compression resulting fromcombining select detections into tracklets and select tracklets into clusters, for the overall purpose of distilling the unstructured data into a presentation that can be quickly assessed by a user, can be combined in a single embodiment where the method for recognizing objects of interest that conform to a specified pattern comprises the steps of providing a pattern wherein an object of interest is detected at one of a group comprising either a location of interest, a specified period of time, or a specified device, performing detections of a class of objects on images from one or more data sources, each detection being characterized by an embedding, combining into a first tracklet a first set of detections of the class of objects that meet a first predetermined threshold of spatial or temporal proximity or device based on a comparison of embeddings, selecting at least one detection as a key image representative of the first tracklet, repeating the combining and selecting steps to develop a plurality of tracklets each of which meets the first predetermined threshold of spatial and temporal proximity but differs in space or time from the previous tracklets and wherein each tracklet is represented by a key image, comparing the embedding for a key image representative of a first tracklet to the embedding for a key image of a second tracklet, aggregating the tracklets into a cluster if the comparison meets a predetermined criteria, repeating the comparing and aggregating steps for a third through nth tracklet, selecting a cluster key image as representative of the cluster based on an analysis of the embeddings, repeating each of the comparing, aggregating and selecting steps on tracklets until a plurality of clusters results, each having a representative cluster key image associated therewith, and applying the pattern to the cluster key images to identify candidates matching the pattern to at least a predetermined pattern threshold, and providing to a user the identified candidates. In some embodiments at least some cluster key images are assigned a confidence level based on the results of applying the pattern and the cluster key images are displayed to a user according to their assigned confidence levels.[000270] It will also be appreciated that, typical of most such inventions, the various functions are coded as instructions. In an aspect of the invention, when executed the stored instructions cause the host system comprising at least one processor and associated data storage to identify one or more objects of interest from among a population of objects detected in a data stream, to receive in the data storageat least some of the data stream, to detect one or more objects in the data stream, to generate an embedding which characterizes at least some invariant features of the detected objects, to combine into a first tracklet a first set of detections of the objects that, based on a comparison of embeddings, meet a first predetermined threshold of spatial proximity, temporal proximity, or source device, to select at least one detection as a tracklet key image representative of the first tracklet, to repeat the combining and selecting steps to develop a plurality of tracklets each of which meets the first predetermined threshold of spatial and temporal proximity but differs in space, time or source device from the previous tracklets and wherein each tracklet is represented by a tracklet key image, to compare the embedding for a first tracklet key image to the embedding for a second tracklet key image, to aggregate the tracklets into a cluster if the comparison meets a predetermined clustering criteria, to repeat the comparing and aggregating steps for a third through nth tracklet key images, to select a cluster key image as representative of the cluster based on an analysis of the embeddings of the tracklet key images, to repeat each of the compare, aggregate and select steps on the tracklets until a plurality of clusters results, each cluster having a representative cluster key image associated therewith, to apply the pattern to the cluster key images to identify candidates matching the pattern to at least a predetermined pattern threshold, and to provide to a user the identified candidates.[000271] In some embodiments described herein, plural instances may implement components, operations, or structures described as a single instance and vice versa. Likewise, individual operations of one or more embodiments may be illustrated and described collectively where, alternatively, one or more of the individual operations may be performed concurrently, and the operations may be performed in an order different than that illustrated. Structures and functionalities presented as separate components in example configurations may be implemented as a combined structure or single component. Similarly, structures and functionalities presented as single components or structures, such as “a key image” or other uses of “a” to describe an object orfunction, may be implemented as a one or more structures or components. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter herein.[000272] Embodiments described herein as including components, modules, mechanisms, functionalities, steps, operations, or logic may comprise either software modules (e.g., code embodied on a machine-readable medium or in a transmission signal) or hardware modules. A hardware module comprises a tangible unit configured or arranged to perform the requisite operations. In example embodiments, one or more computer systems (e.g., a standalone, client or server computer system, colocated or remote from one another) or one or more hardware modules of a computer system (e.g., a processor or a group of processors) may be configured either by software (e.g., an application or application portion) or as a hardware module that operates to perform certain steps or operations as described herein.[000273] In various embodiments, a hardware module may comprise dedicated circuitry or logic that is permanently configured (e.g., as a special-purpose processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)) to perform certain operations. A hardware module may also comprise programmable logic or circuitry (e.g., as encompassed within one or more general-purpose processors or other programmable processors) that is temporarily configured by software to perform certain operations. It will be appreciated that the implementation of a hardware module in a particular configuration may be driven by cost and time considerations.[000274] Embodiments in which one or more hardware modules are temporarily configured (e.g., programmed), each of the hardware modules need not be configured or instantiated at any one instance in time. For example, where the hardware modules comprise a general-purpose processor and / or a graphics processor configured using software, one or more such processors may be configured as respective different hardware modules at different times. Software may accordingly configure a processor, for example, to constitute a particular hardware module at one instance of time and to constitute a different hardware module at a different instance of time.[000275] The one or more processors may also operate to support performance of the relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS). For example, at least some of the operations may be performed by a group of computers (as examples of machines including processors),these operations being accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application program interfaces (APIs).) The performance of certain of the operations may be distributed among the one or more processors, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the one or more processors or processor-implemented modules may be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other example embodiments, the one or more processors or processor-implemented modules may be distributed across a number of geographic locations.[000276] Some portions of this specification are presented in terms of equations, algorithms or symbolic representations of operations on data stored as bits or binary digital signals within a machine memory (e.g., a computer memory). These equations, algorithms or symbolic representations are examples of techniques used by those of ordinary skill in the machine learning and data processing arts to convey the substance of their work to others skilled in the art. As used herein, an “algorithm” is a self-consistent sequence of operations or similar processing leading to a desired result. In this context, algorithms and operations involve manipulation of physical quantities. Typically, but not necessarily, such quantities may take the form of electrical, magnetic, or optical signals capable of being stored, accessed, transferred, combined, compared, or otherwise manipulated by a machine. It is convenient at times, principally for reasons of common usage, to refer to such signals using words such as “data,” “content,” “bits,” “values,” “elements,” “symbols,” “characters,” “terms,” “numbers,” “numerals,” or the like. These words, however, are to be understood merely as convenient labels associated with appropriate physical quantities.[000277] Unless specifically stated otherwise, terms such as “processing,” “computing,” “calculating,” “determining,” “presenting,” “displaying,” “generating”, “emulating” or the like may refer to actions or processes of a machine (e.g., a computer) that manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or a combination thereof), registers, or other machine components that receive, store, transmit, or display information.[000278] As used herein any reference to “one embodiment” or “an embodiment” means that a particular element, feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. The phrase “in an embodiment” used in various places in the specification do not necessarily all refer to the same embodiment.[000279] As used herein, the terms “comprises,” “comprising,” “includes,” “including,” “has,” “having” or any other variation thereof, are intended to cover a nonexclusive inclusion. For example, a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Further, unless expressly stated to the contrary, “or” refers to an inclusive or and not to an exclusive or. For example, a condition A or B is satisfied by any one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).[000280] Having fully described a preferred embodiment of the invention and various alternatives, those skilled in the art will recognize, given the teachings herein, that numerous further alternatives and equivalents exist which do not depart from the invention. Thus, while particular embodiments and implementations have been illustrated and described, it is to be understood that the invention is not limited to the precise embodiments, structures and configurations disclosed herein but is to be limited only by the appended claims.

Claims

We claim:

1. A method for recognizing objects of interest that conform to a specified pattern comprising the steps of: providing a pattern wherein an object of interest is detected at one of a group comprising either a location of interest, a specified period of time, or a specified device, performing detections of a class of objects on images from one or more data sources, combining into a first tracklet a first set of detections of the class of objects that meet a first predetermined threshold of spatial or temporal proximity or device, selecting a key image as representative of the first tracklet, repeating the combining and selecting steps to develop n tracklets each of which meets the first predetermined threshold of spatial and temporal proximity but differs in space or time from the previous tracklets and is represented by a key image, applying the pattern to the key images to identify candidates matching the pattern to at least a predetermined pattern threshold, and providing to a user the identified candidates.

2. The method of claim 1 wherein the data sources comprise images.

3. The method of claim 1 wherein the data sources comprise video data.

4. The method of claim 1 wherein each key image has associated therewith an embedding.

5. The method of claim 1 wherein a key image comprises plural images, each associated with an embedding, where the distance between the embeddings for the key images is less than a second predetermined threshold that is smaller than the first predetermined threshold.

6. The method of claim 4 wherein a confidence level is assigned to the identified candidates based at least in part on embedding distance.

7. The method of claim 4 wherein a confidence level is assigned to the identified candidates based at least in part on linear assignment techniques.

8. The method of claim 4 wherein a confidence level is assigned to the identified candidates based at least in part on conditional probability distribution functions of embedding distance.

9. The method of claim 4 wherein confidence level is can be varied to balance accuracy versus data compression.

10. The method of claim 1 further comprising the steps of aggregating into a first cluster a group of tracklets wherein each of the key images is characterized by an embedding, comparing the embedding for a key image representative of a first tracklet to the embedding for a key image of a second tracklet, combining the tracklets into the first cluster if the comparison meets a predetermined criteria, repeating the comparing and combining steps for a third through nth tracklet, selecting a key image as representative of the cluster, repeating each of the comparing, combining and selecting steps until a plurality of clusters results, each having a representative cluster key image associated therewith, and providing the cluster key images as the key images for applying the pattern.

11. A system for recognizing objects of interest that conform to a specified pattern comprising: at least one processor and associated memory, an input device for receiving input from a user and a display, wherein a pattern for detecting an object of interest at one of a group comprising either a location of interest, a specified period of time, or a specified device is stored in memory, the at least one processor executing instructions to perform detections of a class of objects on images from one or more data sources,the at least one processor further executing instructions to combine into a first tracklet a first set of detections of the class of objects that meet a first predetermined threshold of spatial or temporal proximity or device, the at least one processor further executing instructions for selecting a key image as representative of the first tracklet, the at least one processor further executing instructions to repeat the combining and selecting in order to develop n tracklets each of which meets the first predetermined threshold of spatial and temporal proximity but differs in space or time from the previous tracklets and is represented by a key image, the at least one processor further executing instructions to apply the pattern to the key images to identify candidates matching the pattern to at least a predetermined pattern threshold, and the processor further executing instructions to provide to a user the identified candidates.

12. A method for recognizing objects of interest that conform to a specified pattern comprising the steps of: providing a pattern wherein an object of interest is detected at one of a group comprising either a location of interest, a specified period of time, or a specified device, performing detections of a class of objects on images from one or more data sources, each detection being characterized by an embedding, combining into a first tracklet a first set of detections of the class of objects that meet a first predetermined threshold of spatial or temporal proximity or device based on a comparison of embeddings, selecting at least one detection as a key image representative of the first tracklet, repeating the combining and selecting steps to develop a plurality of tracklets each of which meets the first predetermined threshold of spatial and temporal proximity but differs in space or time from the previous tracklets and wherein each tracklet is represented by a key image, comparing the embedding for a key image representative of a first tracklet tothe embedding for a key image of a second tracklet, aggregating the tracklets into a cluster if the comparison meets a predetermined criteria, repeating the comparing and aggregating steps for a third through nth tracklet, selecting a cluster key image as representative of the cluster based on an analysis of the embeddings, repeating each of the comparing, aggregating and selecting steps on tracklets until a plurality of clusters results, each having a representative cluster key image associated therewith, and applying the pattern to the cluster key images to identify candidates matching the pattern to at least a predetermined pattern threshold, and providing to a user the identified candidates.

13. A non-transitory computer readable storage medium comprising stored instructions to identify one or more objects of interest from among a population of objects detected in a data stream, the instructions when executed causing at least one processor and data storage in communication therewith to: receive in the data storage at least some of the data stream, detect one or more objects in the data stream, generate an embedding which characterizes at least some invariant features of the detected objects, combine into a first tracklet a first set of detections of the objects that, based on a comparison of embeddings, meet a first predetermined threshold of spatial proximity, temporal proximity, or source device, select at least one detection as a tracklet key image representative of the first tracklet, repeat the combining and selecting steps to develop a plurality of tracklets each of which meets the first predetermined threshold of spatial and temporal proximity but differs in space, time or source device from the previous tracklets and wherein each tracklet is represented by a tracklet key image, compare the embedding for a first tracklet key image to the embedding for a second tracklet key image,aggregate the tracklets into a cluster if the comparison meets a predetermined clustering criteria, repeat the comparing and aggregating steps for a third through nth tracklet key images, select a cluster key image as representative of the cluster based on an analysis of the embeddings of the tracklet key images, repeat each of the compare, aggregate and select steps on the tracklets until a plurality of clusters results, each cluster having a representative cluster key image associated therewith, apply the pattern to the cluster key images to identify candidates matching the pattern to at least a predetermined pattern threshold, and provide to a user the identified candidates.

14. The method of claim 12 wherein at least some cluster key images are assigned a confidence level based on the results of applying the pattern.

15. The method of claim 12 wherein the cluster key images are displayed to a user according to their assigned confidence levels.

Images