Online incremental real-time learning for tagging and labeling data streams for deep neural networks and neural network applications

Abstract

To provide a method and system for tagging sequences of images.SOLUTION: A smart tagging utility automatically learns tags and tag images using a feature extraction module and a fast learning classifier module. The feature extraction module and fast learning classifier module can be implemented as an artificial neural network that associates labels with features extracted from images and tags similar features or other images from the image by the same label. The smart tagging utility can further learn from user adjustments to suggested tagging. This reduces tagging time and errors.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] Cross - reference to Related Patent Applications This application claims the benefit of priority under 35 U.S.C. § 119(e) of U.S. Patent Application No. 62 / 472,925, filed on Mar. 17, 2017, the content of which is incorporated herein by reference in its entirety.

Background Art

[0002] Conventional deep neural networks (DNNs) that include convolutional neural networks (CNNs), which include many layers of neurons arranged between an input layer and an output layer, require thousands or millions of iterative cycles for training on a particular dataset. Before this training can occur, all images within the dataset must be tagged by a human user. The tagging process may involve tagging the entire image as well as classifying or labeling individual regions of each image as specific objects for the classification and detection / segmentation of individual objects.

[0003] Conventional image tagging is a slow and time - consuming process. A human looks at an image on a computer, tablet, or smartphone, identifies one or more objects in the image, and tags them with descriptive tags (e.g., "tree," "house," or "car"). The major difficulties in manually tagging objects of interest include slowness and susceptibility to human errors caused by distraction and fatigue. These problems give rise to two types of issues: that data preparation for training can take an unacceptably long time outside of academic settings, and that the quality of tagging directly affects the quality of subsequent learning such that inappropriately tagged data can prevent the DNN from achieving acceptable performance criteria.

Summary of the Invention

[0004] Embodiments of the technology of the present invention include a method and system for tagging a sequence of images. An exemplary method includes tagging, by a user via a user interface, a first instance of a representation of an object in a first image of a sequence of images. At least one processor learns the representation of the object tagged by the user in the first image and tags a second instance of the representation of the object in the sequence of images. The user performs an adjustment of the tag and / or position of the second instance of the representation of the object created by the processor. Then the processor tags a third instance of the representation of the object in the sequence of images based on the adjustment.

[0005] The second instance of the representation of the object may be in the first image of the sequence of images or in another image of the sequence of images.

[0006] In some cases, the user may perform an adjustment of the tag and / or position of the third instance of the representation of the object created by the processor, and the processor tags a fourth instance of the representation of the object in the sequence of images based on the adjustment of the tag and / or position of the third instance of the representation of the object.

[0007] An example of this method can also include classifying via a fast learning classifier executed on a processor, and the representation of the object is the representation of the object tagged by the user in the first image. In this case, tagging the third instance of the representation of the object may include extracting a convolutional output representing the features of the third instance of the representation of the object using a neural network operably connected to the fast learning classifier. The fast learning classifier classifies the third instance of the representation of the object based on the convolutional output.

[0008] An example of this method may also include tagging a second instance of a representation by extracting a convolutional output representing the features of the second instance of the object's representation using a neural network executed on a processor, and classifying the second instance of the object's representation based on the convolutional output using a classifier operably coupled to the neural network.

[0009] A system for tagging a sequence of images may include a user interface and at least one processor operably coupled to the user interface. During operation, the user interface enables a user to tag a first instance of a representation of an object within a first image of the sequence of images. Also, the processor learns the representation of the object tagged by the user of the first image and tags a second instance of the representation of the object of the sequence of images. The user interface enables the user to perform an adjustment of the tag and / or position of the second instance of the representation of the object created by the at least one processor, and the processor tags a third instance of the representation of the object of the sequence of images based on the adjustment.

[0010] Other embodiments of the present technology include methods and systems for tagging objects within a data stream. An exemplary system includes at least one processor configured to implement a neural network and a fast learning module, and a user interface operably coupled to the processor. During operation, the neural network extracts a first convolutional output from a data stream that includes at least two representations of a first category of objects. This first convolutional output represents the features of a first representation of the first category of objects. The fast learning module classifies the first representation into the first category based on the first convolutional output and learns tags and / or the position of the first representation of the object based on adjustments by the user. Also, the user interface displays tags and / or positions for the first representation and enables the user to perform adjustments to the tags and / or positions of the first representation.

[0011] In some cases, the tag is the first tag and the position is the first position. In these cases, the neural network may extract a second convolutional output from the data stream. This second convolutional output represents the features of a second representation of the first category of objects. And in these cases, the classifier classifies the second representation into the first category based on the second convolutional output and the adjustments to the tags and / or positions for the first representation. The user interface may display a second tag and / or a second position based on the first category.

[0012] Optionally, the classifier can determine a confidence value that the tags and / or positions of the first representation are correct. The user interface may display the confidence value to the user.

[0013] If the object is the first object and the tag is the first tag, the classifier can learn a second tag for a second category of the objects represented in the data stream. In these cases, the neural network can extract a subsequent convolutional output from a subsequent data stream that includes at least one other representation of the second category of the objects. This subsequent convolutional output represents the features of the other representation of the second category of the objects. The classifier classifies the other representation of the second category of the objects into the second category based on the subsequent convolutional output and the second tag. Also, the user interface displays the second tag. In these cases, the neural network can generate a plurality of segmented sub-areas of the first image in the data stream and extract a first convolutional output by encoding each of the plurality of segmented sub-areas.

[0014] Yet another embodiment of the present technology includes a method of tagging a plurality of instances of an object. An example of this method includes extracting a first feature vector representing a first instance of the object of the plurality of instances using a feature extraction module. The user tags a first label to the first instance of the object via a user interface. The classifier module associates the first feature vector with the first label. The feature extraction module extracts a second feature vector representing a second instance of the object of the plurality of instances. The classifier module calculates the distance between the first feature vector and the second feature vector, performs a comparison of the difference with a predetermined threshold value, and classifies the second instance of the object based on the comparison. Optionally, the second instance of the object may be tagged with the first label based on the comparison. Then, the classifier module may determine the confidence level of the classification based on the comparison.

[0015] Of course, all combinations of the foregoing concepts and the additional concepts discussed in more detail below (assuming such concepts are not mutually inconsistent) are considered to be part of the subject matter of the invention disclosed herein. In particular, all combinations of the subject matter recited in the claims that appear at the end of this disclosure are considered to be part of the inventive subject matter disclosed herein. Also, of course, terms explicitly used in any disclosure incorporated by reference must be given a meaning that most closely matches the particular concepts disclosed herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Those skilled in the art will understand that the drawings are primarily for illustrative purposes and are not intended to limit the scope of the subject matter of the invention described herein. The drawings are not necessarily to scale, and in some instances, various aspects of the subject matter of the invention disclosed herein may be exaggerated or enlarged within the drawings to facilitate understanding of different features. In the drawings, like reference numerals generally denote like features (e.g., functionally similar and / or structurally similar elements).

[0017]

Figure 1A

[0018]

Figure 1B

[0019]

Figure 2

[0020]

Figure 3

[0021]

Figure 4

[0022]

Figure 5

[0023]

Figure 6

[0024] The capabilities of backpropagation-based neural networks, including deep neural networks and convolutional neural networks, depend on the availability of large amounts of training and test data to improve and verify the performance of these architectures. However, creating large amounts of labeled or tagged data is a manual, cumbersome, and costly process.

[0025] This application relates to automatically tagging, annotating, or labeling objects of interest to be identified and located in a data stream (e.g., a Red / Green / Blue (RGB) image, point cloud data, IR image, hyperspectral image, or a combination of these or other data). One use of these tagged data streams is in the creation of training and ground truth data utilized during the training and testing of a managed neural network, including a backpropagation-based deep neural network that uses thousands of images for proper training. The terms "annotation", "labeling", and "tagging" are used interchangeably in this document. The term "fast learning" is used in this application to describe a method that, unlike backpropagation, can be incrementally updated without the need to retrain the entire system for all of the previously presented data (from, for example, a single example). Fast learning is in contrast to "batch" training, which requires iterative presentation of a large data corpus even to learn a single new instance of an object.

[0026] The techniques described herein accelerate and improve the accuracy of manually labeling data by introducing an automated real-time fast learning step. This fast learning proposes candidate tags for subsequent occurrences of a tagged item in the same image, subsequent images, or both, as the first instance of the tagged item. Conversely, current techniques for tagging data rely on humans to label each object of interest in each frame (e.g., a frame of a video stream).

[0027] The method of the present invention introduces interactive assistance to the user. This interactive assistance is in the form of a neural network-based automatic assistant, also called a smart tagging system or utility, which has the ability to quickly learn new tags during the human process of data labeling. The automatic assistant makes a label or label suggestion for new data, receives corrections from the user for the suggested label, and iteratively refines the quality of the automatic labeling as the user continues to correct possible errors made by the automatic assistant. This has the advantage that the system does more work on its own, distancing itself from what it has learned, while the user can focus on new objects of interest and perform verification of the automatic tags. As a result, the tagging process becomes faster as more images are processed. Our research has shown an improvement in tagging speed of up to 40% for simple human taggers, in other words, the assistance of the smart tagging utility of the invention is 40% of the manual tagging of those who have never tagged an image before.

[0028] Figure 1A shows an example of this tagging process 10. 1. The user tags the first instance of the object in frame 1 (e.g., at 12 in Figure 1A). For example, the user can draw a bounding polygon and tag an object on frame 1, such as a tree. 2. A classifier executed on one or more processors of the smart tagging utility learns the first instance of the object (e.g., at 14 in Figure 1A) by associating features representing the tagged object with the tag defined by the user. In the example of step 1 (tagging the tree), the classifier learns the tree immediately after tagging. 3. The processor can tag subsequent instances of an object (e.g., at 16 in FIG. 1A). For example, a convolutional neural network executed by the processor can extract features from a frame (e.g., at 18 in FIG. 1A). The classifier classifies the extracted features based on their similarity to the extracted features associated with user-defined tags (e.g., at 20 in FIG. 1A). For example, the processor can tag other trees in Frame 1. If different, the neural network extracts features of other trees from Frame 1, and the classifier classifies them appropriately. Each tree has a confidence value associated with the boundary polygon and can be distinguished from other, manually labeled objects via several visual representations (e.g., the color of the boundary polygon, a dotted or dashed boundary polygon contour, etc.). The confidence value can be assigned by several methods for the classification or tracking of objects of interest, where the confidence level can be, for example, a scalar between 0 and 1, where 1 indicates absolute confidence that the object of interest belongs to a particular class. A particular object can have a probability distribution associated with the class hierarchy. For example, an object can be classified as a tree and as a plant simultaneously. 4. Optionally, at 22, the user adjusts the label or edits the position or shape of the boundary polygon generated by the machine for Frame 1, and the classifier utilizes fast learning to update its knowledge to update the proposal of the object in Frame 1 not yet verified by the user. Thereby, optionally, the tagged objects in Frame 1 can be automatically updated until the objects in Frame 1 are fully tagged. 5. The user loads Frame 2. 6. The smart tagging utility automatically tags objects in Frame 2 if the objects learned in Frame 1 appear in Frame 2. In other words, the smart tagging utility automatically tags the objects taking into account the user's adjustments if any of the tags in Frame 1 tag the objects in Frame 2. 7. Optionally, the user may add a new object that does not exist in Frame 1 or proceed to the next frame. In either case, the processes described in Steps 1 - 7 are repeated until the desired objects in the provided image are tagged or the user ends the process.

[0029] This method can be applied to any region of interest, such as rectangle, polygon, or pixel - based tagging. In polygon or pixel - based tagging, the shadow of the object is depicted in more detail than the region of the image, and the background region can be included in the tagged object. Increase the "target pixel" count for rectangle or polygon tags.

[0030] Various techniques can be employed to introduce a fast learning architecture, including, for example, · A combination of a DNN with a fast classifier that provides a set of functions where the DNN, such as another neural network, support vector machine, or decision tree, serves as an input to the fast classifier, and · A feature detection and tracking process that can be initialized in a short time on a target subset of the image (e.g., a keypoint tracker), and · Any combination of the above - mentioned techniques can be included, but is not limited thereto.

[0031] The technology of the present invention enables efficient and cost - effective preparation of a dataset for training a backpropagation - based neural network, particularly a DNN. More generally, it rationalizes the learning of parallel, distributed equation systems for performing data analysis for purposes such as controlling a self - driving vehicle, a drone, or other robots in real time.

[0032] More specifically, embodiments of the technology of the present invention improve or replace the process of manually tagging each occurrence of a particular object in a data stream (e.g., a frame) or sequence of frames, and the process of optimally selecting objects by reducing the manual effort and expense associated with dataset preparation.

[0033] [Process of Incremental Real-Time Learning for Tagging and Labeling of Data Streams] FIG. 1B shows a flowchart of the operation process of the smart tagging system of the invention. In this example, the system operates on images, but it can equally well operate on any data that can have a 2D representation understandable by humans with different objects or regions of interest to be tagged. The user starts the system (100) and loads a set of images or a video file into the system. In the case of a video file, an additional processing step is to decompose the video into a sequence of key frames to reduce the number of redundant images. This decomposition can be done automatically using the key frame information encoded in the video file, manually by the user, or both. When the sequence of images is ready, the system checks (105) if there are untagged frames, and if none remain, it ends (110). If there are frames to be tagged, the system loads the first frame (115) and performs feature extraction on this frame (120). In other words, the system loads the first frame and the neural network on one or more processors of the system extracts a convolutional output and creates a feature vector representing the features on the first frame. Details of the implementation of feature extraction are outlined in the corresponding section below.

[0034] At the same time, the system checks (125) whether it already has knowledge. This knowledge may include a set of previously learned associations between the extracted feature vectors and the corresponding labels. If the previously learned knowledge includes an association between the extracted feature vectors and the corresponding labels, a classifier executed on one or more processors classifies the extracted feature vectors with their respective labels. To classify the extracted feature vectors, the system performs feature matching. For example, the system compares the extracted feature vectors with features (and feature vectors) known to the system (e.g., previously learned knowledge). The comparison is performed based on a distance metric (e.g., the Euclidean norm of the associated feature space) that measures the distance in the feature space between the extracted feature vectors and the features known to the system. Next, the system classifies the object based on the difference. If the difference between the extracted features and the features for the first object of the system's existing knowledge is less than a threshold, the system classifies the feature as a potential first object. The actual distance or difference between the distance and the threshold can be a confidence value indicating the quality of the match or can be used to derive it.

[0035] The system can save such knowledge after a tagging session, and this saved knowledge can be loaded by the user at the start of a new session. This can be particularly useful if the current set of tagged images comes from the same domain that the user and the system have previously tagged. If the knowledge is not pre-loaded, the system displays a frame to the user (130) and waits for user input (135). In the case of the first frame with no previous knowledge for the system, the user manually tags one or more instances of the objects in the first image via the user interface (140). When the user tags the first instance of the first object in the image, the system learns the function and the associated label of the tagged object (145). Details of the fast learning classifier related to this stage are described in the corresponding section below.

[0036] After the system learns the characteristics of the tagged objects in the frame, the system processes the frame to check (150) whether it can discover other instances of the same objects in the frame. If the system pre-loads knowledge from a previous session, note that it can attempt to discover (150) known objects before the first frame is presented to the user through the same process. For the instances of the objects discovered by the system in the image, the system creates (155) a bounding polygon with an attached label, overlays (160) the bounding polygon on the image, and presents (130) the image with the overlaid bounding polygon and tags to the user. In some examples, if the user is not satisfied with the tags created by the system, the user can adjust the tags via the user interface. The classifier learns the adjusted tags and updates its knowledge. Then, the inner loop (170) continues for the user to add new objects and modify the system prediction until the user is satisfied with the tagging for this frame. When the user is satisfied, the system checks (105) whether there are still frames to be tagged. If so, the system loads (115) the next frame, performs feature extraction (120), and enters the inner loop (170) again. In this case, since the system has prior knowledge from at least one previous frame, note that the inner loop (170) enters through the lower branch of the workflow and the system makes predictions (150, 155, 160) before presenting the frame to the user (130).

[0037] The entire process continues until the images are tagged or the user ends the workflow. Before ending, the system can save the knowledge obtained from the user in this session so that it can be reused in the next session.

[0038] [Operating Procedures from the User's Perspective] Figures 2A - 2D show the operation workflow of FIG. 1B from the user's perspective. As shown in FIG. 2A, the system can provide the user with several input modalities (e.g., mouse (200) or touch screen (210)) for controlling system operations and for tagging images. The sequence of frames to be tagged can be loaded by the user as either a directory occupied by images on the local computer where the system is installed, a remote directory occupied by images, or a plain text or markup document containing local or remote file names for the images, or as one or more video files. In the latter case, the video is split by the system into a sequence of key frames. In any case, when the system has defined the sequence of frames (220) to process and the first frame is loaded into the system, it is ready to operate, and as a result, the user sees the image on the screen. If the user desires, the user can load a trained version of the system described herein to speed up the tagging process and even reduce the manual part of the task.

[0039] In FIG. 2B, the user selects, for example, a tree with a rectangular bounding box (230) in frame 1. Alternatively, other ways of selecting candidate regions are possible, for example, the user can draw a polygon as shown in FIGS. 3 - 5. The system learns the combination of features within the range of the bounding polygon and associates that combination of features with a label (e.g., "tree") provided by the user. The learning process is a fast learning procedure that completes within 100 milliseconds in the exemplary implementation. Since the process is very fast, the system can provide suggestions to the user for other yet - untagged instances of the same object in the frame. In this exemplary implementation, these suggestions also take less than 100 milliseconds to compute, so for the user it is very seamless that immediately after the user finishes tagging one object, the system proposes tagging for other instances (240) of the same object in the image.

[0040] First, especially when a system that has not been pre-trained before is not pre-loaded, the suggestions (240) created by the system may be quite off from the user's perspective. Next, the user can reject predictions that are completely wrong, adjust the wrong labels for the correct boundary polygon, adjust the boundary polygon suggested by the classifier for the correct label as shown in Figure 2C, and / or accept the correct proposals. To simplify the process, the boundary polygons suggested by the classifiers (240, 260) can be displayed as dotted lines, and the user's original taggings (230, 270) and the accepted corrections (250) can be displayed as dashed or solid lines as shown in Figures 2C and 2D. The accepted corrections constitute another input that can be used by the system to further refine the specific class of the object and re-train to improve further suggestions for tagging.

[0041] Next, the process continues for subsequent frames as shown in Figure 2D, where new objects (270) can be tagged by the user, and previously tagged objects (260) can be visualized by their associated classes and confidence values that indicate how confident the system is in its suggestions for the objects it marks in the image. As the system becomes increasingly trained, the interaction process with the user shifts from mainly correcting suggestions to accepting the suggestions made by the system, which is significantly less laborious and much faster than manually tagging the image. For naive taggers, the overall tagging speed of the test dataset was seen to increase by up to 40%, and for expert taggers, there was a less dramatic but significant increase.

[0042] Figures 3 to 5 provide screenshots from the graphical user interface of an exemplary smart tagging system. The user launches the manual tagging tool (300), selects a label from a list of existing labels or creates a new label, and draws a polygon around the person (310) on the first bicycle. The color of the polygon represents the label selected in this example. Since this polygon was created manually, it automatically obtains an approved status (320). The results of these actions are shown in Figure 3. Then, the fast classifier of the smart tagging system learns the features within the range of the polygon (310) and associates them with the user's label for the person on the first bicycle. Next, the system looks at the features of the entire frame, tries to find people on other bicycles in the image and create polygons around them (400), and shows the user the polygons superimposed on the people on other bicycles as shown in Figure 4. Note that since the polygon is created by the system, it is marked as "proposed" along with a confidence value of the proposal (410). Finally, Figure 5 shows that the user has restarted the manual correction tool (300) to update the polygon for the person (500) on the second bicycle and that it now has an approved label (510). The user can here tag other objects in the frame if necessary, go to the next frame using the arrow button (530), or complete the session by pressing the "Tagging complete" button (540).

[0043] [Smart Tagging System] FIG. 6 shows a smart tagging system, which is a hardware configuration for implementing the smart tagging utility described in this specification. Sensory information (e.g., RGB, infrared (IR), and / or LIDAR images) is derived from robots, drones, autonomous vehicles, toy robots, industrial robots, and / or other devices (610). Alternative embodiments may derive data from isolated or networked sensors (e.g., cameras, LIDAR 620). Further, the data may be compiled within an existing database (630). The data is transmitted to a computing device equipped with a user interface (650) that can be used by a user (640) to visualize the data (image), tag the image, visualize the tagged image, and adjust the tags as needed. The computing device can be a mobile device (e.g., smartphone, tablet, laptop), desktop, or server equipped with one or more processors or processing units (670) such as a digital signal processing unit, a field programmable gate array (FPGA), a central processing unit (CPU), a graphics processing unit (GPU), and / or a combination of these processors sufficient to implement the workflow of FIG. 1B. These processing units (670) can be used to implement feature extractors and classifiers that learn and automatically apply and adjust tags. An architecture including several software modules can be stored in a memory (680) and loaded onto the processing unit (670) for execution. The tagged image can be visualized on the UI (650) for display to the user (640) and stored in a separate database (660).

[0044] [Feature Extraction Module] The feature analysis module (120 in FIG. 6, 672 in FIG. 1) is one of the two core components of the automatic smart tagging system described herein. It receives inputs to the system in the presented format. For visualization and tagging convenience in the system's graphical user interface, a 2D representation of the raw external input is preferred, although it does not limit the input format to visual video only. Sound can be represented in 2D after a fast Fourier transform, and other input formats can be made into 2D representations by other means. Text input can be displayed as text and supplied to the system. With the advancement of 3D displays such as virtual and augmented reality systems, it becomes possible for the system to accept and tag 3D data.

[0045] The output of the feature extraction module is a set of feature vectors. Depending on the nature of the tagging, the set of feature vectors can be one feature vector per image (in the simple case where, for example, the entire image is tagged at once for scene recognition), or multiple feature vectors having associated regions of the image where these features are found. These regions can be as simple as rectangular bounding boxes, more complex shaped polygons, or even pixel-wise masks, depending on the ultimate purpose of the tagging process.

[0046] The exemplary implementations described herein use deep convolutional neural networks for feature extraction. A convolutional neural network (CNN) uses convolutional units where the receptive field of the unit filter (weight vector) is moved stepwise over the entire height and width dimensions of the input. Since each filter is small, the number of parameters is significantly reduced compared to a fully connected layer. If a set of features can be extracted for an object when the object is at one spatial position, the same set of features can be extracted for the same object when it appears at any other spatial position because the features containing the object are independent of the object's spatial position. These invariances provide a feature space in which the encoding of the input has improved stability against visual changes, meaning that as the input changes (e.g., the object translates slightly and rotates in the image frame), the output values change very little compared to the input values.

[0047] Convolutional neural networks are also good at generalization. Generalization means that within the trained mode, the network can generate similar outputs for test data that is not the same as the trained data. A large amount of data is required to learn the main regularities that define a set of class-specific features. When the network is trained on many classes, the lower layers where the filters are shared among classes provide a good set of regularities for the same style of input. Thus, a CNN trained for one task can provide excellent results when used as an initialization for other tasks or when the lower layers are used as a processor for new high-level representations. For example, natural images share a common set of statistical properties. Since the learned features of the lower layers are quite independent of the class, they can be reused even if the class the user is trying to tag is not one of the classes the CNN was related to. It is sufficient to incorporate these feature vectors and supply them as inputs to the fast learning classifier part (150 in FIG. 1B) of the system.

[0048] Depending on the final result of the target of the tagging process, different CNNs can function as feature extractors. For the recognition of the overall scene, versions of Alexnet, GoogLeNet or ResNet can be used according to the computing power of the available hardware. An average pooling layer must be added after the last feature layer of these networks to pool over the entire position of the image to create an overall scene feature vector. When the system is used to create a dataset for the training detection network, the same network can be used without average pooling, or a region proposal network such as fRCNN can be used instead for better spatial accuracy. When image segmentation is the target, segmentation networks such as Mask RCNN, FCN or U-Net can be used for feature extraction and mask generation. These masks can be converted to polygons for display and modification. For the exemplary implementation shown in FIGS. 3-5, a custom-made version of FCN was used.

[0049] Alternative implementations of the feature extraction module include, but are not limited to, scale-invariant feature transform (SIFT), speeded-up robust features (SURF), Haar-like feature detectors, dimensionality reduction, component analysis, and others, as long as they are sufficiently discriminative for different objects that need to be system-tagged and can operate fast enough so that the user does not have to wait for a noticeable amount of time while the system computes the feature set, any suitable technique for feature extraction can be used.

[0050] [Fast learning classifier module] The high-speed learning classifier module (150 in FIG. 1B, 674 in FIG. 6) takes in the feature vectors created by the feature extraction module (120 in FIG. 1B) and performs classification or feature matching to output a class label for each given set of features. One exemplary implementation can be simple template matching, where the system stores template feature vectors obtained based on user input for each known object. When an input set of feature vectors is presented, each of these vectors is compared to the template vectors based on some distance metric (e.g., the Euclidean norm of the associated feature space) to measure the difference between the current input and the template. Next, the classifier marks the feature vectors that have a difference metric smaller than a threshold set as the possible object. The reciprocal value of the distance can function as a confidence measure for this classification scheme.

[0051] Other techniques following high-speed learning can be substituted for the classifier in this template matching technique, and the techniques include regression analysis methods (e.g., linear regression, logistic regression, minimax analysis), kernel methods (e.g., support vector machines), Bayesian models, ensemble methods (e.g., ensemble of experts), decision trees (e.g., incremental decision trees, extremely fast decision trees and their derivatives), adaptive resonance theory-based models (e.g., Fuzzy ARTMAP) and linear discriminant online algorithms (e.g., online passive aggressive algorithms). For example, the implementations shown in FIGS. 3 - 5 use a modified ARTMAP for high-speed learning. The modifications reduce the sequential following attribute of ARTMAP, improve statistical consistency, and reduce the growth of the system memory footprint as it learns new objects.

[0052] [Conclusion] Although various inventive embodiments have been described and illustrated herein, those skilled in the art will readily be able to develop various other means and / or structures for performing the functions described herein and / or obtaining the results and one or more of the advantages, and each such variation and / or modification is to be regarded as being within the scope of the embodiments of the invention described herein. More generally, those skilled in the art will readily understand that all parameters, dimensions, materials, and configurations described herein are exemplary, and that the actual parameters, dimensions, materials, and / or configurations will depend on the particular application or the application for which the teachings of the invention are used. Those skilled in the art can recognize or confirm, using ordinary experimentation, numerous equivalents to the particular inventive embodiments described herein. Accordingly, it is to be understood that the foregoing embodiments are presented by way of example only and that embodiments of the invention can be practiced otherwise than as particularly described in the appended claims and their equivalents. The inventive embodiments of the present disclosure are directed to the individual features, systems, articles, materials, kits, and / or methods described herein. Further, any combination of two or more of such features, systems, articles, materials, kits, and / or methods is included within the scope of the present invention of the present disclosure if such features, systems, articles, materials, kits, and / or methods do not mutually conflict.

[0053] The above-described embodiments can be implemented in any of a number of ways. For example, embodiments of the techniques disclosed herein may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can be executed by any appropriate processor or collection of processors, whether provided on a single computer or distributed among multiple computers.

[0054] Furthermore, it should be understood that the computer can be embodied in any of a number of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. Additionally, the computer may be generally considered a computer, including a personal digital assistant (PDA), a smartphone or any other suitable portable or fixed electronic device, but may be embedded within a device having appropriate processing capabilities.

[0055] Also, the computer may have one or more input and output devices. These devices can be used, among other things, to present a user interface. Examples of output devices that can be used to provide a user interface include printers or display screens for visual representations of output and speakers or other audio generating devices for audible representations of output. Examples of input devices that can be used for a user interface include keyboards and pointing devices such as mice, touch pads, and digitizer tablets. As another example, the computer may receive input information in speech recognition or other audible formats.

[0056] Such computers may be interconnected by one or more networks in any suitable form, such as a local area network, a wide area network such as an enterprise network, an intelligent network (IN) or the Internet. Such networks may be based on any suitable technology, may operate according to any suitable protocol, and may include wireless networks, wired networks, or fiber optic networks.

[0057] The various methods or processes outlined in this specification may be encoded as software executable by one or more processors using any one of a variety of operating systems or platforms. Further, such software may be described using any of a number of suitable programming languages and / or programming or scripting tools, and may be compiled as executable machine code or intermediate code to be executed on a framework or virtual machine.

[0058] In this regard, various inventive concepts may be embodied as a computer-readable storage medium (or multiple computer-readable storage media) (e.g., computer memory, one or more floppy disks, compact disks, optical disks, magnetic tapes, flash memory, field programmable gate array or other semiconductor device circuitry or other non-transitory media or tangible computer storage media) encoded with one or more programs that, when executed on one or more computers or other processors, perform the methods of the various embodiments of the present invention described above. The computer-readable medium or media may be movable, as described above, to load the program or programs stored thereon onto one or more different computers or other processors so as to implement the various aspects of the present invention.

[0059] The term "program" or "software" as used herein generally means any set of computer code or computer-executable instructions that can be used to program a computer or other processor, as described above. Further, of course, according to one aspect, the one or more computer programs that perform the methods of the present invention being executed need not belong to a single computer or processor, but may be distributed in modular form among a number of different computers or processors for implementing the various aspects of the present invention.

[0060] Computer-executable instructions can be in many forms such as program modules executed by one or more computers or other devices. Generally, program modules can include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Usually, the functions of program modules can be combined or distributed as desired in various embodiments.

[0061] Also, data structures can be stored in a computer-readable medium in any suitable form. For the sake of simplicity of illustration, a data structure may be shown as having fields related to the position of the data structure. Such relationships can be similarly achieved by allocating storage for fields having positions in a computer-readable medium that convey the relationships between fields. However, any suitable mechanism can be used to establish relationships between the information of the fields of a data structure, including using pointers, tags, or other mechanisms to establish relationships between data elements.

[0062] Also, various inventive concepts may be embodied as one or more methods, and examples are provided. The operations performed as part of a method can be ordered in any suitable way. Thus, embodiments can be made in which the operations are performed in an order different from that illustrated, which can include performing some operations simultaneously, even if they are shown as sequential operations in the exemplary embodiments.

[0063] All definitions defined and used in this specification are to be understood as controlling dictionary definitions, definitions of incorporated documents by reference, and / or the ordinary meaning of the defined terms.

[0064] As used in this specification and the claims, the indefinite articles "a" and "an" are to be understood to mean "at least one" unless clearly indicated otherwise.

[0065] As used in this specification and the claims, the phrase "and / or" means "any or both" of the associated elements, i.e., elements that may be present conjunctively in some cases and disjunctively in others. Multiple elements listed as "and / or" should be construed in the same form, i.e., as "one or more" of the associated elements. Other elements may optionally be present in addition to those identified by the "and / or" clause, whether or not they are related to the specifically identified elements. Thus, by way of non-limiting example, a reference to "A and / or B" can, in one embodiment, when used in conjunction with open-ended syntax such as "comprising", refer to only A (optionally including elements other than B), in another embodiment, refer to only B (optionally including elements other than A), and in yet another embodiment, refer to both A and B (optionally including other elements), and so on.

[0066] As used in this specification and the claims, "or" should be understood to have the same meaning as "and / or" as defined above. For example, when separating items in a list, "or" or "and / or" is to be construed inclusively, i.e., as including at least one of a number of elements or a list of elements, and optionally additional items not in the list, but also including two or more. In contrast, only terms that clearly indicate otherwise, such as "only one of", or "exactly one of", or when used in the claims, only the term "consisting of", refer to the inclusion of exactly one element out of a number of elements or a list of elements. Generally, as used in this specification, the term "or" is to be construed as indicating an exclusive alternative (i.e., "either but not both") only when preceded by an exclusive term such as "any", "one of", "only one of", or "exactly one of". "Consisting essentially of" shall have the ordinary meaning as used in the field of patent law when used in the claims.

[0067] As used herein, the terms "about" and "approximately" generally mean plus or minus 10% of the stated value.

[0068] As used herein and in the claims, the phrase "at least one" referring to a list of one or more elements means at least one element selected from any one or more of the elements of the list of elements, but is not necessarily meant to include at least one of each specifically recited element within the scope of the list of elements, nor is it meant to exclude any combination of elements of the list of elements. It should be understood that this definition also allows for the optional presence of elements other than those specifically identified within the scope of the list of elements referred to by the phrase "at least one", whether or not those other elements are related to the specifically identified elements. Thus, by way of non-limiting example, "at least one of A and B" (or, equivalently, "at least one of A or B", or, equivalently "at least one of A and / or B") can, in one embodiment, refer to at least one, optionally two or more, A's (optionally including elements other than B) where B is absent, in another embodiment can refer to at least one, optionally two or more, B's (optionally including elements other than A) where A is absent, and in yet another embodiment can refer to at least one, optionally two or more, A's, and at least one, optionally two or more, B's (and, optionally, other elements).

[0069] In the claims as well as in the above specification, all transitional phrases such as "comprising", "including", "carrying", "having", "containing", "involving", "holding", "consisting of", and the like are to be understood to be open-ended, i.e., to mean including but not limited to. Only the transitional phrases "consisting of" and "consisting essentially of" are to be considered as closed or semi-closed transitional phrases as set forth in the USPTO Patent Examination Guidelines, Section 2111.03.

Claims

1. 1. A method for tagging instances of objects appearing in a sequence of image frames, comprising: drawing, by a user through a user interface, a first bounding polygon around a first instance of an object appearing in said sequence of image frames; tagging a first instance of the object with a first label by the user through the user interface; extracting, using a feature extraction module, a first feature vector representative of the content of the first bounding polygon; a classification module associating the first label of the object with the first feature vector based on tagging by the user; the classification module generating a second bounding polygon in the sequence of image frames; extracting, using the feature extraction module, a second feature vector representative of the content of the second bounding polygon; calculating a similarity between the first feature vector and the second feature vector using the classification module; using the classification module to compare a similarity between the first feature vector and the second feature vector with a predetermined threshold; classifying, using the classification module, the content of the second bounding polygon as a second instance of the object based on the comparison; displaying the second bounding polygon and a suggested label for the contents of the second bounding polygon to the user through the user interface, the suggested label being the same as the first label; accepting modifications to at least one of the second boundary polygon and the proposed label from the user through the user interface; updating the classification module based on the modifications; A method comprising:

2. 2. The method of claim 1, further comprising determining a confidence value for correctly classifying the content of the second bounding polygon as the second instance based on a comparison of the distance between the first feature vector and the second feature vector and the predetermined threshold.

3. The method of claim 2 , further comprising displaying the confidence value to the user through the user interface.

4. the object is a first object, tagging, by the user through the user interface, a first instance of a second object with a second label; extracting, with the feature extraction module, a third feature vector representing the contents of a third bounding polygon that includes the first instance of the second object; the classification module associating the second label with the third feature vector; The method of claim 1 further comprising:

5. extracting, using the feature extraction module, a fourth feature vector from the content of a fourth bounding polygon in the sequence of image frames; calculating a distance between the third feature vector and the fourth feature vector using the classification module; using the classification module to compare the distance between the third feature vector and the fourth feature vector to the predetermined threshold; classifying, using the classification module, the content of the fourth bounding polygon as a second instance of the second object based on the comparison; once the second instance of the second object is classified, displaying the second instance of the second object with the second label through the user interface; The method of claim 4 further comprising:

6. 2. The method of claim 1 , wherein extracting the first feature vector comprises extracting a first convolution output of a first image in the sequence of image frames, the first convolution output comprising the first feature vector representing a first instance of the object.

7. The extraction of the first convolution output includes: generating a plurality of divided sub-areas in the first image; the feature extraction module encoding each of the plurality of divided sub-areas; The method of claim 6, comprising:

8. The method of claim 1 , wherein the feature extraction module extracts the second feature vector from the first image in the sequence of image frames.

9. The method of claim 1 , wherein the feature extraction module extracts the second feature vector from a second image in the sequence of image frames.

10. extracting, using the feature extraction module, a third feature vector from the content of a third bounding polygon in the sequence of image frames; calculating a distance between the first feature vector and the third feature vector using the updated classification module; using the updated classification module to compare the distance between the first feature vector and the third feature vector with the predetermined threshold; using the updated classification module to classify the content of the third bounding polygon as a third instance of the object based on a comparison of the distance between the first feature vector and the third feature vector to the predetermined threshold; The method of claim 1 further comprising:

11. The method of claim 1 , wherein the step of accepting the modification comprises accepting a change to at least one of a shape and a position of the second bounding polygon from the user through the user interface.

12. The method of claim 1 , wherein the step of accepting the modifications comprises accepting, from the user through the user interface, changes to the first label relating to the content of the second bounding polygon.

13. The method of claim 1 , wherein the sequence of image frames comprises keyframes extracted from a video file.

14. The method of claim 1 , wherein the first bounding polygon or the second bounding polygon comprises a rectangular bounding box or a pixel-wise mask.

15. 1. A system for tagging instances of objects appearing in a sequence of image frames, comprising: a user interface that allows a user to tag a first instance of a representation of the object that appears in the sequence of image frames; at least one processor operatively connected to the user interface; and The at least one processor drawing, by the user through the user interface, a first bounding polygon around a first instance of the object appearing in the sequence of image frames; tagging a first instance of the object with a first label by the user through the user interface; extracting, using a feature extraction module, a first feature vector representative of the content of the first bounding polygon; a classification module associating the first label of the object with the first feature vector based on tagging by the user; the classification module generating a second bounding polygon in the sequence of image frames; extracting, using the feature extraction module, a second feature vector representative of the content of the second bounding polygon; calculating a similarity between the first feature vector and the second feature vector using the classification module; using the classification module to compare a similarity between the first feature vector and the second feature vector with a predetermined threshold; classifying, using the classification module, the content of the second bounding polygon as a second instance of the object based on the comparison; displaying the second bounding polygon and a suggested label for the contents of the second bounding polygon to the user through the user interface, the suggested label being the same as the first label; accepting modifications to at least one of the second boundary polygon and the proposed label from the user through the user interface; updating the classification module based on the modifications; To do system.

16. 16. The system of claim 15, wherein extracting the first feature vector comprises extracting a first convolution output of a first image in the sequence of image frames, the first convolution output comprising the first feature vector representing a first instance of the object.

17. The extraction of the first convolution output includes: generating a plurality of divided sub-areas in the first image; the feature extraction module encoding each of the plurality of divided sub-areas; 17. The system of claim 16, comprising:

18. The system of claim 15 , wherein the feature extraction module extracts the second feature vector from the first image in the sequence of image frames.

19. The system of claim 15 , wherein the feature extraction module extracts the second feature vector from a second image in the sequence of image frames.

Images