Online, incremental real-time learning for tagging and labeling data streams for deep neural networks and neural network applications

Abstract

To provide a method and a system for creating a tagged large data set used for training an artificial neural network.SOLUTION: A feature extraction unit and fast-learning classifiers can be implemented as artificial neural networks that associate a label with features extracted from an image and tag similar features from the image or other images with the same label. Moreover, a smart tagging system can learn from user adjustment to its proposed tagging. This reduces tagging time and errors.SELECTED DRAWING: Figure 1A

Claims

1. 1. A method for tagging a sequence of images, comprising the steps of: tagging, by at least one processor, a representation of a first object in a sequence of images; tagging, by a user, a first instance of a representation of a second object in a first image in the sequence of images; learning, by a fast-learning classifier executing on the at least one processor, a representation of the second object tagged by the user in the first image without the fast-learning classifier being retrained on data previously presented to the fast-learning classifier; tagging, by the at least one processor, second instances of the representation of the second object in the sequence of images, comprising: (1) extracting, by a feature extraction neural network executed on the at least one processor, convolution outputs representative of features of the second instances of the representation of the second object; and (2) classifying, by the fast learning classifier, the second instances of the representation of the second object based on the convolution outputs. performing, by the user, an adjustment of a tag and / or a position of the second instance of the representation of the second object created by the at least one processor; tagging, by the at least one processor, a third instance of the representation of the second object within the sequence of images based on the adjustment; and The method includes:

2. The method of claim 1 , wherein the second instance of the representation of the second object is in the first image in the sequence of images.

3. The method of claim 1 , wherein the second instance of the representation of the second object is in another image in the sequence of images.

4. performing, by the user, an adjustment of a tag and / or a position of the third instance of the representation of the second object created by the at least one processor; tagging, by the at least one processor, a fourth instance of the representation of the second object within the sequence of images based on the adjusting of the tag and / or position of the third instance of the representation of the second object; The method of claim 1 further comprising:

5. 2. The method of claim 1, further comprising the step of: the fast learning classifier executing on the at least one processor classifying a representation of the second object tagged by the user in the first image.

6. tagging the third instance of the representation of the second object, extracting, by the feature extraction neural network, a convolution output representative of features of the third instance of the representation of the second object; classifying, by the fast learning classifier, the third instance of the representation of the second object based on the convolution output; and The method of claim 5 , comprising:

7. 1. A system for tagging a sequence of images, comprising: a user interface that enables a user to tag a first instance of a representation of an object in a first image in the sequence of images; at least one processor operatively connected to the user interface for executing a neural network to extract a first convolution output representative of features of the first instance of a representation of an object in the first image, a second convolution output representative of features of a second instance of a representation of an object in the sequence of images, and a third convolution output representative of features of a third instance of a representation of an object in the sequence of images, and for executing a fast-learning classifier to learn the representation of the object tagged by the user in the first image without being retrained on data previously presented to the fast-learning classifier, and to tag the second instance of the representation of the object in the sequence of images based on a comparison of the first convolution output and the second convolution output; Including, the user interface allows the user to perform an adjustment of tags and / or positions of the second instances of the representations of the object created by the at least one processor, and the fast learning classifier is configured to tag the third instances of the representations of the object within the sequence of images based on the adjustment and the third convolution output. system.

8. 8. The system of claim 7, wherein a user interface enables the user to adjust a tag and / or position of the third instance of the representation of the object created by the at least one processor, and the at least one processor is configured to tag a fourth instance of the representation of the object within the sequence of images based on the adjustment of the tag and / or position of the third instance of the representation of the object.

9. The system of claim 7 , wherein the second instance of the representation of the object is in the first image in the sequence of images.

10. The system of claim 7 , wherein the second instance of the representation of the object is in a second image in the sequence of images.

11. The system of claim 7 , wherein the fast learning classifier is configured to classify representations of the objects tagged by the user in the first image.

12. 1. A method for tagging objects in a data stream, comprising the steps of: extracting, by a neural network executed on at least one processor, a first convolution output from a data stream, the data stream including at least two representations of a first class of objects, the first convolution output representing features of a first representation of a first category of objects; classifying, by a classifier executing on the at least one processor, the first representation into the first category of object based on the first convolution output; displaying, via a user interface coupled to the at least one processor, tags and / or locations for the first representation based on the first category; performing an adjustment of said tags and / or said positions for said first representation by a user via said user interface; learning, by the classifier, the tag and / or the location of the first representation of the object based on the adjustment without the classifier being retrained on data previously presented to the classifier; The method includes:

13. the tag is a first tag and the location is a first location; extracting, by the neural network, a second convolution output from the data stream, the second convolution output representing features of a second representation of the first category of objects; classifying, by the classifier, the second representation into the first category based on the second convolution output and the adjustment of the tag and / or the position for the first representation; displaying, via the user interface, a second tag and / or a second location based on the first category; The method of claim 12 further comprising:

14. determining, by said classifier, a confidence value that said tag and / or position of said first representation is correct; displaying the confidence value to the user; The method of claim 12 further comprising:

15. the object is a first object and the tag is a first tag; 13. The method of claim 12, further comprising: learning, by the classifier, second tags for a second category of objects, the data stream including at least one representation of the second category of objects.

16. extracting, by the neural network, a subsequent convolution output from a subsequent data stream, the subsequent data stream including at least one other representation of the second category of objects, the subsequent convolution output representing features of the at least one other representation of the second category of objects; classifying, by the classifier, the at least one other representation of the second category of object into the second category based on the subsequent convolution output and the second tag; displaying the second tag via the user interface; The method of claim 15 further comprising:

17. The step of extracting the first convolution output comprises: generating a plurality of divided sub-areas of a first image of the data stream; encoding each of the plurality of divided sub-areas by the neural network; 17. The method of claim 16, comprising:

18. At least one processor, a neural network for extracting a first convolution output from a data stream, the data stream including at least two representations of a first category of objects, the first convolution output representing features of the first representation of the first category of objects; and a rapid learning module that classifies the first representation into the first category based on the first convolution output and learns tags and / or locations of the first representation of the object based on user adjustments without retraining on previously presented data; and a user interface coupled to the at least one processor and configured to display the tag and / or the location for the first representation and to accept the adjustment of the tag and / or the location of the first representation from the user; A system including:

19. the tag is a first tag, the position is a first position, the neural network is configured to extract a second convolution output from the data stream, the second convolution output representing features of a second representation of the first category of objects; the fast learning module is configured to classify the second representation into the first category based on the second convolution output and the adjustment of the first tag and / or the first position for a first instance of the first representation of the object; The system of claim 18 , wherein the user interface is further configured to display a second tag and a second location based on the first category.

20. the fast learning module is configured to determine a confidence value that the first tag and / or the first location is correct; The system of claim 18 , wherein the user interface is further configured to display the confidence value to the user.

21. 20. The system of claim 18, wherein the tag is a first tag, the rapid learning module is further configured to learn a second tag for a second category of objects, and the data stream includes at least one representation of the second category of objects.

22. the neural network is further configured to extract a subsequent convolution output from a subsequent data stream, the subsequent data stream including at least one other representation of the second category of objects, the subsequent convolution output representing features of the at least one other representation of the second category of objects; the fast learning module is further configured to classify the at least one other representation of the second category of objects into the second category based on the subsequent convolution output and the second tag; The system of claim 21 , wherein the user interface is further configured to display the second tag.

23. The neural network comprises: generating a plurality of partitioned sub-areas of a first image of the data stream; The system of claim 18 , further configured to code each of the plurality of divided sub-areas.

24. prior to learning objects in the first image tagged by the user, extracting a feature vector in the first image; checking associations between features of the feature vector and labels corresponding to the features that have been learned so far; The method of claim 1 further comprising:

25. 25. The method of claim 24, further comprising classifying, with the fast learning classifier, objects in the first image based on the associations learned so far.

Images