Online, incremental, real-time learning for tagging and labeling data streams for deep neural networks and neural network applications

Abstract

To provide a method and a system for creating a tagged large data set used for training an artificial neural network.SOLUTION: A feature extraction unit and fast-learning classifiers can be implemented as artificial neural networks that associate a label with features extracted from an image and tag similar features from the image or other images with the same label. Moreover, a smart tagging system can learn from user adjustment to its proposed tagging. This reduces tagging time and errors.SELECTED DRAWING: Figure 1A

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED PATENT APPLICATIONS This application is a continuation of U.S. Patent Application No. 62 / 472,925 (filed March 17, 2017). 119(e) of the U.S. Patent Act, the subject matter of which is a It shall form part of the [Background technology]

[0002] A convolutional neural network that contains many layers of neurons placed between the input and output layers. Conventional deep neural networks, including convolutional neural networks (CNNs), A Deep Neural Network (DNN) is trained on a specific dataset. This training requires thousands or even millions of repeated cycles to complete. Before this can happen, all images in the dataset must be tagged by human users. The tagging process includes classification and detection / segmentation of individual objects. Classification or labeling of individual regions of each image as specific objects for application In some cases, it may be necessary to label the entire image for classification.

[0003] Traditional image tagging is a slow and time-consuming process. Humans, computers, Look at an image on your tablet or smartphone and identify one or more objects in the image and tag those objects with descriptive tags (e.g., "tree," "house," or "car"). The main difficulties in manually tagging objects of interest include slow speed. and susceptibility to human error caused by distraction and fatigue. These issues create two kinds of challenges: data preparation takes an unacceptably long time anywhere outside of an academic setting. and that improperly tagged data can prevent the DNN from performing acceptable The quality of tagging directly affects the quality of subsequent learning. This means that it has an impact. Summary of the Invention

[0004] An embodiment of the present technique provides a method and system for tagging a sequence of images. An exemplary method includes a system for selecting a sequence of an image by a user via a user interface. tagging a first instance of a representation of an object in a first image of a sequence The at least one processor detects the tagged image by the user in the first image. We learn a representation of the object to be imaged and then compute a second representation of the object in the image sequence. The user tags the instance of the object created by the processor. performing tag and / or position adjustments of the current second instance; and Based on the adjustment, the third instance of the representation of the object in the sequence of images is tagged. Add tag.

[0005] The second instance of the representation of the object is located in the first image of the sequence of images. or in another image of a sequence of images.

[0006] In some cases, the user may request a third party to create a representation of the object created by the processor. and performing tag and / or location adjustment of the instances of adjusting the tag and / or position of a third instance of the representation of the object based on the A fourth instance of the representation of the object in the sequence of images is tagged.

[0007] An example method includes classifying via a fast learning classifier running on a processor. The representation of the object may include an object tagged by a user in the first image. In this case, the tagging of a third instance of the representation of the object The present invention relates to a method for detecting a plurality of objects using a neural network operatively coupled to a fast learning classifier. The method may further include extracting a convolution output representative of features of a third instance of the representation of the object. The fast learning classifier calculates the third instance of the object representation based on the convolution output. Classify.

[0008] An example of this method is to generate objects using a neural network running on a processor. The representation is then extracted by extracting the convolution output that represents the features of the second instance of the representation of the object. and operatively tagging a second instance of the neural network. A second instance of the representation of the object is generated based on the convolution output using a classifier connected to the This may also include sorting dressers.

[0009] The system for tagging a sequence of images includes a user interface and and at least one processor operably coupled to the user interface. In operation, the user interface allows a user to select an object in a first image of a sequence of images. It also allows the processor to tag the first instance of a representation of the object. The system learns representations of objects tagged by a user in a first image and synthesizes the images. Tags the second instance of a representation of an object in a sequence. The interface is a representation of an object created by a user at least one processor. This allows you to perform tag and / or position adjustments for the second instance of the the processor adjusts the representation of the object in the sequence of images based on the third instance of the representation. Tag.

[0010] Another embodiment of the present technology includes a method for tagging objects in a data stream. Exemplary systems include neural networks and fast learning. At least one processor configured to implement a learning module; In operation, the neural network includes a user interface operatively coupled to the The task is a data stream that includes at least two representations of a first category of objects. The first convolution output is extracted from the first The fast learning module performs a feature extraction based on the first convolution output. and classifying the first representation into a first category based on the first representation and based on the user's adjustments, the tags and and / or learning the location of the first representation of the object. Also, the user interface displaying tags and / or locations for the first representation, and allowing the user to select the tags and locations for the first representation; and / or position adjustments can be made.

[0011] In some cases, the tag is a first tag and the location is a first location. In this case, the neural network can extract the second convolution output from the data stream. The output of this second convolution is the feature of the second representation of the first category of objects. In these cases, the classifier uses the second convolution output and the first representation Classifying the second representation into the first category based on the tag and / or position adjustment. The user interface may display a second tag and / or a second category based on the first category. The location may be displayed.

[0012] Optionally, the classifier may generate a confidence value that the tag and / or location of the first representation is correct. The user interface may display the confidence value to the user. .

[0013] If the object is a first object and the tag is a first tag, the classifier Learn a second tag for a second category of objects represented in the data stream. In these cases, the neural network will pick up a small number of second categories of objects. Extract the subsequent convolution output from a subsequent data stream that contains at least one other representation. The output of this subsequent convolution can be used to find other representations of the second category of objects. The classifier classifies the object based on the subsequent convolution output and the second tag. Other expressions of the second category of the user interface are classified into the second category. In these cases, the neural network performs a generating a plurality of divided sub-areas of a first image in the frame; The first convolution output may be extracted by encoding each of

[0014] Yet another embodiment of the present technology provides a method for tagging multiple instances of an object. An example of this method is to use a feature extraction module to extract objects from multiple instances. The method further comprises extracting a first feature vector representing a first instance of the object. Assign a first label to a first instance of an object via a user interface Tagging: The classifier module associates the first feature vector with a first label. The feature extraction module represents a second instance of the object of the plurality of instances. A second feature vector is extracted. The classifier module classifies the first feature vector and the second feature vector. Calculate the distance between the vectors, perform a comparison of the difference with a given threshold, and select the option based on the comparison. Classify the second instance of the object. If necessary, classify the second instance of the object. The stance may be tagged with a first label based on the comparison. The module may determine a confidence in the classification based on the comparison.

[0015] Of course, any combination of the above concepts and the additional concepts discussed in more detail below may be used. Combinations (provided that such concepts are not mutually exclusive) are within the scope of the present disclosure. These and other features are considered to be part of the subject matter of the present invention, particularly as set forth in the claims appearing at the end of this disclosure. All combinations of the subject matter listed are considered to be part of the inventive subject matter disclosed herein. It is also understood that any disclosure incorporated by reference expressly states: The terms used should be given the meanings that are most consistent with the specific concepts disclosed herein. be. [Brief description of the drawings]

[0016] Those skilled in the art will appreciate that the drawings are primarily for illustrative purposes and that the It is understood that the drawings are not intended to limit the scope of the present subject matter. Not to scale, in some instances, various aspects of the inventive subject matter disclosed herein are In the drawings, the dimensions of the different features may be exaggerated or enlarged to facilitate understanding. Similar reference characters generally refer to similar features (e.g., functionally similar and / or structurally similar). "substantially similar elements" means

[0017] [Figure 1A] FIG. 1A illustrates the process of online, incremental, real-time learning for tagging and labeling data streams with a processor-implemented convolutional neural network and fast-learning classifier.

[0018] [Figure 1B] FIG. 1B illustrates an operational workflow for tagging objects in one or more images (eg, frames of a video or images in a database) using the smart tagging system of the present invention.

[0019] [Diagram 2] 2A-2D show the operational workflow of the inventive smart tagging system from a user's perspective.

[0020] [Diagram 3] FIG. 3 illustrates a screenshot of an example implementation after a user tag has tagged a first instance of a first object.

[0021] [Figure 4] FIG. 4 shows a screenshot of an example implementation after the system has learned from the previous steps and provided a suggestion to the user to tag a second instance of the first object.

[0022] [Diagram 5] FIG. 5 shows a screenshot of an example implementation after a user has modified a suggestion made by the system.

[0023] [Figure 6] FIG. 6 is a schematic overview of an implementation of a smart tagging utility that can automatically tag objects similar to a user-selected object within the same or different frames of video or other image data. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0024] Backplane including deep neural networks and convolutional neural networks The power of shaping-based neural networks improves the performance of these architectures. It depends on the availability of large amounts of training and test data to first validate the Creating large amounts of labeled or tagged data can be manual, tedious, or It is a costly and expensive process.

[0025] This application concerns automatically tagging, annotating, or labeling objects of interest to be identified and located in a data stream (e.g., red / green / blue (RGB) images, point cloud data, IR images, hyperspectral images, or a combination of these or other data). One use of these tagged data streams is to create training and ground truth data that is utilized during training and testing of supervised neural networks, including backpropagation-based deep neural networks that use thousands of images for proper training. The terms "annotation," "labeling," and "tagging" are used interchangeably in this document. The term "rapid learning" is used in this application. To Leave, B Block Propagation Unlike , for example, from a single example, for (all of) the data presented previously Overall System without the need to retrain Gradual How can it be updated? explanation Fast learning is used to 、 Even to learn a single new instance of an object Large amounts of data Corpus Repetition This contrasts with "batch" training, which requires time-consuming presentations. .

[0026] The techniques described herein introduce an automated, real-time, rapid learning step. This accelerates and improves the accuracy of manually labeling data. The learning occurs when the same image as the first instance of the tagged item, subsequent images, or both. In turn, it suggests candidate tags for subsequent occurrences of tagged items. Current techniques for tagging data require a unique number of bits for each frame (e.g., a frame of a video stream). It relies on humans labeling each object of interest.

[0027] The method of the present invention introduces interactive assistance to the user. Active assistance is provided by Neural Networks, also known as smart tagging systems or utilities. This is a form of automated assistant based on a human network that helps to label data. It has the ability to rapidly learn new tags during the process. The automatic assistant Label or suggest labels for new data and provide feedback on the suggested labels. Accept corrections from the user and allow the user to correct possible mistakes made by the automated assistant. As we continue to correct the errors, we iteratively refine the quality of the automatic labeling. It does more work on itself, moving what it learns away from the user, Users can focus on new objects of interest and perform automatic tag validation. As a result, the tagging process accelerates as more images are processed. Our work has shown that tagging speed is up to 40% faster than that of simple human taggers. In other words, the improvement in the speed of the inventive smart tagging utility. The assistance is 40% of manual tagging for people who have never tagged images before.

[0028] FIG. 1A shows an example of this tagging process 10 . 1. The user tags the first instance of an object in frame 1 (e.g. For example, the user can draw a boundary polygon and In this way, objects on system 1, such as trees, can be tagged. 2. The classifiers run on one or more processors in the Smart Tagging utility. ,Associate features that represent tagged objects with user-defined tags. By doing so, the first instance of the object is learned (e.g., at 14 in Fig. 1A). In the example of step 1 (tree tagging), the classifier learns the trees immediately after tagging. 3. The processor performs the following on subsequent instances of the object (e.g., at 16 in FIG. 1A). For example, the convolutional neural network executed by the processor The network can extract features from a frame (e.g., 18 in Figure 1A). The classifier associates the extracted features (e.g., at 20 in FIG. 1A) with user-defined tags. The processor then classifies the images based on their similarity to the extracted features. can be tagged to other trees in frame 1. If they are different, the neural network The classifier extracts the features of other trees from frame 1 and classifies them appropriately. , have a confidence value associated with the boundary polygon, and have some visual indication (e.g. Other, manually labelled, via polygon colour, dotted or dashed boundary polygon contours etc. The confidence value is used to classify or track an object of interest. The king can be assigned in several ways, where the confidence is, for example, For example, it can be a scalar between 0 and 1, where 1 indicates that the object of interest belongs to a particular class. A particular object is associated with a hierarchy of classes. For example, an object can be classified as a tree and as a plant. They can be classified simultaneously as: 4. If necessary, at 22, the user adjusts the label or the machine in frame 1 produces We edit the position or shape of the boundary polygons that make up the model, and the classifier uses rapid learning to apply that knowledge. Update the suggestions for objects in frame 1 that have not yet been verified by the user. This allows you to tag objects as much as you need to until they are fully tagged in frame 1. to automatically update tagged objects in frame 1. . 5. The user loads frame 2. 6. The Smart Tagging utility will now display the object being studied in Frame 1. appears in frame 2, then tag them automatically. The tagging utility checks whether any of the tags in frame 1 are associated with an object in frame 2. When tagging objects, the system automatically tags the objects, taking into account the user's adjustments. do. 7. If necessary, the user can add new objects that do not exist in Frame 1. or proceed to the next frame, in either case the process described in steps 1-7 continues. ,The ,process ,until ,the ,desired ,object ,of ,the ,provided ,image ,is ,tagged, ,or ,the ,user ,processes. Repeat until the end.

[0029] This method can be used to tag any region of interest, such as rectangles, polygons, or pixel-based tags. In polygon or pixel-based tagging, The shadows of the object are detailed rather than the image regions, and the background regions are tagged. "Tags" refers to rectangular or polygonal tags that can be included in objects that are Increase the "target pixel" count.

[0030] A variety of techniques can be employed to implement rapid learning architectures, including e.g. For example, A DNN, such as another neural network, a support vector machine, or a decision tree provides a set of features that serve as input to the fast classifier. Combination of DNNs and Quick initialization on a target subset of images (e.g., a keypoint tracker) A feature detection and tracking process that can - may include, but is not limited to, any combination of the above techniques .

[0031] The technique of the present invention is based on backpropagation-based neural networks, particularly DNs. Allows for efficient and cost-effective preparation of datasets for training on N, More generally, controlling autonomous vehicles, drones or other robots in real time Streamline the learning of parallel, distributed equation systems, performing data analysis for purposes such as controlling do.

[0032] More specifically, embodiments of the present technique may include a method for converting a data stream (e.g., a frame) into a The process of manually tagging each occurrence of a particular object in a sequence of and object creation by reducing the manual effort and expense associated with dataset preparation. The present invention improves or replaces the process for optimally selecting a target.

[0033] Incremental real-time tagging and labeling of data streams The process of learning FIG. 1B shows a flow chart of the operational process of the smart tagging system of the invention. In this example, the system performs an action on the image, which is a human with different objects. Any data that can have an understandable 2D representation or a region of interest to be tagged. The user starts the system (100) and selects the image. Load a video or a photo file into the system. In the case of a video file, additional processing The step decomposes the video into a sequence of keyframes to reduce the number of redundant images. This decomposition is done automatically using keyframe information that is encoded into the video file. This can be done automatically, manually by the user, or both. When the frame is ready, the system checks for untagged frames (10 5) If there are no frames left, exit (110). There are frames to be tagged. In that case, the system loads the first frame (115) and performs feature extraction on this frame. In other words, the system loads the first frame and executes the The neural network on one or more processors of the system extracts the convolution output and A feature vector representing the features on one frame is created. Details of the feature extraction are described below. are outlined in the corresponding section.

[0034] At the same time, the system checks (125) whether it already has the knowledge. This knowledge is based on previously learned associations between extracted feature vectors and corresponding labels. The previously learned knowledge may include a set of extracted feature vectors and corresponding labels. If the features include associations between the features, a classifier running on one or more processors can be used to classify the extracted features. The vectors are classified by their labels. To classify the extracted feature vectors, The system performs feature matching. For example, the system may match the extracted feature vectors with Compare features (and feature vectors) with those already known to the system (e.g., previously learned knowledge) The comparison is made by comparing the feature space between the extracted feature vector and the features known to the system. A distance metric that measures the distance between the features (e.g., the Euclidean norm of the relevant feature space) The system then classifies the objects based on the difference. The difference between the features obtained and the features for the first object in the system's existing knowledge is less than a threshold. If it is smaller, the system classifies the feature as a potential first object. The actual distance or difference between the matched values ​​can be a confidence value indicating the quality of the match, or can be used to derive It can be used for this purpose.

[0035] The system can store such knowledge after a tagging session. The stored knowledge can be loaded by the user at the start of a new session. The set of currently tagged images is the same as the set of images previously tagged by users and the system. This can be especially useful when knowledge comes from the domain. The system displays the frame to the user (130) and waits for user input (135). For the first frame, for which there is no prior knowledge, the user is prompted to select the first frame via the user interface. One or more instances of an object within an image are manually tagged (140). When a user tags a first instance of a first object in an image, the system The system learns features and associated labels for tagged objects (145). Details of the fast learning classifiers involved in the stages are given in the corresponding sections below. do.

[0036] After the system learns the features of tagged objects in a frame, the system Process the frame and discover other instances of the same object within the frame. The system checks (150) whether the If you preload the frame, it will be displayed to the user through the same process before the first frame is displayed to the user. Note that it can also try to discover known objects (150). For each instance of an object that the system finds in an image, the system: Create a boundary polygon with an attached label (155) and map the boundary polygon to an image. Overlaying (160) and displaying the image with the overlaid bounding polygon and tags to the user. (130). In some cases, users may not be satisfied with the tags the system creates. The user can adjust the tags through the user interface. The inner loop (170) then learns the refined tags and updates its knowledge. The user can add new objects until they are satisfied with the tagging for this frame. The system continues to refine the system predictions by updating the tag. Checks whether there are any more frames to be sent (105), and if so, The system loads the next frame (115), performs feature extraction (120), and then starts the inner loop ( 170), where the system uses prior knowledge from at least one previous frame. Since the inner loop (170) has The system predicts (150, 155, 160) a frame before displaying it to the user (130). Note that

[0037] The entire process continues until the image is tagged or the user finishes the workflow. Before terminating, the system collects the knowledge it has gained from the user during this session. You can save it so you can reuse it in your next session.

[0038] [Operation procedure from the user's point of view] 2A-2D show the operational workflow of FIG. 1B from the user's perspective. As shown in FIG. 2A, several input modalities (e.g., mouse (200) or touch screen) are available. The user interface (210) is used to control the system operation and to tag images. The sequence of frames to be tagged can be provided to the system at installation. The directory occupied by the images on the local computer that will be downloaded. A program containing a remote directory or local or remote file name for the image. Either as lane text or markup documents, or as one or more videos In the latter case, the video is stored in the system as a file or loaded by the user. In either case, the system divides the image into a sequence of key frames. A sequence of frames (220) is defined and the first frame is loaded into the system. When the camera is ready to operate, the user will see the image on the screen. A user loads a trained version of the system described herein; It can even speed up the tagging process and reduce the manual portion of the task.

[0039] In FIG. 2B, the user may select, for example, a rectangular bounding box (230) for frame 1. Alternatively, other methods of selecting candidate regions are possible, e.g. The user can draw polygons as shown in Figures 3 to 5. The system automatically calculates the boundary polygons. Learn a range of feature combinations and map those combinations to user-provided labels (e.g. The learning process is completed within 100 milliseconds in the example implementation. The process is so fast that the system can learn the frames simultaneously. Suggests to the user about other untagged instances of the same object. In this example implementation, these suggestions also take less than 100 milliseconds to compute. This makes it very seamless for the user to use one object. Immediately after tagging, the system searches for other instances of the same object in the image ( Suggest tagging for 240).

[0040] First, make sure that the system is up and running, especially if it is not preloaded with a previously trained system. The suggestions made by the system (240) may be quite far removed from the user's point of view. The user can then reject predictions that are completely wrong and choose the correct boundary polygons. Then, adjust the incorrect labels to the correct labels as shown in Figure 2C. You can adjust the suggested boundary polygon and / or accept the correct suggestion. To simplify the process, the boundary points suggested by the classifiers (240, 260) are The polygons can be displayed as dotted lines, and are the original tagging of the user (230, 270) and the received The modifications made (250) are shown in dashed or solid lines as shown in Figures 2C and 2D. Accepted modifications can be used to further refine and reclassify specific classes of objects. This can be used to train the system and refine further suggestions for tagging. Configure a separate input.

[0041] The process then continues for the subsequent frame, as shown in FIG. 2D, where the new The new object (270) can be tagged by the user and can be used to The marked object (260) is the object that the system marks in the image. their associated classes and their associated As the system becomes more and more trained, the The interaction with the user is mainly to modify the suggestions, and therefore to accept the suggestions made by the system. This shifted to including tags in images, which is significantly less effortful than manually tagging images. , and is significantly faster. For simple taggers, Overall tagging speed has increased by up to 40%, with faster speeds for expert taggers. There was a notable, but not sudden, increase.

[0042] 3-5 show graphical user interfaces of an exemplary smart tagging system. A user launches the manual tagging tool (300). , select a label from the list of existing labels or create a new label, and then select Draw a polygon around the person in the car (310). The color of the polygon will be the one chosen in this example. Represents a label. Since this polygon was created manually, it is in the approved state (320) The result of these actions is shown in Figure 3. Then, Smart Tagging The system's fast classifier learns features within the polygon (310) and classifies them into a first The system then associates the user's label for the bike rider with the entire frame. By looking at the features, we find another cyclist in the image and create a polygon around it (400). and try to create a polygon superimposed on the other cyclists as shown in Figure 4. As polygons are created by the system, they are compared with the confidence value of the proposal. Note that all the suggestions are also marked as “suggested” (410). Restart the manual correction tool (300) for the second cyclist (500) You have updated the polygon and it now has an accepted label (510). The user can now tag other objects in the frame as desired. or use the arrow buttons (530) to go to the next frame. or complete the session by pressing the "Done tagging" button (540). It is possible.

[0043] [Smart tagging system] FIG. 6 illustrates a hardware implementation of the smart tagging utility described herein. The smart tagging system is a hardware configuration. Infrared (IR), and / or LIDAR imagery are used by robots, drones, and autonomous vehicles. , toy robots, industrial robots, and / or other devices (610). Alternative embodiments include the use of isolated or networked sensors (e.g., cameras, In addition, the data can be derived from existing databases. The data may be organized in a table (630). Visualize, tag images, visualize tagged images, and adjust tags as needed. A computing device equipped with a user interface (650) that can be used to The computing device includes a digital signal processing unit, a field processor, and a Programmable Gate Array (FPGA), Central Processing Unit (CPU), Graphics Processing Unit A GPU (graphics processing unit), and / or a processor sufficient to carry out the workflow of Figure 1B. One or more processors or processing units (670), such as a combination of processors , mobile devices (e.g., smartphones, tablets, laptops), desktops These processing units (670) can be a server, a service provider, or a service provider. It can be used to implement feature extractors and classifiers that learn and automatically adapt and adjust. The architecture includes several software modules stored in memory (680). The tagged images can be stored in the memory and loaded into a processing unit (670) for execution. , visualized in a UI (650) for display to the user (640) and stored in a separate database (660) can be stored in the

[0044] [Feature extraction module] The feature analysis module (120 in FIG. 6, 672 in FIG. 1) performs the automatic scanning described herein. It is one of the two core components of the Mart tagging system. It is displayed in the form Receives input to the system. Displays the system's graphical user interface. For convenience of classification and tagging, a 2D representation of the raw external input is preferred, but it is possible to We do not limit the formula to visual images only. Sound can be represented in 2D after Fast Fourier Transformation. Other input modalities can be rendered into a 2D representation by other means. The images can be displayed as text and fed into systems such as virtual and augmented reality systems. Advances in 3D displays will allow systems to accept and tag 3D data. This makes it possible.

[0045] The output of the feature extraction module is a set of feature vectors. Depending on the nature of the tagging The set of feature vectors is then calculated (e.g., the whole image is tagged at once for scene recognition). one feature vector per image (in the simple case where these features are It can be multiple feature vectors with relevant regions of the image found. Regions can be rectangular bounding boxes, or more complex shapes, depending on the final goal of the tagging process. It can be as simple as a polygon or even a pixel-like mask.

[0046] The exemplary implementations described herein use deep convolutional neural networks for feature extraction. Convolutional neural networks (CNNs) use convolutional units. , where the acceptance field of the unit filter (weight vector) is the height and width dimensions of the input Since each filter is small, the number of parameters is fully connected. When an object is in one spatial position, its If a set of features can be extracted for an object, the same set of features can be used to Since the features, including the points, are independent of the spatial location of the object, it is possible to distinguish them from any other spatial location. These invariants can be extracted for the same object when it also appears in the input Force coding has increased stability to visual changes, meaning that the input changes (e.g., as an object translates and rotates slightly in the image frame) This provides a feature space in which the output values ​​vary much less than the input values.

[0047] Convolutional neural networks are also good at generalization. Within the received format, the network is run on test data that is not identical to the training data. This means that multiple networks can generate similar outputs. A large amount of data is needed to learn the main patterns that govern the network. If the filter is trained on a single class, the lower layers, where the filters are shared between classes, will be similarly trained on a single class. It provides a good set of regularities for input expressions. Therefore, for a task The trained CNN can then be used as initialization for other tasks or as a sub-class of Layers can provide superior results when used as processors for new higher-level expressions. For example, natural images share a common set of statistical properties. The learned features at the lower layers are Being fairly class independent, they are not related to the class you are tagging. The class can be reused even if it is not one of the classes that the CNN was involved in. It takes these feature vectors and feeds them into the fast learning classifier part of the system. It is sufficient to provide this as input to the NI 61104 (150 in FIG. 1B).

[0048] Depending on the target final result of the tagging process, different CNNs are used as feature extractors. For whole scene recognition, Alexnet, GoogleNet, et or ResNet version is used depending on the available hardware computational power. The average pooling layer pools across image locations to generate the entire scene feature vector. must be added after the last feature layer of these networks to create the The system is used to create a dataset for training the detection network. If the average pooling is not applied, the same network can be used without average pooling, or A region proposal network such as RCNN is used instead for better spatial accuracy. If image segmentation is the target, Mask RCN Segmentation networks such as N, FCN or U-Net are used for feature extraction and Can be used to generate masks. These masks can then be applied to polygons for display and modification. For the example implementation shown in Figs. 3-5, a custom version of FCN can be used. John was used.

[0049] Alternative implementations of the feature extraction module include, but are not limited to, scale-invariant feature transforms. Transformation (SIFT), Fast Robust Features (SURF), Haar-like feature detector, dimensionality reduction regression, component analysis, and others that require system tagging Because it is clear enough for different objects and can operate fast enough The user does not wait a noticeable amount of time while the system computes the feature set. Any suitable technique for feature extraction may be used, so long as it produces a feature vector. It is possible.

[0050] [Fast learning classifier module] The fast learning classifier module (150 in FIG. 1B, 674 in FIG. 6) is a feature extraction module The feature vectors generated by (120 in FIG. 1B) are then used for classification or feature matching. It performs a filtering process to output a class label for each given set of features. The implementation can be simple template matching, where the system matches known objects. For each object, we store a template feature vector based on user input. When an input set of vectors is presented, each of these vectors is To measure the difference between the rates, some distance metric (e.g., related feature space) is used. The vectors are compared to the template vector based on the Euclidean norm between the vectors. has a difference metric smaller than a threshold set as a possible object Mark the feature vector. The inverse distance value is used as a confidence measure for this classification scheme. It can function.

[0051] Other techniques that are amenable to fast learning are substituted with classifiers for this template matching technique. The techniques include regression analysis (e.g., linear regression, logistic regression, micro regression, etc.) Nimax analysis), kernel methods (e.g., support vector machines), Bayesian models, Ensemble methods (e.g., ensemble of experts), decision trees (e.g., incremental fast decision trees and their derivatives), adaptive resonance theory-based models (e.g., F uzzy ARTMAP) and linear discriminant online algorithms (e.g., For example, the implementation shown in Figures 3-5 is a fast learning We use a modified ARTMAP for Improve statistical consistency and improve system memory as it learns new objects. Reduce the growth of the harpoon's footprint.

[0052] [Conclusion] While various inventive embodiments have been described and illustrated herein, those skilled in the art will appreciate that Various methods for performing the functions and / or results and obtaining one or more advantages described in the specification. Other suitable means and / or structures may be readily developed, and such modifications and / or Each of the variations is considered to be within the scope of the inventive embodiments described herein. In essence, one of ordinary skill in the art can easily understand all of the parameters, dimensions, materials, and The configurations are illustrative and the actual parameters, dimensions, materials and / or configurations may vary for any particular application. It will be readily understood that the method or teachings of the present invention will depend on the application for which they are used. Those skilled in the art will recognize, using no more than routine experimentation, numerous equivalents to the specific inventive embodiments described herein. Therefore, the above-mentioned embodiment are presented by way of example only and are intended to provide a fuller understanding of the invention as claimed in the appended claims and their equivalents. It is understood that the embodiments may be practiced other than as specifically described and claimed. It should be understood that inventive embodiments of the present disclosure include each individual feature, system, and method described herein. The present invention covers articles, materials, kits, and / or methods. Any combination of the features, systems, articles, materials, kits, and / or methods may be used. the features, systems, articles, materials, kits, and / or methods disclosed therein are not mutually inconsistent, This disclosure is included within the scope of the present invention.

[0053] The above-described embodiments can be implemented in any of numerous ways. The embodiments of the technology disclosed herein may be implemented in hardware, software, or a combination thereof. When implemented in software, the software code may be , whether it is provided on a single computer or distributed across multiple computers; Regardless, the implementation may be implemented on any suitable processor or collection of processors.

[0054] In addition, computers can be rack mounted, desktop, or A computer in one of many forms, such as a laptop or tablet It should be understood that the computer may be embodied in a personal digital assistant (PDAS) or a computer-based computer. a personal digital assistant (PDA), smartphone or any other suitable handheld or Any fixed electronic device generally regarded as a computer, but with adequate processing power It may be embedded within the device.

[0055] A computer may also have one or more input and output devices. , among other things, can be used to present a user interface. Examples of output devices that can be used to provide an interface include an output and a speaker. A printer or other sound-producing device for a visual representation of the output, or an audible representation of the output. Examples of input devices that can be used in a user interface include a display screen. The HDMI port supports keyboards and point devices such as mice, touchpads, and digitizer tablets. As another example, a computer may be used for speech recognition or other audible filtering. The input information may be received in a format.

[0056] These computers can be part of a local area network or an enterprise network. Wide area networks such as networks, Intelligent Networks (IN) or Inter-networks The networks may be interconnected by one or more networks of any suitable form, such as a cellular network. Such networks may be based on any suitable technology and may use any suitable protocol. The network may operate according to a wireless network, a wired network, or optical fiber. The network may include:

[0057] The various methods or processes outlined herein may be implemented using a variety of operating systems or or platform. Moreover, such software may be coded as software. Any suitable programming language and / or programming or scripting tool It may be written using any of the above and may be implemented to run on a framework or virtual machine. It may be compiled into executable machine code or intermediate code.

[0058] In this regard, various inventive concepts may be implemented on one or more computers or other processors. One or more of the above-described methods for implementing the various embodiments of the present invention, when executed in accordance with the present invention. A computer-readable storage medium (or a plurality of computers) encoded with a program data-readable storage medium) (e.g., computer memory, one or more floppy disks, Compact disk, optical disk, magnetic tape, flash memory, field programmer Circuitry of a power gate array or other semiconductor device or other non-transitory medium or The present invention may be embodied as a computer readable medium or a tangible computer storage medium. The body, as described above, may have a program or programs stored thereon that can be read in one or more different ways. and loading the program into a computer or other processor to implement various aspects of the present invention. It may be movable so that

[0059] The term "program" or "software" is used herein as defined above. Any program that can be used to program a computer or other processor Generally refers to a type of computer code or set of computer executable instructions. It will be further understood that, according to one aspect, One or more computer programs implementing the methods of the present invention may be implemented on a single computer or The present invention may be implemented in a variety of different ways, without necessarily residing on one or more processors. The present invention may be implemented in a modular manner and distributed among multiple computers or processors.

[0060] Computer-executable instructions are instructions executed by one or more computers or other devices. The program may take many forms, such as a program module executed by the program. A module contains rules that perform a particular task or implement a particular abstract data type. This includes functions, programs, objects, components, and data structures. The functionality of the program modules may be combined as desired in various embodiments. The particles may be dispersed or dispersed.

[0061] Also, data structures may be stored in computer-readable media in any suitable form. For ease of illustration, the data structure has fields that relate to the location of the data structure. Such relationships may be represented as computer-generated representations that convey relationships between fields. This may also be accomplished by allocating storage of fields with locations within the readable medium. However, pointers, tags or other mechanisms that establish relationships between data elements may be used. Use appropriate procedures to establish relationships between information in the fields of a data structure, including using Any mechanism may be used.

[0062] Also, various inventive concepts may be embodied in one or more methods, examples of which are provided. The actions performed as part of the methods may be ordered in any suitable way. It is therefore possible to create embodiments in which the operations are performed in an order different from that illustrated. Even though it is shown as a continuous operation in the exemplary embodiment, It may involve performing several actions simultaneously.

[0063] All definitions defined and used herein are dictionary definitions, which are incorporated by reference. The definitions in this document should be understood to govern the ordinary meaning of the terms defined. It is.

[0064] As used in this specification and claims, the indefinite articles "a" and "an" are expressly intended to Unless otherwise indicated, "at least one" should be understood to mean "at least one."

[0065] As used in this specification and claims, the term "and / or" is used interchangeably. means "either or both" of the elements listed, i.e., in some cases, "and" should be understood to mean elements which are present in other cases and which are present separately in other cases. Multiple elements listed with "and / or" are treated in the same form, i.e., as "one" of the combined elements. The other elements are not relevant to the elements specifically identified. Any element identified by an "and / or" clause, whether relevant or unrelated, Thus, as a non-limiting example, "A and / or B" References to, when used in conjunction with open-ended language such as "including," are intended to mean, in one embodiment, In the above, only A (optionally including elements other than B) can be referred to, and another embodiment In this case, it can refer to only B (optionally including elements other than A) and can refer to another In an embodiment, it can refer to both A and B (optionally including other elements). And so on.

[0066] As used in the specification and claims, "or" means any of the above defined terms. "and / or" should be understood to have the same meaning as "and / or." For example, When separated, "or" or "and / or" will be construed as inclusive and That is, a number of elements or a list of elements, and optionally additional items not included in the list. shall be construed as including at least one of the following, but also including two or more. Only the terms indicated in, e.g. "only one of" or "exactly one of", or When used in the claims, the term "comprising" alone shall mean a number of elements or links. Generally, as used herein, "inclusion of exactly one element of a list" refers to the inclusion of exactly one element of a list. The term "or" is used in conjunction with terms of exclusivity, e.g., "any," "one of," "of," When preceded by "only one" or "exactly one of", it indicates exclusive alternatives (i.e. "Consists essentially of" is interpreted as meaning "either one or the other, but not both." When used in the claims, they have their ordinary meaning as used in the field of patent law. It shall be so.

[0067] As used herein, the terms "about" and "approximately" refer to a range of values ​​stated. This generally means minus 10%.

[0068] As used in the specification and claims, "at least" refers to a list of one or more elements. The phrase "at least one" refers to at least one selected from any one or more of the elements of the list of elements. Each element refers to an element, but not necessarily to the elements specifically listed within the list of elements. It does not necessarily contain at least one of each of the elements in the list of elements. It should be understood that no combination of elements is excluded. Specifically identified within the list of elements referred to by the phrase "at least one" Elements other than those specifically identified, whether or not related to the elements specifically identified, Thus, as a non-limiting example, "A and B" may be at least one of" (or, equivalently, "at least one of A or B"; (equivalently, "at least one of A and / or B") in one embodiment , B is absent, and A (optionally including at least one, optionally including two or more) In another embodiment, A is absent and at least It also refers to one, and optionally two or more, B (optionally including elements other than A). In another embodiment, at least one, and optionally two or more A including the above, and B including at least one, optionally two or more (and optionally Alternatively, it may refer to other elements.

[0069] In the claims and in the above specification, all transitional words, such as "comprising," "including," "carrying," "having," "contain "involving," "holding," "consisting" and and the like are understood to be open-ended, i.e., including but not limited to. Only the transitional phrases "consisting of" and "consisting essentially of" are used in the U.S. Patent 2111.03, respectively. It shall be a semi-closed transitional phrase.

Claims

1. 1. A method for tagging a sequence of images, comprising the steps of: tagging, by at least one processor, a representation of a first object in a sequence of images; tagging, by a user, a first instance of a representation of a second object in a first image in the sequence of images; learning, by a fast-learning classifier executing on the at least one processor, a representation of the second object tagged by the user in the first image without the fast-learning classifier being retrained on data previously presented to the fast-learning classifier; tagging, by the at least one processor, second instances of the representation of the second object in the sequence of images, comprising: (1) extracting, by a feature extraction neural network executed on the at least one processor, convolution outputs representative of features of the second instances of the representation of the second object; and (2) classifying, by the fast learning classifier, the second instances of the representation of the second object based on the convolution outputs. performing, by the user, an adjustment of a tag and / or a position of the second instance of the representation of the second object created by the at least one processor; tagging, by the at least one processor, a third instance of a representation of the second object within the sequence of images based on the adjustment; and The method includes:

2. The method of claim 1 , wherein the second instance of the representation of the second object is in the first image in the sequence of images.

3. The method of claim 1 , wherein the second instance of the representation of the second object is in another image in the sequence of images.

4. performing, by the user, an adjustment of a tag and / or a position of the third instance of the representation of the second object created by the at least one processor; tagging, by the at least one processor, a fourth instance of the representation of the second object within the sequence of images based on the adjusting of the tag and / or position of the third instance of the representation of the second object; The method of claim 1 further comprising:

5. 2. The method of claim 1, further comprising the step of: the fast learning classifier executing on the at least one processor classifying a representation of the second object tagged by the user in the first image.

6. tagging the third instance of the representation of the second object, extracting, by the feature extraction neural network, a convolution output representative of features of the third instance of the representation of the second object; classifying, by the fast learning classifier, the third instance of the representation of the second object based on the convolution output; and The method of claim 5 , comprising:

7. 1. A system for tagging a sequence of images, comprising: a user interface that enables a user to tag a first instance of a representation of an object in a first image in the sequence of images; at least one processor operatively connected to the user interface for executing a neural network to extract a first convolution output representative of features of the first instance of a representation of an object in the first image, a second convolution output representative of features of a second instance of a representation of an object in the sequence of images, and a third convolution output representative of features of a third instance of a representation of an object in the sequence of images, and for executing a fast-learning classifier to learn the representation of the object tagged by the user in the first image without being retrained on data previously presented to the fast-learning classifier, and to tag the second instance of the representation of the object in the sequence of images based on a comparison of the first convolution output and the second convolution output; Including, the user interface allows the user to perform an adjustment of tags and / or positions of the second instances of the representations of the object created by the at least one processor, and the fast learning classifier is configured to tag the third instances of the representations of the object within the sequence of images based on the adjustment and the third convolution output. system.

8. 8. The system of claim 7, wherein a user interface enables the user to adjust a tag and / or position of the third instance of the representation of the object created by the at least one processor, and the at least one processor is configured to tag a fourth instance of the representation of the object within the sequence of images based on the adjustment of the tag and / or position of the third instance of the representation of the object.

9. The system of claim 7 , wherein the second instance of the representation of the object is in the first image in the sequence of images.

10. The system of claim 7 , wherein the second instance of the representation of the object is in a second image in the sequence of images.

11. The system of claim 7 , wherein the fast learning classifier is configured to classify representations of the objects tagged by the user in the first image.

12. 1. A method for tagging objects in a data stream, comprising the steps of: extracting, by a neural network executed on at least one processor, a first convolution output from a data stream, the data stream including at least two representations of a first class of objects, the first convolution output representing features of a first representation of a first category of objects; classifying, by a classifier executing on the at least one processor, the first representation into the first category of object based on the first convolution output; displaying, via a user interface coupled to the at least one processor, tags and / or locations for the first representation based on the first category; performing an adjustment of said tags and / or said positions for said first representation by a user via said user interface; learning, by the classifier, the tag and / or the location of the first representation of the object based on the adjustment without the classifier being retrained on data previously presented to the classifier; The method includes:

13. the tag is a first tag and the location is a first location; extracting, by the neural network, a second convolution output from the data stream, the second convolution output representing features of a second representation of the first category of objects; classifying, by the classifier, the second representation into the first category based on the second convolution output and the adjustment of the tag and / or the position for the first representation; displaying, via the user interface, a second tag and / or a second location based on the first category; The method of claim 12 further comprising:

14. determining, by said classifier, a confidence value that said tag and / or position of said first representation is correct; displaying the confidence value to the user; The method of claim 12 further comprising:

15. the object is a first object and the tag is a first tag; 13. The method of claim 12, further comprising: learning, by the classifier, second tags for a second category of objects, the data stream including at least one representation of the second category of objects.

16. extracting, by the neural network, a subsequent convolution output from a subsequent data stream, the subsequent data stream including at least one other representation of the second category of objects, the subsequent convolution output representing features of the at least one other representation of the second category of objects; classifying, by the classifier, the at least one other representation of the second category of object into the second category based on the subsequent convolution output and the second tag; displaying the second tag via the user interface; The method of claim 15 further comprising:

17. The step of extracting the first convolution output comprises: generating a plurality of divided sub-areas of a first image of the data stream; encoding each of the plurality of divided sub-areas by the neural network; 17. The method of claim 16, comprising:

18. At least one processor, a neural network for extracting a first convolution output from a data stream, the data stream including at least two representations of a first category of objects, the first convolution output representing features of the first representation of the first category of objects; and a rapid learning module that classifies the first representation into the first category based on the first convolution output and learns tags and / or locations of the first representation of the object based on user adjustments without retraining on previously presented data; and a user interface coupled to the at least one processor and configured to display the tag and / or the location for the first representation and to accept the adjustment of the tag and / or the location of the first representation from the user; A system including:

19. the tag is a first tag, the position is a first position, the neural network is configured to extract a second convolution output from the data stream, the second convolution output representing features of a second representation of the first category of objects; the fast learning module is configured to classify the second representation into the first category based on the second convolution output and the adjustment of the first tag and / or the first position for a first instance of the first representation of the object; The system of claim 18 , wherein the user interface is further configured to display a second tag and a second location based on the first category.

20. the fast learning module is configured to determine a confidence value that the first tag and / or the first location is correct; The system of claim 18 , wherein the user interface is further configured to display the confidence value to the user.

21. 20. The system of claim 18, wherein the tag is a first tag, the rapid learning module is further configured to learn a second tag for a second category of objects, and the data stream includes at least one representation of the second category of objects.

22. the neural network is further configured to extract a subsequent convolution output from a subsequent data stream, the subsequent data stream including at least one other representation of the second category of objects, the subsequent convolution output representing features of the at least one other representation of the second category of objects; the fast learning module is further configured to classify the at least one other representation of the second category of objects into the second category based on the subsequent convolution output and the second tag; The system of claim 21 , wherein the user interface is further configured to display the second tag.

23. The neural network, generating a plurality of partitioned sub-areas of a first image of the data stream; The system of claim 18 , further configured to code each of the plurality of divided sub-areas.

24. prior to learning objects in the first image tagged by the user, extracting a feature vector in the first image; checking associations between features of the feature vector and labels corresponding to the features that have been learned so far; The method of claim 1 further comprising:

25. 25. The method of claim 24, further comprising classifying, with the fast learning classifier, objects in the first image based on the associations learned so far.

Images