System for and method of allowing continuous learning limited by memory in artificial intelligence and deep learning continuously operating application over network calculation edge

Abstract

To carry out analysis of an object in an environment by efficient learning of a lifetime deep neural network (L-DNN).SOLUTION: An L-DNN technique accompanies a high-speed learning subsystem (module B) such as an adaptive resonance theory network so that it should not forget previously learnt features and learns new features rapidly and also uses a subsystem (module A) based on a deep neural network (DNN) having rich expression. When classifying an object in an environment, a sensor collects a data stream representing the object in the environment. The module A based on the DNN operating on a processor operably connected with the sensor extracts a convolution output representing a feature of the object from the data stream. The module B including a classifier connected to the DNN classifies the object using the feature of the object.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a continuation of U.S. Patent No. 6,929,999 filed December 31, 2017 under 35 U.S.C. § 119(e). Application No. 62 / 612,529, filed May 9, 2017, and U.S. patent application Ser. No. 503,639, each of which is incorporated by reference in its entirety. and is incorporated herein by reference. [Background technology]

[0002] Deep neural networks that contain many layers of neurons placed between the input and output layers Traditional neural networks, including deep learning networks (DNNs), are limited to specific datasets. In contrast, it requires thousands or even millions of iteration cycles for training. These cycles occur frequently on high performance computing servers. Depending on the size of the dataset, some traditional DNNs can take days or weeks to train. It may take some time.

[0003] One technique for training DNNs involves the backpropagation algorithm. The rhythm is achieved by applying the chain rule to backpropagate the error gradient. Calculate the change in all weights of the DNN proportional to the error gradient from the filtered dataset. changes the weights for each data point by a small amount, spreading it across all data points in the set for many epochs. .

[0004] The larger the learning rate per iteration, the more likely the gradient of the loss function will be a local minimum instead of a minimum. loss function, which may result in poor performance. To increase the chances of settling on a minimum, the DNN reduces the learning rate, thereby , weights are changed slightly in every training epoch. This allows The number of cycles and total learning time increases.

[0005] Advances in graphics processing unit (GPU) technology have made it possible to Highly parallel operations used to accomplish training jobs that took hours or months These jobs can now be run in minutes or even hours on a GPU. This can be completed in a few days, but this is still not fast enough to update knowledge in real time. Furthermore, if a high-performance computing server is used to update the DNN, the server price and This is costly in terms of time and energy consumption. It is often desirable to update the knowledge of DNN-based systems on the fly. , becomes very difficult.

[0006] Furthermore, the gradient of the loss function calculated for any single training example is Since it can affect all weights in the network (due to the usual distributed representation), DNNs tend to forget previous knowledge when learning new objects. Repetitive representation of the same input mitigates this problem, but it is difficult to quickly add new knowledge to the system. This is because learning is a computationally limited process. Edge devices (e.g., mobile phones, tablets, or small form factor processors) ) is difficult or completely impossible to achieve. Even if this problem is solved, learning on edge devices will require high computational load for training, small The small training steps and recursive representation of all inputs may still be infeasible. .

[0007] These limitations are due to the single point in time that an edge may need to update its knowledge over the life of the deployment. Rapidly deploy newly acquired knowledge not only at the compute edge but across the deployment lifecycle In distributed multi-edge systems ( For example, smartphones connected in a network, networked smartphones This also applies to cameras, drones or fleets of autonomous vehicles, and similar.

[0008] The processor running the backpropagation algorithm calculates the error contribution of each neuron at the output. The weights of all neurons are then calculated as a loss function The gradient of the network is then calculated, so that the network can correctly interpret old examples. To avoid losing the ability to classify, new training examples are added without retraining old examples. However, it cannot be added to a pre-trained network. This loss of ability is called "catastrophic forgetting." This forgetting problem occurs when In real time, where there is often a need to rapidly learn and incorporate new information on the fly This is particularly relevant when considered in connection with a working machine.

[0009] Real-time machines that use traditional DNNs to learn knowledge are It may be necessary to accumulate a large amount of data in order to retrain the NN. The data is then processed by the DNN that runs on the edge, after getting the labels from the operator. To train it, we need machines that operate in real time, called "edges" (i.e., from the device itself (e.g., a self-driving car, drone, robot, etc.) to a central server (e.g., The more data that is stored, the more time it takes to The transfer process is more expensive in terms of network bandwidth and data transfer costs. Interleaving training on the system allows new data to be applied to the system's lifecycle. It must be combined with the original data that is stored for the entire cycle. Transmission bandwidth and data storage limitations are created.

[0010] In summary, we aim to improve the performance of traditional backpropagation-based DNN training in real-time operating environments. When applied to a network system, it suffers from the following drawbacks: a. It is not possible to update the system with new knowledge on the fly. b. Edge computing without regular communication with the server and significant latency for knowledge updates. Learning throughout the deployment cycle is not possible. c. Learning new information requires storing all input data indefinitely for further training. Storing data requires server space, energy consumption, and disk space consumption. d. Learning cannot be done on small form factor computing edge devices. e. Multiple edge deployments are not possible without slow and expensive server-side retraining and redeployment. Inability to merge knowledge across pages. Summary of the Invention

[0011] Lifelong Deep Neural Networks (L-DNN) perform time-consuming computations It is possible to develop artificial neural networks on lightweight computing devices (edge) without the need for intensive training. Neural Networks (ANN) and Deep Neural Networks (DNN) L-DNN enables continuous online lifelong learning. This allows real-time training from the input data for multiple iterations of backpropagation training. This avoids the need to store data.

[0012] L-DNN technology allows for fast yet stable learning of features that represent entities or events of interest. To achieve this, we developed a subsystem (Module A) based on a representation-rich DNN for rapid learning. Combine with the subsystem (Module B). These feature sets are used in low-level programming, such as backpropagation. It can be pre-trained using fast learning methodologies, as detailed in this disclosure (as well as other features). (The description of this is possible by adopting a non-DNN methodology in Module A)DN For N-based examples, the high-level feature extraction layer of the DNN classifies known entities and events. and the high level of module B to add knowledge of unknown entities and events on the fly. It serves as an input to a fast-learning system. Module B can learn to: It can learn important information and capture environmental features that are descriptive and highly predictive.

[0013] L-DNN techniques are used to analyze visual data, structured light data, and Lidar (LI), among other modalities. DAR data, SONAR data, RADAR data, or It can be applied to audio data. For visual data or similar data, the L-DNN technique: Global image classification (e.g., scene detection), bounding box-based object recognition, pixel-wise segmentation It can be applied to visual processing, such as enabling mentation and other visual recognition tasks. L-DNN techniques are also useful for non-visual recognition tasks such as non-visual signal classification, and for robots, autonomous vehicles, and other applications. As a self-driving car, drone, or other device navigates its environment, it gradually acquires knowledge. By adding SLAM, it is possible to simultaneously estimate the position and create a map of the environment. It can also perform other tasks, such as updating the generated map.

[0014] L-DNN can learn more entities or events (in visual terms, "objects" or "categories"). When learning the ML-DNN, we reduce the memory requirements by aggregating memory within the L-DNN. Module B is under the control of the L-DNN methodology. Computing devices move across the edge to classify their knowledge (or input data) The merge is a new function between two modules B. by direct exchange of the neural network representation or by multiple modules from some edges. This can happen peer-to-peer, via an intermediate server, merging the representations of Rule B and Rule B. Additionally, L-DDN does not rely on backpropagation, thereby reducing training time, power requirements, and It uses new input data to update the L-DNN knowledge, dramatically reducing the computational resources required. do.

[0015] Of course, all of the above concepts and the additional concepts discussed in more detail below Combinations (provided such concepts are not mutually exclusive) may be used in any combination disclosed herein. In particular, the claims appearing at the end of this disclosure are considered to be part of the subject matter of the present invention. All combinations of subject matter described in the ranges are part of the inventive subject matter disclosed herein. It is also understood that any disclosure incorporated by reference is expressly Terms used expressly are to be given the meaning that most closely matches the specific concepts disclosed herein. It is necessary.

[0016] Upon review of the following diagrams and detailed descriptions, other systems, processes, and features may be applicable. Such additional systems, processes, and features will become apparent to those skilled in the art. All such modifications and variations are intended to be included within this description, be within the scope of the present invention, and be protected by the accompanying claims. It is intended that [Brief explanation of the drawings]

[0017] Those skilled in the art will appreciate that the drawings are primarily for illustrative purposes and that the It will be understood that no limitation of the scope of the present subject matter is intended. In some instances, various aspects of the inventive subject matter disclosed herein are not to scale. may be shown exaggerated or enlarged in the drawings to make different features easier to understand. In this respect, like reference characters generally refer to like features (e.g., functionally similar and / or or structurally similar elements).

[0018] [Figure 1] Figure 1 shows an overview of a Lifelong Deep Neural Network (L-DNN) when it is associated with multiple computational edges, either operating on data streams individually or connected peer-to-peer or via intermediate computational servers. [Figure 2] Figure 2 shows an example L-DNN architecture. [Figure 3] Figure 3 shows the implementation of the concept of unknowns in neural networks. [Figure 4] Figure 4 shows a VGG-16 based L-DNN classifier as an example implementation. [Figure 5]Figure 5 shows heterogeneous multi-scale object detection. [Figure 6] Figure 6 shows an L-DNN based on Mask R-CNN for object segmentation. [Figure 7A] FIG. 7A shows aggregation and blending using an adaptive resonance theory (ART) neural network. [Figure 7B] Figure 7B shows how locally ambiguous information, e.g., a pixelated image of a camel (in the desert in the first scene) or a dog (in the suburbs in the second scene), can be found using global information about the scene and past associations learned between objects. [Figure 8] Figure 8 shows the application of L-DNN to the drone-based industrial survey use case. [Figure 9] In Figure 9, we extend the drone-based industrial survey use case from Figure 8 to a situation where multiple drones with L-DNNs operate in concert. [Figure 10] Figure 10 shows the application of L-DNN to a warehouse inventory use case. [Figure 11] Figure 11 shows a large number of smart devices using L-DNN to collectively acquire and share knowledge. [Figure 12] Figure 12 shows an example where L-DNN replaces traditional DNN in a data center-based application. DETAILED DESCRIPTION OF THE INVENTION

[0019] Real-time machine learning Lifelong Learning Deep Neural Networks or Lifelong Deep Neural Networks (L-DNN) allows machines to operate in real time, and the machines can run on a central server or cloud. This allows for on-the-fly learning at the edge, without the need for on-the-fly learning. to eliminate network latency, increase real-time performance, and prioritize when necessary. In some cases, real-time machines use L-DNNs. For example, L-DNN can be used to perform survey dropouts. A robot can learn how to identify problems at the top of a cell tower or solar panel, No privacy concerns as data is not shared outside of the local device Instead of relying on a smartphone, users can personalize their smart toys based on their preferences, Knowledge learned at the edge, without having to send information to a central server for learning, which always takes a long time. knowledge can be shared (peer-to-peer or globally with all devices) or autonomous vehicles can When it works, knowledge can be learned and shared.

[0020] L-DNN also makes it possible to learn new knowledge without forgetting old knowledge. In other words, this technology: A machine that operates in real time can: a) capture input images without the need to transmit or store them; and b) and c) without significant computing resources. This allows for continuous and optimal behavioral adjustments at the edge based on user input. By learning using L-DNN, a machine operating in real time can learn to understand its environment and users. It adapts to changes in user interactions, addresses deficiencies in the original dataset, and provides customized It becomes possible to provide a user with an experience.

[0021] The disclosed technology also allows for merging knowledge from multiple edge devices. is a "cloud collection," as well as the labeling and dissemination of this collected knowledge. Includes sharing across edge devices, eliminating tedious and intensive labeling hours In other words, the brains from one or more of the edge devices are transferred from one to another (pilot). peer-to-peer merge or pushback to some or all of the edge devices L-DNN allows for either one of the two to be shared and merged into a common brain. Merging / mixing / sharing / combining knowledge grows linearly with the number of objects. This results in an increased footprint and occurs in real time, resulting in data being exchanged between devices. These features ensure that L-DNN can be used in real-world situations. This makes it practical for applications in

[0022] L-DNN is a heterogeneous neural network characterized by the following two modules: Implement a network architecture. 1) Pre-trained and fixed in the factory, or backpropagated, or data input or configured to be trained by other learning algorithms based on the sequence of Slow learning, including neural networks (e.g., deep neural networks) Module A of Learning. 2) Instantaneous changes to synaptic weights and representations with few training samples. Module B provides a gradual classifier that can be updated. An instance of this gradual classifier is Examples of neural networks include, for example, Adaptive Resonance Theory (ART) networks, or contrastive neural networks. Training neural networks with divergence, the Restricted Boltzmann Machine (RBM), and support vector machines (SVM) or other fast learning supervised learning This includes non-neural methods such as class processes.

[0023] Typical applications of L-DNN include, but are not limited to, using Internet of Things (IoT) devices that learn patterns of behavior and driving styles Can it adapt to the user's needs, quickly learn new skills on the fly, or new roads, and new roads, with autonomous vehicles and on-the-fly repairs to damage to infrastructure. A drone that can learn a new class and can distinguish this damage during operation after a short period of learning. (Near) instant learning without pinging the cloud to identify ownership Domestic robots, such as toys or companion robots, and robots never seen before. Recognize and react to new objects, avoid new obstacles, and discover new objects in the world map. It can learn to manipulate robots, new parts, and how to manipulate them on the fly. Able to learn, industrial robots and new individuals or objects and connect to the network Security cameras and cameras that can be quickly detected in images provided by other cameras. The above application is overcome and made possible by the innovations described herein. The training is done on a server using expensive and lengthy iterative training. Embedded computing in specific applications without the need for This can occur directly on the device.

[0024] The techniques disclosed herein can be used to analyze video streams, data from active sensors (e.g., Infrared (IR) imagery, lidar data, sonar data, and similar), acoustic data , and other time-series data (e.g., sensor data, real-time data including data generated in factories) data streams, IoT device data, financial data, and similar), and This includes any multimodal linear / non-linear combination of such data streams, It is applicable to several input formats, but is not limited to these.

[0025] Overview of L-DNN As disclosed above, L-DNN combines fast and slow learning modes. We implement a heterogeneous neural network architecture to enable fast learning. The paper proposes that real-time machines implementing L-DNN can respond to new knowledge almost instantly. This mode allows you to learn new knowledge and new experiences so that you can answer questions quickly. The learning rate of the brain system is high enough to favor new knowledge and corresponding new experiences. On the other hand, the learning rate of the slow learning subsystem is set to preserve old knowledge and corresponding old experiences. otherwise, it is set to a low value or zero.

[0026] Figure 1 shows a master edge / central server and several compute edges (e.g., drones) , robots, smartphones, or other IoT devices) to run L-DNNs. We provide an overview of the L-DNN architecture, where multiple devices work in concert. The device receives sensory input 100 and transmits it to a slow learning module A102 and a fast learning module A103. Each module A is provided with a learning module B 104 and supplies the learning module B 104 to the corresponding L-DNN 106. 102 acts as a feature extractor based on a pre-trained (fixed weight) DNN. Module A 102 receives input 100 and extracts relevant features into a compressed representation of the object. and supplies these representations to the corresponding module B104. These object representations can be learned quickly through user interaction. receives the correct labels for unknown objects and compares each feature vector with the corresponding label. They quickly learn associations between these objects, resulting in immediate recognition of these new objects. The L-DNN106 mixes newly acquired knowledge when learning different inputs. (combined, merged, or combined) and then combine it with other L-DNNs, as disclosed below. Can be connected peer-to-peer (dashed line) or to a central server (dotted line) for sharing with 06 .

[0027] The example of L-DNN implementation for object detection presented below is based on the traditional object detection DNN "You o The next test will compare it to "You only look once" (YOLO). The same custom dataset, smaller (600 images), with one object produced the results. We trained and validated both networks using the same image. Four training sets of different sizes (100, 200, The L-DNN train (300, 300, and 400 images) was generated from the remaining 400 images. In training, each image in the training set was presented once. creates batches by randomly shuffling the training set, Training proceeded by repeating these batches multiple times. After training, both networks Validation is performed on the network, and the mean average precision (mAP) is calculated as follows: This resulted in a positive decision. [Table 1]

[0028] Additionally, training of L-DNN using a training set of 400 images The training time was 1.1 seconds, and the training time for YOLO was 21.5 hours. This is a surprisingly large performance improvement. The memory footprint of L-DNN is 320MB. , while YOLO's footprint was 500MB. These results are DNN achieves better accuracy than traditional DNN YOLO and with a smaller dataset , and be able to do so with faster training times and lower memory requirements. This clearly shows that:

[0029] Example of an L-DNN architecture Figure 2 shows the real-time data flow of a robot, drone, smartphone, or IoT device. This shows an example of an L-DNN architecture used by a machine operating in real time. NN106 consists of two subsystems: a slow learning module A102 and a fast learning module A103. In one implementation, module A uses a pre-trained DNN. Module B is based on the Adaptive Resonance Theory (ART) paradigm for fast learning, and The ART is then fed to one of the latter feature layers (usually the DNN itself classifies the fully connected layer). Multiple DNN layers feed the output of one or more can provide input to module B above (e.g., in multi-scale, voting, or hierarchical formats) ), other configurations are possible.

[0030] An input source 100, such as a digital camera, a detector array, or a microphone, collects information from the environment. / data (e.g., video data, structured light data, audio data, combinations thereof, and / or the like). If the input source 100 includes a camera system, It can acquire a video stream of the environment surrounding the machine in real time. The input data from the module A102 is processed in real time by the module A102. A102 provides the compressed feature signal as input to module B104. In the example, the video stream is a series of images sent in real time by modules A and B. Module A and Module B can be treated as a frame. A graphics processor with volatile memory and suitable input / output interfaces. a processor unit, field programmable gate array, or application specific integrated circuit It can be implemented on any suitable computer processor.

[0031] In one implementation, the input data is processed by a pre-trained deep neural network in module A. The DNN 200 is fed into a neural network (DNN) 200, which is described in the Example Implementation section. As detailed in the section below, features that can be used to represent the input information / data are extracted. The DNN 200 preferably includes a stack 202 of convolutional layers 204 that are used to It can be pre-trained in the factory to achieve a high level of data representation before deployment. The NN200 uses a configuration file that determines its architecture and the The metric can be completely defined by a set of weights representing the knowledge to be acquired.

[0032] The L-DNN system 106 takes advantage of the fact that DNN weights are good feature extractors. A module B104 including one or more fast learning neural network classifiers is used. ,To connect to DNN200 of module A102, the first DNN is used for classification. Do we ignore only some of the upper layers of the DNN (e.g., layers 206 and 208 in Figure 2)? or even removed from the system altogether. The convolution output of the module B104 is accessed and serves as an input to the module B104. For example, the original DNN200 mostly uses gradient descent techniques to optimize weights during training. In addition to the cost layer 208, multiple fully connected averaging techniques are used to make it possible These layers are used during DNN training or 200 to obtain predictions directly, but generates input for module B 104. (The shading in FIG. 2 indicates that layers 206 and 208 are unnecessary.) Instead, the input to the neural network classifier of module B104 is a DNN The layers are taken from a subset of the 204 convolutional layers. Different layers or multiple layers can be used to The input to module B104 can be provided by the oscilloscope.

[0033] Each convolutional layer on the DNN200 uses local receptive fields to extract information from a small region in the previous layer. These filters are then passed through the convolutional layers of the DNN. The output from one or more later convolutional layers 204 of the feature extractor (illustratively The neural network component in module B104 is The input neurons are fed to the input neuron layer 212 of a classifier (e.g., an ART classifier). As detailed in the section below, L-DNN106 is used for global image classification or object detection. Depending on how the module A102 is designed, each later convolutional layer 204 and the module B104 each fast learning neural network classifier with one-to-one or one-to-one There can be a one-to-many correspondence.

[0034] The tensor 210 transmitted from the DNN 200 to the module B system 104 is the original input It can be viewed as a representation of an n-layer stack from force data (e.g., raw images from sensor 100). In this example, each element of the stack represents the same spatial topography as the input image from the camera. Each grid element across n stacks is represented as a module B is the actual input to the neural network.

[0035] The neural network classifier in the initial module B is trained on the fly after deployment. To facilitate learning, any initial knowledge or training in module A102 The input source 100 can then process data related to the environment into the L-DN When provided to N106, the neural network classifier uses the data from DNN200 ( For example, tensor 210) is processed sequentially. Neural network of module B The classifier uses fast, preferably one-shot learning. The ART classifier is bottom-up (input-based) learning between neuron-like elements ) and top-down (feedback) associative projection, as well as competition between categories. Use a horizontal projection to do this.

[0036] In fast learning mode, a new feature set is presented as input from module A102. When this is done, the ART-based module B104 sends the features to the F1 layer 212 as input vectors and calculate the distance between this input vector and the existing weight vector 214. ,F2 layer 216 determines the activation of all category nodes.,The distance is calculated by fuzzy AND (AR the default version of T), the dot product, or the Euclidean distance between vector ends. The category nodes are then sorted from most activated to least activated. The categories are then sorted into the most desirable order, and a competition is held between the category nodes, with the categories in this order considered to be potential winners. If the label of a potential winner matches the label provided by the user, the corresponding The weight vector is updated to match the new input and the existing input to the winning node in the simplest implementation. generalize new inputs through a learning process that takes a weighted average with the weight vector of If there is no winner with the correct label, the new category node will include a copy of the input. In both cases, Module B 104 now knows this input and can recognize it on the next presentation.

[0037] The result of module B104 depends on the task that L-DNN106 is solving. as the output of L-DNN106 in the field, or from a specific DNN layer from module A102. It works either as a combination of outputs from the When classifying body images, the output of module B may be sufficient. Module B 104 overlays the bounding box determined from the activity of module A. The class labels are provided so that each object can be correctly identified by module A102. They are found and correctly labeled by module B104. For this purpose, the bounding box from module A102 is placed on the pixel-wise mask. The masks may be replaced and module B 104 may provide labels for these masks. Further details of module A 102 and module B 104 are provided below.

[0038] Real-time behavior and the concept of the unknown in neural networks. The normal L-DNN and specific module B operate in real time in response to continuous sensory input. The neural network in module B is designed to operate on what it knows. It should be implemented so that it doesn't get confused when no body is presented at all. The neural network targets datasets that typically contain labeled objects in the input. As a result, if there is no known object, there is no need to act on the input. Therefore, to use such a network in module B of L-DNN, A further special category called "Nothing I know" was created in the network. Module B attempts to add to the queue and mistakenly classify unknown objects as known (false positives). The impact of these measures should be reduced.

[0039] This concept of "knowing nothing" exclusively encompasses previously invisible and unlabeled objects. This is useful when processing live sensory streams. and L-DNN potentially misidentifies unknown objects as known objects instead of In other words, we describe unfamiliar objects as "unknown" or "not seen before" (not previously). It becomes possible to distinguish between "not knowing anything" and "not knowing anything" Extending a conventional design with a bias node is equivalent to adding a bias node to the network. The concept of "knowing nothing" can also be expressed as the number of known object classes and their These can be implemented in a version that automatically increases or decreases the influence according to the corresponding activation.

[0040] One possible implementation of the concept of "knowing nothing" is the uniform distribution of knowledge into several categories. In contrast to the internal knowledge distribution, it clearly favors predictions that are concentrated in common categories. In other words, the neural network of Module B A classifier is a classifier that indicates that for a given object, there is a clear winner among the known object classes. However, if multiple different objects have similar activations, the object is recognized as belonging to the winning class. If yes (i.e., there is no clear winner), the system reports the object as unknown. Because we explicitly use labels in the training process, our "know-nothing" implementation allows us to It may only affect the mode and may not interfere with the learn mode.

[0041] An exemplary implementation of the "know nothing" concept using an ART network is presented in Figure 3. During input presentation, the categorical layer F2 216 generates activation patterns across its nodes. For inputs containing known objects, such as the top example in Figure 3, 300. For input that does not contain known objects, the bottom example in Figure 3 shows As shown in Figure 1, the activity in the F2 layer is likely to be more uniformly distributed. Since there may be nodes 302 with higher activity than the dotted line in Figure 3, we calculate the average of all activations, Using the threshold is insufficient to distinguish between these two cases. As in the example, only clear winners 300 remain above the threshold (dashed line in Figure 3). The threshold is increased by multiplying the average by a parameter of 1 or more.

[0042] The exact value of this parameter depends on several factors and is determined by the categories that the network has been trained on. It can be calculated automatically based on the number of categories and the total number of category nodes in the network. An example of the calculation is as follows:

number

[0043] By training the independent module B using the concept of "knowing nothing," the following results are obtained: Columbia Object Image Library10 50 objects from the 100 objects in the COIL-100 dataset were selected as the traces. The 50 novel objects were presented as "nothing" by independent module B. We selected all 100 objects from the COIL-100 dataset so that they would be recognized as “unknown.” , was used as the test set. During training, we simulated real-time behavior. The ART classifier in the independent module B is not shuffled at all. After training, the ART classifier achieved a 95.5% recognition rate (object and "non-object" recognition rate). For comparison, we have demonstrated the effectiveness of the COIL-100 dataset. If we feed the unshuffled dataset of all 100 objects into a conventional ART, The recognition rate was only 55%. This is due to the order dependency of ART, which is discussed below. Ugh.

[0044] If the input is not recognized by the ART classifier in module B, a correction is introduced to It is up to the user to label the correct input. If unrecognized input is not important , the user can ignore that input and the ART classifier will continue to identify it as "I don't know anything." If an object is important to the user, the user can label it, and the The object features and corresponding labels are added to the knowledge by the network of learning module B. Module B will be added to enhance the feature set associated with this new object. So you can activate a tracker system that keeps watching this object and adds more perspectives. Cut.

[0045] Example Implementation of Module A In operation, Module A extracts features and creates a compressed representation of the object. Neural networks are well suited to this task, as outlined below.

[0046] A convolutional neural network (CNN) is a DNN that uses convolutional units. The receptive field of the unit's filter (weight vector) is determined by the height and width dimensions of the input. When applied to visual input, the input to the early layers of the CNN is The length (h), width (w), and dimensions of one to three channels (c) (e.g., red, green, and and blue pixel components), while the input to the subsequent layers of the CNN is , height (h), width (w), and the number of filters from the previous layer (c). Because each filter is small, the number of parameters is significantly reduced compared to fully connected layers. , there is a unique weight projecting from each of (h,w,c) to each unit on the next layer. For the layer, each unit has a number of weights equal to (f,f,c), where f is either h or is the spatial filter size (usually 3), which is much smaller than any of the w filters. By applying this at different spatial positions in the input, the object can be classified when it is at one spatial position. If the feature containing the object is independent of the object's spatial location, the object will be It offers the desirable property of being translationally invariant in that it can be classified at any position.

[0047] Convolutional layers are usually followed by subsampling (downsampling) layers. by reducing a small spatial window (e.g., 2x2) of the input to a single value. , reduce the height (h) and width (w) of the input. This reduction involves averaging (average pooling). Or maximum value (max pooling) is used. The response of the subsampling layer is , which is invariant to small changes in the image, and this effect accumulates over multiple layers of a typical CNN. Inference is achieved by applying several convolutional and subsampling layers to the image. The output responds well to various transformations of the input, including translation, rotation, scaling, and warping. It exhibits a remarkable stability, e.g., continuous handwritten digits (written without lifting the pen) The network trained on the digit "3" in the training set and the small It responds similarly to the number "3" written by connecting small circles.

[0048] These invariances allow the coding of inputs to be more robust to visual fluctuations. A feature space is provided, i.e., the input changes (e.g., the object changes in size in the image frame). When the object moves (slightly translating and rotating), the change in the output value is much smaller than the change in the input value. For example, two frames with an object that has been translated by a few pixels It is difficult to learn on top of another method that has little or no similarity to the coding of Possible.

[0049] Furthermore, recent advances in GPU-accelerated algorithms have been used to learn filters from large datasets. By using gradient descent techniques, CNNs can learn to predict well-trained object classes. , we can achieve excellent generalized performance. Generalization means that the performance of the trained class is For test images that are not identical to the training images, the network This means that it can produce output. Learning the key regularities that define classes requires a large amount of If the network is trained on many classes, In this case, the lower layer, in which the filter is shared among all classes, can provide a good regularity set for all natural inputs. Therefore, a DNN trained on one task can be used to train a Used as task initialization or to pre-empt lower layers to a new high-level representation. When used as a processor, it can provide excellent results. Natural images share common statistical properties. Share a common set. Visualize the inside of a well-trained neural network. As recent research shows, the features learned in the lower layers are largely class-independent. On the other hand, the higher the layer, the more class-dependent it becomes.

[0050] In L-DNN, module B generates high-quality compressed and generalized object features for classification. We take advantage of these CNN capabilities in Module A to obtain a representation of the features. To maximize the number of distinct objects, the DNN used in L-DNN is Therefore, the object specificity of the high-level feature layer can be Does not interfere with the DNN's ability to learn quickly.

[0051] Example implementation of Module B In operation, Module B learns new objects quickly and without catastrophic forgetting.

[0052] Adaptive Resonance Theory (ART) An exemplary implementation of Module B is the ART neural network. In ART, By utilizing competition between category nodes to determine the winner node for each object presentation, This avoids fatal forgetting. The winning node is the one associated with the correct label of the object. If and only if a node is found to be a sparse node, the learning algorithm updates the weights for that node. Each node is associated with only one object, and the learning algorithm prioritizes only the winning node. Because the learning process updates the learning time, any learning episode in ART affects only one object. Thus, when a new object is added to the system, there is no interference from previous knowledge. Rather, ART only creates new category nodes and updates the corresponding weights. .

[0053] Unfortunately, the ARTs described in the literature do not have a successful implementation as module B of L-DNN. There are several disadvantages that prevent it from being used in reverse. The lack of a concept of "knowing nothing" is not unique to ART and is discussed above. A list of ART-specific problems and solutions to these problems is disclosed below. do.

[0054] Classical fuzzy ARTs, due to the complement coding that is an integral part of their design, Does not handle sparse inputs well. When sparse inputs are complement coded, the sparse The complement part is high for most of the components because the complement of 0 is 1, which is abundant in the input. Since all of these 1s are in the complement part of the input, they are different during distance calculation. The system becomes confused because it becomes very difficult to separate the inputs from each other. Powerful feature extractors like NNs provide mostly sparse signals when extracting high-level features. It tends to retain the ART paradigm but incorporates classical fuzzy design and complementary coding. Moving away from the dummy is useful for using ART in module B of L-DNN. One solution is to remove the complement coding and use the fuzzy ART. The idea is to replace the fuzzy AND distance metric used in the previous method with a metric based on the dot product. For dot product based metrics, the results remain normalized, and other methods for fuzzy ART The advantage is that no modifications are required.

[0055] The ART family of neural networks is highly sensitive to the order in which the inputs are presented. In other words, ART lacks the consistency property, and different input orders can lead to different pairs of inputs in the ART network. Unfortunately, real-time operations like L-DNN require different representations of the corresponding objects. The rating system consumes training data as it receives it from the sensors. Therefore, the training data cannot be shuffled to provide consistency. The sensor frequently samples most or all of the first object during real-time operation. It then provides all subsequent object samples, allowing the system to learn object representations one at a time. This allows the system to learn without competing with other objects, and therefore the object representation This leads to a situation where only some nodes present the primary object. On the other hand, subsequent objects may be represented by a representation of the first object that the system already largely occupies. It can be exaggerated because it will force the expression into the hyperspace in which it is expressed. The "no idea" mechanism described introduces competition early on and The aggregation described above results in an over-representation of the subsequent object, which Reduce or eliminate it.

[0056] Aggregation also reduces the memory footprint of object representations, making it suitable for memory-limited tasks. This is particularly useful for edge devices, as the system can identify all objects that cannot be classified in other ways. For every viewpoint, creating a new category node means that new objects are tracked as inputs. When more memory is added, the memory footprint of the ART system increases at a constant rate. During the real-time operation and subsequent presentation of the object described above, the system , and for each subsequent object, we create a superlinearly increasing number of nodes. ,The system experiences an exponential growth in the number of nodes with the number of objects.,Therefore, Using conventional ART, the memory footprint of Module B is linear with the number of objects. In the worst case, this growth can be exponential. The memory growth is controlled to a rate that is linear with the number of objects, and each object that the L-DNN learns is It becomes possible to create a fixed-size, near-optimal representation of the field.

[0057] Example of a complete L-DNN implementation

[0058] L-DNN classifier Figure 4 shows the overall performance using the modified VGG-16 DNN as the core of module A. An example L-DNN implementation for image classification using softmax and fully connected The last two layers are removed from the original VGG-16 DNN, and the ART-based module B is , which is connected to the fully connected first layer of the VGG-16 DNN. A similar but simpler L -DNNs can be created using Alexnet instead of VGG-16. This is a very simple and computationally inexpensive system that can be run on any modern smartphone. It works without the need for a GPU or any other dedicated processor, and is compatible with smartphone cameras. Any set of objects can be learned from a few input frames provided by

[0059] L-DNN grid-based detector One way to detect objects of interest in an image is to divide the image into grids and This implementation of L-DNN takes advantage of the following features of CNN: It is useful for

[0060] In addition to the vertical hierarchical organization across layers described above, each layer also contains data that maintains a topographical organization. This is done by specifying the network or kernel, stride, or pad size. Features corresponding to specific areas of interest in the image, regardless of depth, This means that similar areas of a layer can be found on all layers at various resolutions. For example, When an object is in the upper left corner of the image, the corresponding feature is found in the upper left corner of each layer along the layer hierarchy. Therefore, by attaching module B to each of the layer locations, Module B performs classification on a specific location in the image and determines whether any known object is in this location. It allows to determine whether a

[0061] Furthermore, the same feature vector represents the same object regardless of its location in the image, For each DNN layer (or scale) used as input, create only one module B. Therefore, by studying one object in the upper right corner, This allows module B to recognize the object anywhere in the image. By using multiple DNN layers of different sizes (scales) as input to module B, This allows detection at multiple scales, which in turn allows for finer detection in the next process. It can be used to fine-tune the position of an object in an image without processing the entire image at a large scale. Can be used.

[0062] In this process, Module A selects the coarsest scale (e.g., publicly available) for classification. Module B provides a 7x7 image of possible ExtractionNets. If Rule B indicates that the object is found in the second cell from the left and the fourth cell from the top, The corresponding finer DNN input (e.g., 14x14 for the same ExtractionNet) Only a small portion of the image should be analyzed to further refine the object location.

[0063] Another application of multiscale detection is to use DNN designs where the layer sizes are not multiples of each other. For example, if a DNN has 30x30 layers, a 2x2 (compression factor of 15) , 3 × 3 (compression factor 10), and 5 × 5 (compression factor 6) layers. As shown in Figure 1, when we attach module B to each of these compressed DNNs, the object size is increased. Rough locations (shown as 502, 504, and 506) are obtained. However, these modules When the outputs of modules B are combined (shown as 508), the spatial resolution is The result is an uneven 8x8 grid with higher resolution at the edges and lower resolution towards the edges.

[0064] To achieve this resolution, the system must reduce the calculations for module B to (2 × 2) + (3 × 3 ) + (5 × 5) = 38 times, whereas to calculate a uniform 8 × 8 grid, it takes 64 times. Note that the calculation of Joule B is performed. In addition to being calculated with fewer calculations, ,The resolution of the multi-scale grid in Figure 5 for the central 36 locations is a uniform 8× Equal to or finer than the 8-grid resolution. Through detection, the system achieves better performance using only 60% of the computational resources of an equivalent uniform grid. This performance difference is due to the sum squared (uniform gradient) The sum of squares (representing the number of calculations for a non-uniform grid) is It grows faster than the stratum (which is the thickness of the stratum), so it increases for larger layers.

[0065] Heterogeneous (multi-scale) detection detects that an object in the center of the field of view is likely to be in the robot's path. are more likely to be detected accurately than surrounding objects that present no signs of a dangerous collision. This can be particularly beneficial for moving robots, as they benefit from

[0066] L-DNN for Image Segmentation For images, object detection typically involves placing a bounding box around the object and detecting the associated class ( For example, the task of labeling a group of objects (e.g., "dog") is defined as the grid in the previous section. In addition to methods based on object detection, object detection techniques typically identify one or more regions of an image with a bounding box. Select a box and then classify the features within that box as a specific class while simultaneously filtering the boundaries. This object detection method is implemented by regressing the offset of the box location. The algorithms used to implement the method are Region-based CNN (R-CNN), Fa Includes st R-CNN and Faster R-CNN, but without localization. Both methods rely directly on classification information, which may be substituted as a detection module. .

[0067] Image segmentation involves performing a segmentation analysis on all or a subset of the pixels in an image. The task of segmentation is to determine the class label. Semantic segmentation, which disambiguates individual pixels from objects, ,individual pixels from two distinct objects of the same class can be uniquely identified or instantiated. It can be divided into image segmentation and instance segmentation. The algorithms are typically based on object detection methods (such as Fast R-CNN, Fast R-CNN, or Faster R - Take the bounding box output of a neural network (CNN, etc.) and segment the most prominent object in that box. The class labels associated with the bounding boxes are After that, the class labels that can be attributed to the bounding boxes are associated with the segmented objects. If there is no segmentation result, the segmentation result is discarded. The object may or may not have instance information. One algorithm that implements this method is Mask R-CNN.

[0068] for image detection or segmentation, based on the R-CNN family of networks The L-DNN design is presented in Figure 6. We use a static classification module, such as Mask R-CNN. Consider the image segmentation process you will use. In this scenario, we will use static classification. The module 600 may be replaced with the module B 104 of the L-DNN. The network segmentation pathway remains unchanged and the area continues as normal. As with the static classification module, L- Module B104 of the DNN does not return any positive class predictions that pass the threshold (e.g. For example, areas where the network is not trained or segmented can be When a person recognizes that they "know nothing," as described above, the segmentation Similarly, the L-DNN module B104 selects the acceptable classes. When returning predictions, the segmentation results are preserved, just like in the static classification module. Unlike the static classification module 600, the L-DNN module B104 Provide continuous adaptation so that the state changes from the former to the latter through user feedback. .

[0069] User feedback is based on the user's selection of an object on their social media profile. directly through bounding boxes and class labels, or through user The user selects an object in the video, and then the object is tracked through the video, and new objects are Through indirect feedback, such as when providing continuous feedback on body classes. This feedback may be used to classify new classes over time. This process is used to train the L-DNN to It does not affect the segmentation components of the network.

[0070] There is also some flexibility in the placement of module B 104 in this paradigm. The input to module B104 is the class labels combined with the segmentation output. The modules may be combined to produce a segmented and labeled output 602. The output of the convolutional layer 202 of module A should be directly linked to the output of the convolutional layer 202 of module A. This constraint applies to both modules. This can be satisfied by having modules A and B take the output of the region proposal stage. A should not depend on any dynamic parts of module B. Module B adapts to the network weights, but module A is static, so module If B should change the weights and then pass its output to module A, module A should Most static neural networks cannot cope with sudden changes in the network's input representation. will likely see a degradation in performance because

[0071] Brain aggregation and brain blending Multiple real-time machines implementing L-DNN can generate new In some cases, the following information can be learned individually on the fly. As outlined in several use cases, knowledge sharing between machines operating in real time is It can be advantageous for machines to learn new knowledge at the edge in real time. So, to share new knowledge, each machine operates in real time, from the edge to the center. Compressed and generalized transfer of new information to a server or other real-time machine. representation (represented in the network in terms of the synaptic weight matrix of module B) By performing the following steps, Extract the knowledge obtained by the analysis, either at a central server or directly on the edge device, In addition, other real-time operations can be performed by aggregating and centralizing communication or peer-to-peer communication. It can be shared with other machines.

[0072] Learn new information in the field where it is deployed – as discussed above, it operates in real time. The machine can learn new information on the fly through L-DNN. Real-time machines are aware that they are encountering new objects and / or new knowledge. and machines operating in real time can create new objects and / or new knowledge on the fly. To allow the system to learn quickly, labels can be provided for new objects, triggering a fast learning mode. In this way, machines operating in real time can modify their behavior and rapidly adapt to new objects and and / or be able to adapt to new knowledge.

[0073] New knowledge aggregation – After learning one or more objects on the fly, the system The fast learning module B runs the aggregation process. This process generates a representation of the new object. It compresses the representation of the object and integrates it with previously known object representations, improving the network's generalization ability. This reduces the memory footprint of module B. The practical implementation is detailed below.

[0074] Transferring the collective intelligence to other devices at any time during operation, or After completing the mission, the machine, operating in real time, will learn from its fast learning module. The aggregated weight matrix of each node (node ​​B) is transmitted to a central server (e.g., In some cases, the data can be transmitted to a cloud-based server (e.g., a cloud-based server). The weight matrix of the machine's high-speed learning module can be downloaded to an external storage device and stored in a central location. Physically linked to a central server when a central server is not available or desirable Communication can occur in a peer-to-peer fashion between machines (edge ​​devices) operating in real time. .

[0075] Brain blending (or fusion, merging, combination) - some real-time After the weight matrices from the machines are collected by a central server or one of the edge devices, The central server or edge device receives newly acquired data from each real-time operating machine. Combine the knowledge collected, compress it into a single weight matrix, and run a composite utility that aggregates it. The mixed utility allows you to optimize results while maintaining overall system accuracy. This reduces the memory footprint of the resulting matrix and removes redundancies. An exemplary implementation based on the IEEE 802.11 standard is detailed below.

[0076] Individual brain updates after blending - the resulting weight rows produced during the blending The string is then transmitted through a wired or wireless communication channel or to a physical external storage / memory. Download to device and physically transfer storage / memory device to machine operating in real time downloaded to one or more real-time machines by electronically transporting the data. do.

[0077] In this way, knowledge from multiple machines operating in real time can be aggregated, The new knowledge learned by each of these machines is shared with other machines operating in real time. It is possible.

[0078] Example implementation of the brain aggregation and blending process using ART FIG. 7A illustrates an exemplary brain aggregation and blending process using ART. Extend the default ART process as follows: Each node represents an object, and each object has one or more category nodes that represent it. The left side of Figure 7A shows the categories in layer F2 at 216 that are activated by layer F1 at 212. The weight patterns 702 for the nodes are shown. Each weight pattern is provided to 212 of layer F1. 2 represents a generalized input pattern learned from multiple real feature inputs 210. Learning in ART is based on the fact that category nodes and corresponding objects win the competition and solve the problem. Note that this only occurs if you correctly identify the object.

[0079] In the center of FIG. 7A, multiple inputs 210 for different objects are presented to the ART network. 7 shows the weight patterns 702 of the different category nodes in layer F2 216 after the weighting process. Each weight pattern 702 is set so that the corresponding node is considered to be the same as the corresponding object label. The weight pattern 702 in the center of Figure 7A represents a generalized version of the input learned as , resulting in the aggregate input 704 shown on the right of FIG. 7A.

[0080] The original input 210 is generally not available to the system at the time of aggregation or mixing. The collection of weight patterns 702 represents the inputs that the ART network encountered during training. 210. In this way, the weight pattern 702 is a generalization of all the important The features also represent important features that are better than the original input 210 during the training process. , which can act as a substitute for real-world input.

[0081] The aggregation uses weight patterns as a proxy for the real input. During aggregation, the next step A pop occurs.

[0082] The weight vector 702 of the existing weight matrix (for example, the matrix of the weight vector 214 in FIG. 2) , aggregated input set (right side of Figure 7A) a i =w i where a is the input vector, w is a weight vector, and i ranges from 1 to the number of existing category nodes in the network. If the ART network uses complement coding, half of the weight vectors Decomplement the min and average it over the first half of the vector (a i =(w i +(1-w ic )) / 2). Each vector in the aggregated input set is Receives the corresponding labels extracted from the category nodes.

[0083] All existing F2 nodes and corresponding weights are removed from the ART network, and Therefore, the ART network has a blank initial state.

[0084] The aggregate input set is randomly shuffled and the ART network learns the original inputs. By randomly shuffling, we train this set in the same way as we did in the previous This reduces the influence of order dependency in the ART network, making the ART network more Compact (fewer category nodes created) and more optimal (better generalization) ) representations can be constructed.

[0085] Using weights on an aggregated input set involves making a single vector appear as many original input vectors as possible. has the added benefit of replacing the and the computation time is faster than the original learning process.

[0086] The aggregation process can occur at any time during the operation of an L-DNN-based system. By using the ART-based implementation of module B, we can reduce the memory footprint of module A. This reduces the order dependency of RT-based systems. There is no way to change the order of sensory inputs entering the system as they progress, so The reduction in dependency will be beneficial for any real-time machine based on L-DNN. Aggregation is initiated by user action or when the memory footprint becomes too large. automatically upon reaching or exceeding a threshold size, or during an operational period. It can be triggered periodically based on time.

[0087] The example aggregation works in real time, as if you were looking at objects one after the other, and the ART This was done on the COIL dataset, which was intentionally presented to the network. Training took 4.5 times longer than aggregate training. Aggregation reduces memory usage. 25% reduction in footprint and improved object recognition performance from 50% accuracy to 75% accuracy The training dataset was first shuffled to reduce ordering artifacts. In the reduced case, aggregation still showed improved performance. There was no significant reduction in memory footprint, as the system was already well compressed. The average accuracy rate for object recognition increased from 87% to 98%. This represents an unexpectedly large performance improvement.

[0088] Fusion is the process of combining the weight matrices of two or more ART networks to form an aggregate training set. Fusion is an extension of aggregation, which is combined with ART network. As a result, multiple fused ART networks can be created. ,When they have knowledge of the same object, similar representations of this object across multiple ART,networks. All are naturally combined together by the ART learning process, while all distinctive This allows for intelligent compression of object representations and the memory of the fused system. This will lead to a further reduction in footprint.

[0089] For example, one ART instance trains 50 objects from the COIL dataset. Then, in another ART instance, 33 objects (17 objects are the same in the two sets) are learned. When the first instance is correct, 92.9% of the time, and the second instance is correct. By fusing them together, both ART instances It produces a network that is 97% accurate across all 66 unique objects trained. In addition, the fusion version will have a brute force combination of the two networks. It has 83% of the memory footprint of the fused version. The toplining is a combination of the first network and only the new objects in the second network. (excluding the 17 overlapping objects). Intelligently compresses and refines object representations to improve accuracy. Randomly shuffles inputs. Without it, the blending results were even more impressive in terms of accuracy, 85.3% and 77.6%. Mixing the correct networks reduces the memory footprint of the combined two networks. The result of these mixed experiments is a network with 84.6% of the inputs and a 96.6% accuracy rate. The results show an unexpectedly large performance improvement.

[0090] Using context information to improve performance L-DNN-based systems also combine context information with current object information. By using context L-DNN, it is possible to improve the performance accuracy. It can learn things that are likely to co-occur in a stream, e.g., camels, palm trees, and sand dunes. , and off-road vehicles are typical objects in desert scenes (see Figure 7B), while houses A sports car, an oak tree, and a dog are context-typical objects in a suburban scene. The locally ambiguous pixel-level information obtained as drone input is Depending on the text, it can be mapped to two object classes (e.g., camel or dog). In both cases, the focus of the object is fuzzy, as is typical of low-resolution images. So, the pixelated image of a camel is the probability that "camel" can be inferred by L-DNN. The fourth highest class, and most likely based on local pixel information alone, is "horse." However, it is based on global information about the scene and past associations learned between objects. The objects in the context (sand dunes, off-road vehicles, palm trees) are Therefore, the context classifier recognizes "horse" as a " class and select the "Camel" class. In an urban scene containing a "car" and an "oak tree," the same set of pixels is used to create a "dog" can be mapped to

[0091] As a side note, when an object is identified as ambiguous or abnormal, like the camel in the example above, L- The DNN system may prompt the human analyst / user to look more closely at the object. This anomaly detection and alerting subsystem uses context to distinguish between normal and abnormal objects. By resolving the ambiguity of the identification of the objects of interest, which do not belong to the scene, You can take advantage of this opportunity.

[0092] The infinite regress problem, i.e., before the context module can produce object classes, The need for object classification is to select the label with the highest probability as the input to the context classifier. This way, at each fixation of the object, the context classifier The guesses for object labels can be iteratively refined.

[0093] L-DNN can leverage huge amounts of unlabeled data The sheer volume of unstructured content means that even without labels, valuable training data is The data is provided to module A of L-DNN. Greedy layer-wise pre-training (gre through a technique known as layer-wise pre-training. By training each layer in turn, the DNN can achieve unsupervised learning from the bottom up. The layer-by-layer training mechanism allows for contrastive learning. Convergence, i.e., denoising autoencoders and convolutional autoencoders An autoencoder takes an input and encodes it using weights and transfer functions. After training a layer, its output is evaluated in terms of the input reconstruction error. The pre-trained network is the input of any deep network. It also benefits from the learning of edges on layer 1, corners on layer 2, and and other edge groups, as well as features specific to higher-order data in later layers. Furthermore, the convolutional variants often capture the hierarchical feature relationships of convolutional networks. It enjoys the inherent translational invariance of the

[0094] This process tends to precede later supervised learning ("fine-tuning"), so In many cases, the performance of a pre-trained network is affected by the Outperforms networks without pre-training, especially when there is a large amount of labeled data. ,Since the labels impose some burden on the analyst, the network to be pre-trained However, it cannot compete with networks without pre-training. The "Area" network achieves the recognition performance of the L-DNN system while keeping the labeling burden low. This will improve the performance over other pre-trained networks. In addition to the limited labels resulting from analyst reports, the data is also traceable to unlabeled data. The DNNs being trained are based not only on the reports of a relatively small number of analysts, but also on other, massively labeled This leads to improved performance over DNNs trained from a similar dataset.

[0095] Finally, ART, as an implementation of module B, can also perform unsupervised learning. ART has another advantage: it does not require labels for training, but can use them when they are available. can be considered "semi-supervised," meaning that it can utilize labels. For a node, storing search information for the frame and image region of the best matching observation. It then operates in an unsupervised learning mode to help organize unlabeled data. Each ART node allows the analyst to access and explore many similar observations. It may be possible.

[0096] Example Use Cases of L-DNN The following use cases demonstrate how L-DNN can address technical problems in various fields: are some non-limiting examples.

[0097] Automate the investigation using L-DNN: single or multiple image sources For example, the inspection process for industrial infrastructure such as power lines, base station towers, or wind turbines. Consider a drone service provider who wants to automate their operations. So, the investigators will use the drone to find the frame containing the key components that need to be examined. It takes hours of watching video. Investigators need to identify these key components in each frame. must be identified manually.

[0098] In contrast, L-DNN-based assistants can be deployed in identification tools. Data containing body or anomaly labels is pre-trained during traditional slow DNN factory training. This can be provided as a pre-trained set to the L-DNN-based assistant. The user can add to this set during the fast learn mode, as described below.

[0099] Figure 8 shows the results of a "smart" drone versus a "dumb" drone. L- may be included on a computer used to review the captured video. The DNN-based assistant in action. The drone 800 is connected to a communication tower 820, a solar panel 830, and a A structure such as a panel 830, a wind turbine farm 840, or a power distribution line 850 (which (These are only exemplary structures, and other structures may be envisioned.) 810 may use manual control of the drone or may operate the drone automatically. A human analyst 805, such as an analyst in a control room, may monitor the drone 800. The L-DNN system has sensory inputs (e.g., video, lidar, etc.) from Module B104 of module B106 provides labels while the drone is flying, or Flights can be posted.

[0100] First, drone 800 receives a copy of L-DNN 106 as a separate local classifier. The drone 800 detects these power lines 850, base station towers 820, and wind turbines. While investigating the L-DNN 840, we acquire video frame 100. Module A102 generates a video frame 10 based on the pre-trained data. Module B104 then extracts image features from the image based on these features. It provides a likely label for each object. This information is passed to the user 805. 05: If the label is not satisfactory, the fast learning mode is activated to improve the network of module B. This way, the information provided by the user can be used to update the network with the correct labels. Therefore, the fast learning subsystem can correct the label of the first frame after the update. It can learn to recognize objects it has already learned, such as power lines, cell towers, and wind turbines, as quickly as the One-trial learning can be used to determine the location and characteristics of a previously captured video. When analyzing, it means immediately after the user introduces the correction. 6 becomes more knowledgeable over time and provides better discrimination over time with the user's help. do.

[0101] FIG. 9 illustrates the L-DNN techniques described herein applied to multiple drones (e.g., data processing) The non-capable drones 800 and 900, as well as the smart drone 910, can be synchronized How can this be applied as a generalized example of Figure 8, where data is collected either sequentially or asynchronously? The information learned by the L-DNN associated with each drone is merged. They are then pushed back (combined or mixed) to other drones or drawn. It can be shared peer-to-peer between the users or with a central server containing the module B104. The central server merges the information learned by each L-DNN and sends it to the drones. The merged information is pushed back to all drones, including drone 910, and drone 910 The data obtained by the satellites 800 and 900 includes the communication tower 820, the solar panel For more information on wind turbine farms 830, wind turbine farms 840, or power distribution lines 850, However, thanks to the merge process, these items can now be understood and categorized.

[0102] Automating Warehouse Operations Using L-DNN: Aggregating and Blending Knowledge from Multiple Sources The system described above is a system of multiple machines or cameras (fixed or can be extended to multiple different geographic locations (such as a drone, a drone mounted on a mobile device, etc.) Consider a company with a large warehouse. Manually taking inventory of a large warehouse can lead to many This can take man-hours and often requires the warehouse to be closed during that time. The solution makes it difficult to identify stacked objects that may be hidden. Additionally, with existing automated solutions, information learned in one geographic location is not shared with other In some cases, vast amounts of data are collected in different geographic locations. Therefore, these automated solutions have the ability to learn and act on new data. It can take weeks to do so.

[0103] In contrast, the L-DNN techniques described herein rely on sensors (e.g., fixed cameras 101 0a-1010c, or a mobile camera mounted on a robot or drone) New items in inventory are updated on-the-fly via various L-DNN modules that connect to sensors. The learning environment shown in Figure 10 can be used in a warehouse, industrial facility, or distribution center. In addition, operators at 805 and 1005 can New information can be taught to various L-DNN modules. This new knowledge is then centrally integrated. After mixing, the data is either sent to each individual device (e.g. , camera 1010).

[0104] For example, fixed cameras 1010a to 1010c (collectively referred to as cameras 1010) in FIG. Each of these cameras 1010 captures a corresponding view of an object on the conveyor belt. A video image 100 is acquired, and the image 100 is input to the corresponding L-DNNs 106a to 106c (or The L-DNN 106 performs, for example, investigation, classification, or recognizing known objects in the image 100 for other distribution center functions.

[0105] Each L-DNN 106 is then run through the network for evaluation by human operators 805 and 1005. Or tag the unknown object as "nothing known." For example, unknown object 104 When presented with 0, the L-DNN 106a performs classification by a human operator 805. In order to do so, the L-DNN 106c flags the unknown object 1040. Similarly, the L-DNN 106c Flag unknown objects 1060 for classification by operator 805. L-DN N106b indicates that when an unknown object 1050 is presented, the subject simply sees the unknown object 1050 as “nothing.” The independent module B104d connected to the L-DNN106 is , by the L-DNNs 106a and 106c from the human operators 805 and 1005 The acquired knowledge is merged and module B 104b calculates the current state of objects 1040 and 1060. To recognize later instances, the module B 104b of the L-DNN 106b is informed Push awareness.

[0106] The L-DNN106 for each device can be equipped with features such as pillar markings, exit signs, or combinations thereof. and / or similar objects in the warehouse. This allows the system to be trained to use sensors (e.g. For example, the position of the unmanned vehicle appearing in the images acquired by the camera 1010 is triangulated. The L-DNN in each vehicle operates in exactly the same way as the use cases described above. In this way, knowledge from multiple unmanned vehicles is aggregated, blended, and returned to each unmanned vehicle. The aggregation and mixing of knowledge from all places can be achieved through the aggregation and mixing center above. This can be done by a central server, as described in the previous section, and also by peer-to-peer A mix of these can also be applied. This allows for multiple warehouses to be You can take inventory and aggregate knowledge.

[0107] Using L-DNN on a fleet of mobile devices Consumers' mobile devices, such as their smartphones and tablets, or Mobile cameras, body-worn cameras, and public safety first responders and public safety officers A distributed network of specialized devices, such as handheld LTE devices, used by Consumers' devices are used to understand their surroundings, such as when taking a photo. In these cases, the L-DNN technique described herein can be used to Applicable to smartphone or tablet devices 1110, 1120, and 1130 Individuals (e.g., users 1105 and 1106) can access devices 1110 and 113. Each L-DNN module 106 is taught knowledge, and this information is shared between peers or The merge may be performed on a server 1190 containing module B 104. ,The merged knowledge may be applied to devices that did not participate in the original training 1020. Push back to some or all connected devices, including

[0108] The L-DNN module can, for example, apply image processing methods to photos taken by the user. The user can train each L-DNN to learn some customizations associated with aspects of the photo. Applying a filter or image distortion to these object classes or areas The combined learned actions can be taught across devices. They may be shared, merged, or combined peer-to-peer or collectively. -DNN technology allows input variables to be sensory or non-sensory (all of the smartphone The input variables and usage patterns can be applied to generalized smartphone usage cases. These usage patterns can be any combination of input and output variables. The L-DNN modules 104 are trained at the individual level and then pushed to the central L-DNN module 104, where they are merged and Push back to individual devices.

[0109] In another example, police officers could use specialized devices powered by L-DNN to identify lost, suspected, or In such a situation, the police and / or first responders may Responders cannot afford to waste time. Provided to police officers and / or first responders Existing solutions require manual analysis and organization of video feeds from cameras. Such solutions use a central server to analyze and identify objects. This solution requires too much time to process the video data. Since the analysis needs to be done on a cloud / central server, there is a big latency issue. This poses a significant obstacle for first responders / officers who often need to act immediately upon receiving the data. In addition, continuous transmission of video data to a central server can reduce the communication channel. This can be a burden.

[0110] Instead, we use L-DNNs in mobile phones, body-worn cameras, and handheld LTE devices. By using edge computing, data can be learned and analyzed at the edge itself. can learn to customize their devices, allowing officers / first responders to locate people / objects Find and provide, as well as people / things of interest that the officer may not be actively looking at. It can also locate and identify bodies in various locations. L-DNN learns from operators on a remote server. Instead, it utilizes a fast learning mode to learn from officers in the field on the device. It reduces or eliminates latency issues associated with centralized learning.

[0111] FIG. 1 shows how a consumer can point their phone at a scene and select the entire scene or a portion of the scene (an object, e.g., When labeling a scene (such as sky, water, or other parts of a scene), we label the components in the image. In addition, police officers can use video flash to identify the person in question. The mobile phone can access the video frame 10 to identify suspicious people / objects. 0, module A102 will, based on the pre-trained data, Image features can then be extracted from these frames. The features can be used to provide a likely label for each object, e.g., Person A is in Neighborhood B. Person A is labeled as a "resident" of neighborhood B if they live in neighborhood B and have been observed in neighborhood B in the past. Therefore, the fast learning subsystem may start as early as after the first frame of learning. Immediately, the robot is asked to judge the relative positions and characteristics of objects it has already learned, such as houses and trees. More importantly, the dispatcher on the central server 110 can ,New objects to be found can be introduced into the server-side L-DNN, and the new objects, They will be mixed as needed and distributed to local first responders.

[0112] This use case is very similar to the previous one, but without forgetting the old object. It takes better advantage of L-DNN's ability to quickly learn new objects. Concentration is often under little time pressure and allows memory to be aggregated in a slow learning mode, whereas the In the case of a single responder, all devices in the area may begin searching for a suspect or missing child. It is important to aggregate and blend knowledge from multiple devices as quickly as possible so that Therefore, it is important to quickly learn new objects guided by a single first responder. L-DNN, which aggregates the data almost instantly on a server and distributes it to all first responders in the area. This ability is a huge advantage for this use case.

[0113] Replacing traditional DNNs with L-DNNs in data centers The L-DNN technique described herein can be implemented on individual computing nodes or Reducing the computation time of the DNN process on a large data center server 1200 L-DNN technology can be applied as a tool to accelerate learning in DNNs by orders of magnitude. This feature can be used to dramatically reduce the need for computing resources on the server, or It reduces the consumption of computing resources and information is often acquired without requiring hours / days / weeks of training time. It can be trained on the required large datasets in just a few seconds. The use of L-DNN also reduces power consumption and frees up data center server resources. It also leads to better utilization overall.

[0114] conclusion As described above, L-DNN allows for on-the-fly (one-shot) learning. Conversely, traditional DNNs often It takes thousands or even millions of iteration cycles to learn a The larger the step size per loop, the more the gradient of the loss function changes, which actually improves performance. Therefore, these conventional DNNs require a large number of samples per training sample. introduces small changes to the weights, which allows us to add new knowledge on the fly. In contrast, L-DNN, which involves fast learning neural networks, can learn stable object representations with few training examples. ,Even a single training example can be sufficient for L-DNN.

[0115] L-DNN uses fast training neural networks in addition to traditional DNNs. Because it uses a finite element, it is resistant to the "catastrophic forgetting" that plagues conventional DNNs. When a new input is provided to the DNN, all weights of the DNN are adjusted for each sample presented. Learning new inputs causes the DNN to "forget" how to classify old inputs. can be avoided by simply retraining the full set of inputs, including the new input. However, retraining is too time-consuming to be practical. Some existing approaches Selectively limit weights based on their importance or train sub-networks of DNNs or use a modular approach to avoid fatal forgetting? However, such an approach is not only slow, but also It also requires multiple iterative cycles to train the NN. In contrast, L-DNN This provides a means to achieve fast and stable learning capabilities without retraining. L-DNN also allows for the generation of object representations with a single example and / or in a single iteration cycle. It also promotes ongoing, stable learning.

[0116] Although various inventive embodiments have been described and illustrated herein, those skilled in the art will readily appreciate that the present disclosure various methods for performing the functions and / or obtaining the results and one or more advantages described in the document. Various other means and / or structures are readily envisioned, and each such variation and / or modification is incorporated herein by reference. are considered to be within the scope of the inventive embodiments described herein. More generally It is understood by those skilled in the art that all parameters, dimensions, materials, and configurations described herein are illustrative. The actual parameters, dimensions, materials, and / or configurations are intended to be illustrative and not restrictive and may not be used in any way in accordance with the teachings of the present invention. Those skilled in the art will readily understand that the specific application or applications involved will depend on the particular application or applications. Many equivalents to the specific inventive embodiments described herein may be recognized or simply understood. These and other features can be confirmed using routine experimentation. Accordingly, the foregoing embodiments are to be considered as examples only. As set forth and within the scope of the appended claims and their equivalents, embodiments of the invention are It is to be understood that the invention may be practiced otherwise than as specifically described and claimed. Embodiments of the present invention relate to each individual feature, system, article, material, kit, and / or methods. In addition, two or more such features, systems, articles Any combination of such features, systems, materials, kits, and / or methods may be used. Where the articles, materials, kits, and / or methods are not mutually inconsistent, the inventions of this disclosure Included in the range.

[0117] The above-described embodiments can be implemented in any of numerous ways. For example, embodiments can include: It may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code is provided to a single computer. The present invention may be implemented in any suitable processor or processors, whether integrated into one computer or distributed among multiple computers. It may be executed on a collection of processors.

[0118] Furthermore, the computer may be a rack-mounted computer, a desktop computer, It can be used in a variety of forms, including laptops and tablets. It should be understood that a computer may be embodied in any of the following: A device that is not considered a computer, but a personal digital assistant (PDA) , including a smartphone, or any other suitable portable or fixed electronic device; It may be embedded in a device with suitable processing capabilities.

[0119] A computer may also have one or more input and output devices. The interface can be used, among other things, to present a user interface. Examples of output devices that can be used to provide an interface include a printer or visual a display screen for visual presentation, and a speaker or other device for audible presentation of the output Examples of input devices that can be used in a user interface include voice generation devices. This includes keyboards, mice, touchpads, and digitizer tablets. As another example, a computer may be able to recognize a user through voice recognition. The input information may be received in other audible formats.

[0120] Such computers can be part of a local area network or an enterprise network. Wide area networks such as the Internet, and Intelligent Networks (IN) or are interconnected by one or more networks of any suitable form, including the Internet Such a network may be based on any suitable technology and may include any It may operate according to any suitable protocol and may be used over a wireless network, a wired network, or may include a fiber optic network.

[0121] The various methods or processes outlined herein may be implemented on a variety of operating systems or or platform, and can run on one or more processors. In addition, such software may be coded as multi-functional software. A number of appropriate programming languages ​​and / or programming or scripting tools It may be written using either a framework or a virtual machine. The program may be compiled into executable machine code or intermediate code that is executed.

[0122] Also, various inventive concepts may be embodied in one or more methods, examples of which are provided. The acts performed as part of the method may be ordered in any suitable manner. Thus, embodiments may be constructed in which the actions occur in a different order than illustrated, and This means that some acts may be performed simultaneously even though they are shown as sequential acts in the exemplary embodiments. This may include:

[0123] All publications, patent applications, patents, and other references mentioned herein are incorporated by reference. It incorporates the whole of them.

[0124] All definitions and terms used herein are based on dictionary definitions, texts incorporated by reference, and the like. The definitions in this document and / or the ordinary meaning of the defined terms should be understood as governing the is.

[0125] As used in this specification and claims, the indefinite articles "a" and "an" are used interchangeably. Unless expressly indicated otherwise, "at least one" should be understood to mean "at least one." do.

[0126] As used in this specification and claims, the term "and / or" means means "either or both" of the elements connected, i.e., in some cases, "and" should be understood to mean elements that are present in other cases and disjunctively present in others. The elements listed with " / or" are of the same form, i.e., among the coordinated elements. The other elements should be interpreted as "one or more" of the elements specifically identified. specifically identified by an "and / or" clause, whether relevant or unrelated. Thus, as a non-limiting example, "A and / or References to "or B" when used in conjunction with open-ended phrases such as "to have" In one embodiment, only A (optionally including elements other than B), in another embodiment, In yet another embodiment, both A and B ( and optionally including other elements).

[0127] As used in this specification and claims, "or" means "or" as defined above. and / or" should be understood to have the same meaning as "and / or." For example, items in a list When separating terms, "or" or "and / or" is inclusive, i.e., multiple. or list of elements, and optionally additional items not in the list, "inclusive" means one, but shall be construed to include two or more unless expressly indicated to the contrary. Only the indicated term, e.g., "only one of" or "exactly one of", or "Consisting of," when used in the claims, means a number or list of elements. Generally, as used herein, "or" refers to the inclusion of exactly one element of a set of elements. The term "any," "one of," "only one of," or "the When preceded by a term of exclusivity, such as "exactly one of," it indicates exclusive alternatives (i.e. "to essentially "Consisting of," when used in the claims, shall have its ordinary meaning as used in the field of patent law. It shall have the following.

[0128] As used in the specification and claims, refers to a list of one or more elements. The phrase "at least one" refers to a selection of any one or more of the elements in a list of elements. means at least one element, but not all, specifically listed within the list of elements. Any element in a list of elements, without necessarily including at least one of every element. It should be understood that this definition does not exclude elemental combinations. The phrase "at least one" refers to any element other than those specifically identified in the list of elements. Anything that may be present, whether related or unrelated to the specifically identified element. Thus, a non-limiting example would be "at least one of A and B." (or, equivalently, "at least one of A or B" or, equivalently, "A and / or at least one of B), in one embodiment, B is absent. , including at least one, optionally two or more elements A (optionally including elements other than B) In another embodiment, A is absent and has at least one, and optionally two. In another embodiment, B includes at least one of the above (optionally including elements other than A). at least one, optionally including two or more A, and at least one, optionally including two or more can refer to two or more B (optionally including other elements), etc.

[0129] In the claims, as well as in the above specification, all transitional phrases, such as "comprises" "including," "carrying," ", "having," "containing," "accompanying" "convolving," "holding," "composed of" "ed of"), and the like, are understood to be without limitation, i.e., including but not limited to: It means that the definition is not given. and "consisting essentially of" The only transitional phrases that appear in the United States Patent Office Manual of Patent Examining Procedure, Section 2111.03 are: The two phrases are closed or semi-closed transitional phrases, respectively.

Claims

1. 1. A method for analyzing objects in an environment, comprising: collecting, by a sensor, a data stream representative of the objects in the environment; by a neural network running on a processor operatively coupled to said sensor , extracting a convolution output from the data stream, the convolution output being , representing characteristics of the object; and a classifier operatively coupled to the neural network, said convolution output and classifying the object based on

2. The method of claim 1 , wherein the sensor is an image sensor and the data stream includes an image. How to do it.

3. Extracting the set of features comprises: generating a plurality of segmented sub-areas of the first image; The neural network computes each of the plurality of segmented subareas. and coding the data.

4. Extracting the feature set comprises: Allowing the user to select portions of the data stream that interest them And, The user may divide the portion of interest into multiple segments. And, encoding each of the plurality of segments by the neural network. The method of claim 1 , comprising:

5. The sensor is at least one of a lidar, a radar, or an acoustic sensor. The data stream is selected from the group consisting of lidar data, radar data, and acoustic data. The method of claim 1 , comprising a corresponding one of:

6. A sensor that collects a data stream of an environment, the data stream comprising: a sensor representing an object within the boundary; at least one processor operatively coupled to the image sensor, A convolution output representative of the feature of the object is extracted from the data stream. and (ii) classifying the object based on the convolution output. and at least one processor that executes a classifier that

7. The sensor may be at least one of an image sensor, a lidar, a radar, or an acoustic sensor. The apparatus of claim 6 , further comprising:

8. The neural network includes a deep neural network (DNN).

7. The apparatus of claim 6.

9. 10. The method of claim 9, wherein the neural network comprises an adaptive resonance theory (ART) network.

6. The apparatus according to claim 6.

10. Lifelong learning deep neural networks (L-DN) are being used in real-time machines. N), comprising: The L-DNN (i) obtains information on the environment of the machine operating in real time from sensors. and (ii) determining the real time based on the pre-determined weights of the L-DNN. Predicting the first action on a machine operating in time; The L-DNN calculates a real-time machine operation based on the observation. determining discrepancies between the expectations and perceptions; In response to the discrepancy, a fast learning mode is triggered by the L-DNN. Thus, the fast learning mode can be implemented without changing the pre-determined weights of the L-DNN. and generating a forecast that is revised based on said observations.

11. determining that the real-time operating machine is offline; and In response to determining that the real-time operating machine is offline, a slow learn mode is started. and triggering the slow learn mode based on the observation. and modifying the pre-determined weights of N. Law.

12. It was a method to extract, aggregate, and share knowledge among multiple machines operating in real time. Each real-time operating machine among the plurality of real-time operating machines is Implement the neural network with each copy of the matrix, The first real-time operating machine among the plurality of real-time operating machines learning at least one new object in a fast learning subsystem; via a communication channel from the first real-time operating machine to a server, transmitting a representation of at least one new object; At the central server, the at least one forming an updated weight matrix based at least in part on said representation of the two new objects; And, from the server, at least one second machine among the plurality of real-time operating machines; transmitting a copy of the updated weight matrix to a real-time machine; A method comprising:

13. learning the new object operatively coupled to the rapid learning subsystem of the first real-time operating machine; acquiring an image of the at least one new object with an image sensor; The fast learning subsystem of the first real-time machine includes: and processing the image of another new object.

14. The neural network includes an adaptive resonance theory (ART) neural network. fruit, the representation of the at least one new object in the ART neural network; The method of claim 12 further comprising generating.

15. the representation of the at least one new object includes a weight vector; to reduce the memory footprint of the representation of the at least one new object. the copy of the weight matrix used by the first real-time machine; 15. The method of claim 14, further comprising: aggregating the weight vector.

16. Aggregating the representation of the new object with the representation of at least one previously known object. The method of claim 12 further comprising:

17. Transmitting the representation of the at least one new object comprises: The representation of the at least one new object is then fed to the plurality of real-time operating machines. transmitting the data to the server by a second real-time machine among the machines. The method of claim 12.

18. forming the updated weight matrix The representation of the at least one new object is then fed to the plurality of real-time operating machines. At least one machine from at least one other machine operating in real time 13. The method of claim 12, including blending with representations of other new objects.

19. Neural networks trained to recognize objects across multiple categories 1. A method for classifying objects in a computer, comprising: presenting an object to the neural network; determining a plurality of confidence levels by the neural network, Each confidence level in the plurality of confidence levels indicates whether the object falls into a corresponding category in the plurality of categories. It represents the possibility of falling under the category of "gori" and comparing the plurality of confidence levels to a threshold; determining, based on the comparison, that the object does not fall into any of the plurality of categories; and

20. conducting the comparison determining that no confidence level among the plurality of confidence levels exceeds the threshold; 20. The method of claim 19, comprising:

21. The method of claim 1 further comprising: setting the threshold value to be greater than an average of the plurality of confidence levels. Item 20. The method according to item 19.

Images