Neural network architecture for invariant object representation and classification using local Hebbian updates.

Abstract

The present disclosure relates to improved systems, methods, and techniques for constructing and employing neural network architectures to solve computer vision and other problems. The neural network architectures can have two or three layers, with every node in the first layer connected to every node in the second layer. The nodes in the second layer can be connected to each other. The weights or values ​​of the various connections between these nodes in the first two layers can also be updated during the processing of inputs to the neural network architecture. These neural network architectures can learn continuously without requiring extensive training. Other embodiments are also described herein.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of and priority to U.S. Provisional Patent Application No. 63 / 328,063, filed April 6, 2022, and U.S. Provisional Patent Application No. 63 / 480,675, filed January 19, 2023. The contents of the above-identified applications are hereby incorporated by reference in their entireties.

[0002] Government funding This invention was made with Government support under Award No. NIH R01 DC014701 awarded by the National Institutes of Health. The Government has certain rights in this invention.

[0003] The present disclosure relates to improved machine learning arrangements and techniques for immutable object representation and classification. In certain embodiments, the arrangements and techniques described herein can be implemented to enhance a variety of computer vision functions, including functions including, but not limited to, object detection, object classification, and / or instance segmentation. [Background technology]

[0004] Computer vision systems can be configured to perform a variety of functions, such as functions including object detection, object classification, and / or instance segmentation. These computer vision functions can be applied in many different contexts, such as facial recognition, medical image analysis, smart surveillance, and / or image analysis tasks.

[0005] Computer vision systems must consider various technical issues to accurately realize the above-mentioned computer vision functions. For example, one technical issue relates to accurately extracting features from input images. This is particularly challenging in situations where the input images contain objects (e.g., face objects) that are partially hidden or heavily occluded and / or degraded by noise, poor lighting, and / or non-uniform lighting. Other factors that may impede feature extraction may stem from variations in camera angle, motion, viewpoint, pose, and object appearance (e.g., variations in facial expression) between different images.

[0006] Another technical challenge is designing a computer vision system that can efficiently extract features from images. Many feature extraction mechanisms are computationally expensive and resource intensive. Moreover, they are often built on deep learning models that involve many complex processing steps, which require extensive training datasets to be accurately labeled to facilitate supervised training.

[0007] Frameworks for performing feature extraction also suffer from a variety of other shortcomings. For example, with respect to frameworks that use blind source separation techniques, these frameworks cannot consider the usefulness of features based on their relative abundance. A framework set to capture useful features does not need to know the exact frequency of occurrence of an object, but the relative abundance of features should be considered. However, blind source separation and other related techniques cannot do so.

[0008] Consider a scenario where a blind source separation technique uses a dictionary to represent features. Changing the input matrix to include multiple occurrences of the same input does not change the nature of the dictionary. Multiple occurrences lead to repeated representations with the same level of sparsity and reconstruction error. Thus, the dictionary and the representation remain similar to the one obtained while considering each input only once. In other words, there are no constraints on the dictionary that force it to change according to the relative occurrences of the inputs. Thus, blind source separation approaches cannot exploit the statistical properties of the environment to improve performance.

[0009] Frameworks that utilize sparse non-negative matrix factorization for feature extraction also contain shortcomings. Although these frameworks can successfully generate invariant and efficient representations of inputs in some scenarios, the sparse non-negative matrix factorization-based approaches used in obtaining features are not always technically feasible or feasible in their current form. In some cases, limitations arise because the algorithms utilized by these frameworks do not capture the physiological constraints faced by biological systems.

[0010] Furthermore, in a given feature extraction approach, capturing the most informative structure from the input is often a different process than obtaining a representation of the input. Thus, any network that achieves both typically incorporates two separate structures to achieve these two goals. Many of these limitations can be improved or overcome when examining the mathematical algorithms that underpin these approaches in light of the physiological constraints faced by biological systems that can process visual data and exhibit learning. Some aspects of biological systems that are desirable in any sensory encoding process are not present in known approaches to sensory processing.

[0011] Another drawback of existing technologies is that they do not accurately mimic the processes of biological systems. An essential aspect of biological systems is their evolution. Organisms grow and develop over time, reach maturity, and eventually die. During their lives, organisms experience their surroundings and learn to adapt to them. From a sensory processing perspective, this constitutes a continuous period of sensory experience, allowing the organism to learn and relearn sensory events. As a corollary, a biological system does not encounter at one time all the events and stimuli that it is adapted to. It discovers these events gradually, determines their relevance to the experience, and adapts accordingly to represent them.

[0012] Furthermore, biological systems do not have separate "circuits" for capturing features and generating representations. The same structures adapt to a set of inputs and represent them. Moreover, the input representation is expected to guide the process of adaptation. In contrast, existing feature extraction approaches typically fail to replicate these important aspects of sensory processing and do not integrate the two processes.

[0013] Animals, even those with relatively simple brains, can recognize objects that are deformed, damaged, or occluded. Animal intelligence evolved from nothing, and the ability to learn, represent, and generalize these signals rapidly and consistently in diverse environments is critical to an animal's ability to survive in a constantly changing environment. Despite the great variation in cognitive sophistication, a surprising fact is that cognitive functions are based on local operations and synaptic learning rules. Modifications in synaptic strengths are dictated solely by the activity of presynaptic and postsynaptic neurons. Although they are indifferent to changes in other parts of the brain, the brain can learn to extract environmental signals from a small number of examples, whether simple or complex, generalize them, and recognize the identity and class of objects, driving appropriate behavioral responses. Despite recent advances in understanding biological neural systems, it is not known how the brain can use local learning rules to generate representations of objects that are invariant to signal corruption and variations in size, position, and viewpoint.

[0014] Inspired by early studies of the visual hierarchy, known artificial neural network models and deep learning variants rely on convolution and continuous integration of features to mimic cognitive functions and exhibit remarkable performance. Although these models are suggested to replicate the computations performed in the brain, they operate fundamentally differently from biological neural systems. Designed to address specific engineering problems, the models typically rely on a learning process that minimizes the discrepancy (or error, or cost function) between the desired and actual outputs. This process requires the network to "know" a predetermined set of inputs and their corresponding outcomes, and detected discrepancies can be propagated through the network to update the connection weights to minimize the error. The update and supervised training techniques directed at these goals make these neural networks very accurate in performing specific tasks, but this comes at various costs. For example, these networks do not have the ability to continuously learn in the same way as biological systems. Instead, once training is complete, the updated connection weights are "frozen" and do not change any more. Additionally, exposure to new tasks can lead to catastrophic forgetting. Training on specific examples does not generalize well beyond the training data and also makes the network vulnerable to adversarial attacks. To improve performance and robustness, a large number of layers and large amounts of training data are required.

[0015] In contrast, biological brains do not know specific inputs a priori; they learn without instructions or labels and have no natural mechanism for backpropagating errors. Organic systems are also constantly updated through experience, and in contrast to existing natural networks, they are remarkably robust against adversarial attacks. To capture the advantages inherent in biological systems, artificial network models should use local learning rules to achieve comprehensive success in object feature capture, representation, and classification. This approach has not been realized to date. [Brief description of the drawings]

[0016] In order to facilitate further description of the embodiments, the following drawings are provided, in which like reference numerals are intended to refer to like or corresponding parts, and in which:

[0017] [Figure 1A] FIG. 1A is a diagram of an example system for generating image analytics, according to one embodiment.

[0018] [Figure 1B] FIG. 1B is a block diagram illustrating example features of a computer vision system, according to one embodiment.

[0019] [Diagram 2] FIG. 2 is a diagram of an example neural network architecture, according to one embodiment.

[0020] [Diagram 3] FIG. 3 is a diagram illustrating how inputs in an input sequence can be fed into a representation layer for a neural network architecture according to one embodiment.

[0021] [Figure 4] FIG. 4 is a diagram illustrating how a neural network architecture can learn corrupted inputs in an input sequence, according to one embodiment.

[0022] [Figure 5A] FIG. 5A is a diagram illustrating how object characteristics can be captured in the output of a representation layer for a neural network architecture, according to one embodiment. [Figure 5B] FIG. 5B is a diagram illustrating how object characteristics can be captured in the output of the representation layer for a neural network architecture, according to one embodiment. [Figure 5C] FIG. 5C is a diagram illustrating how object characteristics can be captured in the output of the representation layer for a neural network architecture according to one embodiment.

[0023] [Figure 6] FIG. 6 is a diagram of an example neural network architecture, according to an embodiment.

[0024] [Figure 7A] FIG. 7A is a diagram illustrating object attributes that may be captured in the output for a neural network architecture, according to one embodiment. [Figure 7B] FIG. B is a diagram illustrating the object attributes that may be captured in the output for a neural network architecture, according to one embodiment.

[0025] [Figure 8] FIG. 8 is a flow chart illustrating an example method for neural network architecture, according to one embodiment.

[0026] The terms "first," "second," "third," "fourth," and the like in the specification and claims, when described, are used to distinguish between similar elements and do not necessarily denote a particular sequential or temporal order. It should be understood that the terms so used are interchangeable, where appropriate, such that the embodiments described herein are capable of operation in other orders than those illustrated or described herein.

[0027] Terms such as "left," "right," "front," "back," "rear," "top," "bottom," "above," "below," and the like in the specification and claims, where described, are used for descriptive purposes and do not necessarily describe fixed relative locations. It should be understood that terms so used are interchangeable, where appropriate, such that the devices, methods, and / or articles of manufacture described herein are capable of operation, for example, in orientations different from those illustrated or described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0028] The present disclosure relates to systems, methods, apparatus, computer program products, and techniques for providing a neural network architecture that utilizes local learning rules and a shallow two-layer neural network architecture to extract or generate robust, invariant object representations from objects contained in images. In some embodiments, the neural network architecture can be trained to generate invariant responses to various forms of corrupted image input. The learning process does not require any labeling of the training set or predetermined outcomes, eliminating the need for large training data sets during the learning process. Instead, the neural network architecture can generate invariant object representations using only local learning rules and without requiring backpropagation during the learning process or relying on the use of reconstruction errors or credit assignments. The enhanced object representations generated by the neural network architecture can be utilized to improve the performance of various computer vision functions, such as functions that may include, for example, object detection, object classification, object representation, object segmentation, and the like.

[0029] To overcome the limitations of known feature extraction techniques, a biologically inspired shallow two-layer, redundancy-embedding artificial neural network (ANN) is provided that learns global structure from objects in an experience-dependent manner. In one embodiment, the ANN comprises nodes that can be configured to extract unique input structures and efficiently represent the inputs. In some scenarios, a single ANN can incorporate the capabilities of both blind source separation and sparse restoration techniques. The ANN can include a modified Hopfield network that implements a learning rule that enables redundancy embedding. In one embodiment, the ANN includes biased connectivity and stochastic gradient descent learning to sequentially identify multiple inputs without catastrophic forgetting. The ANN can embezzle structures that uniquely identify individual objects and generate sparse, uncorrelated representations that are robust to various shapes of input corruption. Notably, the ANN can extract uncorrupted features in an unsupervised manner, learn from various corrupted input shapes to separate identity and rotation information from different images of rotated three-dimensional objects, and generate cells tuned to different object orientations in an unsupervised condition. Although an ANN can be trained very well to represent an initial set of data (such as the training set data), the ANN also performs well on images that are similar, but not identical, to the images contained in the initial (or training) data set. In such scenarios, the ANN can employ continuous learning so that it can adapt to new images and represent them more sparsely and robustly.

[0030] In one embodiment, the ANN includes a first layer of input nodes that can be connected in an all-to-all configuration with a second layer of representation nodes. Inhibitory recurrent connections between the representation nodes in the second layer provide negative input values ​​and can also be connected in an all-to-all configuration. The input nodes can be configured to detect patterns in the input data set and these patterns can be projected to the representation nodes in the second layer. Sparsity of the representation from the representation nodes of the ANN is generated by inhibitory recurrent connections between nodes in the representation layer. These inhibitory connections are different from the connections between second layer nodes in a traditional Hopfield network, which are excitatory recurrent connections. Establishing connections between the input nodes and the representation nodes allows the representation nodes to learn information related to the features extracted by the input nodes.

[0031] In an ANN, the incorporation of useful structure can be reflected in the tuning properties of the representation nodes (or second layer nodes). Tuning properties are a measure of how well the ANN is adapted (such as through weight updates) to extract features (or objects) from images input to it. The tuning properties of the representation nodes can be determined by how they are connected to earlier nodes (such as input nodes) in the sensory (signal) pathway. Thus, adaptation to an input can be related to changes in the connections of the ANN.

[0032] ANNs more accurately mimic real-world biological cognitive processes when compared to conventional approaches to neural network design. As mentioned above, many conventional artificial neural networks designed to represent objects utilize an optimization process in which the discrepancy between actual and desired outputs is reduced by updating network connections through mechanisms such as error backpropagation. This approach requires that individual connections at all levels of the artificial neural network sense errors found in later stages of the network. However, learning in biological neural systems is known to occur locally through pre-synaptic and postsynaptic activity. Furthermore, conventional techniques require that the artificial neural network "knows" the exact outcome for a certain set of inputs, which biological neural networks do not. Furthermore, many existing artificial neural networks require a separate training phase, whereas biological neural networks are constantly learning (i.e., the weights of the connections between various neurons / nodes are constantly updated throughout the life of the neural network). These aspects of biological neural networks, despite their complexity, make them less susceptible to adversarial attacks than many existing artificial neural networks. The ANNs described throughout this disclosure are modeled to more accurately mimic these and other aspects of biological neural networks. Moreover, similar to biological systems, representations in ANNs can be non-negative.

[0033] In one embodiment, the ANN described herein dynamically updates or changes the tuning properties for the representation nodes as the node's connections change. Appropriate changes in connectivity can lead to the nodes being tuned to the most beneficial structure. Because connections between two nodes can be both excitatory and inhibitory, changes in these connections can be of either nature as well, and thus updates in different connections can result in different positive or negative signs. Such updates may appear to be in contradiction with the non-negativity constraint on the node's values ​​that helps capture beneficial structures. However, although connectivity changes can be bidirectional, inhibitory connections can only reduce the activity of a node without causing any node's value to fall below zero. In this setting, the ANN does not have to subtract the properties of the nodes from each other. Thus, the non-negativity constraint can be satisfied even if a node receives both excitatory and inhibitory inputs.

[0034] Furthermore, ANNs can extract unique features from inputs in an experience-dependent manner, and based on such structure, can generate sparse and efficient representations of the inputs. Unlike traditional Hopfield network-based neural networks, the ANNs described throughout this disclosure can be designed to be adaptive. The connectivity between the input layer and the representation layer can be changed based on the input to optimize the representation. Updating the connectivity of the ANN can be achieved using a stochastic gradient descent (SGD) type approach. Using this SGD-like approach, the ANN can gradually adapt to new inputs in a way that does not affect its adaptation to other previous inputs. The ANN can adapt to all different inputs by repeatedly encountering the inputs.

[0035] Unlike some methods, such as matrix factorization approaches, whose efficiency decreases with the number of inputs, the ANN design described herein allows for an increase in efficiency with both iterative encounters and the number of inputs. Accommodating a greater number of inputs allows the ANN to contain more information about the inputs, and accommodating more information in the ANN can result in better utilization of the ANN capacity and an increase in efficiency.

[0036] In some embodiments, the two-layer neural network architecture of the ANN can be extended or connected to a classification layer to create a classification network. The discrimination (or representation) layer of the two-layer neural network highlights the differences between different objects received as input by the neural network, while the classification layer identifies shared features between different objects in the input. Nodes in the classification layer can be subject to mutual excitation and global inhibition from other nodes in the classification layer. In some embodiments, these nodes can be one-to-one excitatory connected to nodes in the discrimination layer and inhibitory connected to nodes in the input layer. These design concepts are modeled after observed configurations in vertebrate sensory cortex. As explained in more detail below, the design of the classification network allows it to classify similar objects and to identify the same object from different perspectives, sizes, and / or locations. It further allows the classification network to classify representations of the same object (different sizes, perspectives, etc.) without having processed or experienced special representations.

[0037] Classification networks have the additional advantage over traditional approaches in that they are fully interpretable (so-called white box) and do not suffer from catastrophic forgetting, a phenomenon commonly observed in traditional approaches that results in a neural network forgetting how to perform one task after being trained on another. Classification networks perform their analysis of the inputs in an efficient and robust manner.

[0038] An object's identity is embedded in the structural relationships between its features, and the neural network architecture of the present disclosure can exploit those relationships or dependencies to encode the object's identity. Moreover, as explained in more detail below, the neural network architecture captures these dependencies to the fullest extent possible, so that the presence of an object can be identified and an invariant representation generated or extracted without precise details of the input pattern.

[0039] The techniques discussed herein can be used in a variety of different situations and environments. One useful application of these techniques is in the context of computer vision, which can be applied across a wide variety of different applications. For example, the techniques disclosed herein can be integrated into any application, device, or system that can benefit from using the object representations described herein.

[0040] One example application of these techniques is in the context of facial recognition. Another useful application of these techniques is in the context of surveillance systems (e.g., at security checkpoints). Another useful application of these techniques is in the context of scene analysis applications (e.g., can be used in automated, unmanned, and / or autonomous vehicles that rely on automated, unmanned, and / or autonomous systems to control the vehicles). Another useful application of these techniques is in the context of intelligent or automated traffic control systems. Another useful application of these techniques is in image editing applications. Another useful application of these techniques is in the context of satellite imaging systems. Additional useful applications include quality control systems (e.g., industrial sample checking, and industrial defect detection), agricultural analysis systems, and medical analysis systems (e.g., both human and animal applications).

[0041] The techniques discussed herein can also be applied in many other contexts. For example, these techniques can be used to process and / or analyze DNA and RNA sequences, auditory data, sensory data, or data collected from other sources. In these contexts, the neural network architecture can identify, classify, or extract other information from the input data that relates to objects in the data, which may be certain patterns or other features of the data. The neural network architecture can generally perform the same functions of extracting representations and / or classifying portions of the input data as it can for visual images. The data that is analyzed and / or processed by the neural network architecture can be pre-processed in some way, such as by converting the data into pixels to form an image that is input to the neural network architecture. Other pre-processing steps, such as scaling and / or applying wavelets or Fourier transforms, can be applied to all types of input.

[0042] The embodiments described in this disclosure can be combined in various ways. Any aspect or feature described for one embodiment can be incorporated into any other embodiment mentioned in this disclosure. Furthermore, any of the embodiments described herein can be hardware-based, software-based, or preferably comprise a mixture of both hardware and software elements. Thus, although the description herein may describe certain embodiments, features, or components as being implemented in software or hardware, it should be recognized that any embodiment, feature, and / or component mentioned in this disclosure can be implemented in hardware and / or software.

[0043] Figure 1A is a diagram of an example system 100 according to one embodiment. Figure 1B is a diagram illustrating example features and / or functionality associated with a computer vision system 150. Figures 1A and 1B are considered together below.

[0044] System 100 includes one or more computing devices 110 and one or more servers 120 in communication via a network 190. Computer vision system 150 is stored on and executed by one or more servers 120. Network 190 may represent any type of communication network, such as, for example, a communication network comprising a local area network (e.g., a Wi-Fi network), a personal area network (e.g., a Bluetooth network), a wide area network, an intranet, the Internet, a cellular network, a television network, and / or other types of networks.

[0045] 1A and 1B, including computing device 110, server 120, and computer vision system 150, may be configured to communicate with each other directly and / or over network 190 via wired or wireless communication links, or a combination of the two. Each of computing device 110, server 120, and computer vision system 150 may also be equipped with one or more communication devices, one or more computer storage devices 201, and one or more processing devices 202 (central processing units) capable of executing computer program instructions.

[0046] The one or more computer storage devices 201 may include (i) non-volatile memory, such as, for example, read-only memory (ROM), and / or (ii) volatile memory, such as, for example, random access memory (RAM). The non-volatile memory may be removable and / or non-removable non-volatile memory. Meanwhile, the RAM may include dynamic RAM (DRAM), static RAM (SRAM), and the like. Furthermore, the ROM may include mask programmable ROM, programmable ROM (PROM), one-time programmable ROM (OTP), erasable programmable read-only memory (EPROM), electrically erasable programmable ROM (EEPROM) (e.g., electrically alterable read-only memory (EAROM) and / or flash memory), and the like. In an embodiment, the computer storage device 201 may be a physical non-transitory medium. The one or more computer storage devices 201 may store instructions associated with performing functions performed by the computer vision system 150.

[0047] The one or more processing devices 202 may include one or more central processing units (CPUs), one or more microprocessors, one or more microcontrollers, one or more controllers, one or more complex instruction set computing (CISC) microprocessors, one or more reduced instruction set computing (RISC) microprocessors, one or more very long instruction word (VLIW) microprocessors, one or more graphics processor units (GPUs), one or more digital signal processors, one or more application specific integrated circuits (ASICs), and / or any other type of processor or processing circuitry capable of performing the desired functions. The one or more processing devices 202 may be configured to execute any computer program instructions stored or contained on one or more computer storage devices, including, but not limited to, instructions associated with performing functions performed by the computer vision system 150.

[0048] Each of the one or more communication devices may include wired and wireless communication devices and / or interfaces that enable communication using wired and / or wireless communication technologies. The wired and / or wireless communication may be implemented using any one or combination of wired and / or wireless communication network topologies (e.g., ring, line, tree, bus, mesh, star, daisy chain, hybrid, etc.) and / or protocols (e.g., personal area network (PAN) protocols, local area network (LAN) protocols, wide area network (WAN) protocols, cellular network protocols, power line network protocols, etc.). In some embodiments, the one or more communication devices may additionally or alternatively include one or more modem devices, one or more router devices, one or more access points, and / or one or more mobile hotspots.

[0049] In one embodiment, computing device 110 may represent a desktop computer, a laptop computer, a mobile device (e.g., a smart phone, a personal digital assistant, a tablet device, a vehicle computing device, or any other device that is mobile in nature), and / or other types of devices. One or more servers 120 may generally represent any type of computing device, including any of the computing devices 110 described above. In one embodiment, one or more servers 120 comprise one or more mainframe computing devices running a web server to communicate with computing device 110 and other devices over network 190 (e.g., via the Internet).

[0050] In one embodiment, computer vision system 150 is stored on and executed by one or more servers 120. Computer vision system 150 can be configured to perform any and all operations associated with analyzing images 130 and / or performing computer vision functions, including, but not limited to, functions for performing feature extraction, object detection, object classification, and object segmentation.

[0051] The images 130 provided to and analyzed by the computer vision system 150 can include any type of image. In some embodiments, the images 130 can include one or more two-dimensional (2D) images. In some embodiments, the images 130 can include one or more three-dimensional (3D) images. Additionally, the images 130 can be created from non-visual data sources by pixelating (i.e., converting the non-visual data into an "image" that includes one or more "pixels" that represent portions of the non-visual data), such as DNA or RAN sequences, auditory data, sensory data, and other types of data. The images 130 can be captured in any digital or analog format and using any color space or color model. The images 130 can be excerpts from a video. Exemplary image formats include, but are not limited to, bitmap (BMP), JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), GIF (Graphics Interchange Format), PNG (Portable Network Graphics), STEP (Product Data Exchange Standard), and the like. Example color spaces or models may include, but are not limited to, sRGB (standard red-green-blue), Adobe RGB, grayscale, etc. Additionally, in some embodiments, some or all of the images 130 may be pre-processed and / or transformed before being analyzed by the computer vision system 150. For example, the images 130 may be split into different color components and / or processed via transforms such as Fourier or wavelet transforms, etc. Other pre-processing and transformation operations may also be applied.

[0052] The images 130 received by the computer vision system 150 may be captured by any type of camera device. A camera device may include any device that includes an imaging sensor, a camera, or an optical device. For example, a camera device may represent a still image camera, a video camera, and / or other device that includes an image / video sensor. A camera device may capture and / or store both visible and non-visible spectrums, including but not limited to ultraviolet (UV), infrared (IR), or positron emission tomography (PET), magnetic resonance imaging (MRI), x-ray, ultrasound, and other types of medical and non-medical imaging. A camera device may also include devices that include an imaging sensor, a camera, or an optical device and perform other functions unrelated to capturing an image. For example, a camera device may include a mobile device (e.g., a smart phone, a cell phone), a tablet device, a computing device, a desktop computer, and the like. A camera device may be equipped with an analog-to-digital (A / D) converter and / or a digital-to-analog (D / A) converter based on the configuration or design of the camera device. In some embodiments, the computing device 110 shown in FIG. 1 may include any of the camera devices described above, as well as other types of camera devices.

[0053] Each of the images 130 (or the corresponding scenes captured in the images 130) may include one or more objects 135. Generally speaking, any type of object 135 may be included in the images 130, and the types of objects 135 included in the images 130 may vary widely. The objects 135 included in the images 130 may correspond to various types of inanimate objects (e.g., vehicles, beds, desks, windows, tools, appliances, industrial equipment, curtains, sporting goods, furniture, etc.), animate objects (e.g., humans, faces, animals, plants, etc.), structures (e.g., buildings, houses, etc.), symbols (Roman characters of the alphabet, Arabic numerals, Chinese characters, etc.), and / or the like. When the underlying data being analyzed is inherently invisible (such as DNA or RNA sequences, auditory data captured by a microphone or audio sensor, etc.), the objects 135 may include any significant patterns or features found in the data. The images 130 received by the computer vision system 150 may be provided to a neural network architecture 140 for processing and / or analysis.

[0054] In particular, neural network architecture 140 can extract improved or optimized object representations 165 from image 130. Object representations 165 can represent features, embeddings, encodings, vectors, and / or the like, and each object representation 165 can include encoded data that represents and / or identifies one or more objects 135 contained in image 130. In one embodiment, neural network architecture 140 can sequentially learn patterns presented to it, and this learned knowledge can be leveraged to optimize object representations 165 and to perform other functions described herein.

[0055] The structure or configuration of the neural network architecture 140 may vary. In some embodiments, the neural network architecture 140 may include one or more recurrent neural networks (RNNs). For example, in some cases, the neural network architecture 140 may include a Hopfield network that has been modified and optimized to perform the tasks described herein. In some embodiments, the modified Hopfield network is a shallow two-layer RNN with a first layer of input nodes (or input neurons) and a second layer of representation nodes (or representation neurons). Each of the representation nodes may be connected to each of the input nodes in an all-to-all configuration, and the feedforward weights between the inputs and the representation nodes may be selected to minimize the possibility that two representation nodes are simultaneously active. Additionally, the representation nodes may be connected to each other using recurrent connections. In some embodiments, the biased connectivity between the nodes is combined with a learning mechanism based on stochastic gradient descent (SGD) to enable the neural network architecture 140 to sequentially identify multiple inputs without catastrophic forgetting. The biased connectivity and lateral inhibition in the neural network architecture 140 allow the representation nodes to encode structure that uniquely identifies individual objects.

[0056] In some embodiments, slow synaptic weight changes allow for continuous learning from individual examples. In such embodiments, the slowness (relative to conventional image analysis systems) does not cause impairments in overall network connectivity, but allows specific patterns to be encoded. In some embodiments, there is no normalization step at each learning iteration, which can prevent the generation or assignment of negative synaptic weights. Such results are due to slow synaptic weight changes, which are similar to biological systems (e.g., in animal brains, where synaptic weights are never negative).

[0057] In some embodiments, the number of representation nodes included in the neural network architecture 140 can be directly proportional to the number of images or objects desired to be recognized. In such an example, the representation layer can include approximately the same number of nodes as the number of images to be identified. In some embodiments, there can be a two-fold or greater (up to ten-fold or greater) expansion in the number of nodes from the primary layer to the representation layer. For many applications of the neural network architecture 140, more nodes in each layer will produce better results. There is no upper limit to the total number of nodes that may comprise the neural network architecture 140.

[0058] In some embodiments, the neural network architecture 140 can be configured to be adaptive such that the connectivity between the input layer and the representation layer can change based on a given input image being processed. This dynamic adaptation of the connections between the input layer and the representation layer allows the neural network architecture 140 to optimize the generated object representation 165. The resulting object representation 165 is sparse and the individual nodes of the neural network architecture 140 are uncorrelated, leading to an efficient encoding of the input pattern. Furthermore, because the neural network architecture 140 can extract useful structures from the object 135 in the image 130, the resulting object representation 165 is robust to various forms of degradation, corruption, and occlusion.

[0059] Other configurations of the neural network architecture 140 may also be employed. Although certain portions of the present disclosure describe embodiments in which the neural network architecture 140 includes a modified Hopfield network or an RNN, it should be understood that the principles described herein may be applied to a variety of learning models or networks. In some examples, the layers of the neural network architecture 140 may be appropriately stacked and / or parallelized in various configurations to form a deep neural network that performs the functions described herein. In some embodiments in which the neural network architecture 140 is stacked, the output of the representation layer or the classification layer (in examples in which the neural network architecture 140 includes a third layer), or both, may be used as inputs to the next neural network (such as another two- or three-layer modified Hopfield network). In such embodiments, the inputs to these latter neural networks are derived from the activity from each node of the previous neural network architecture 140 and may be treated as pixels of input to the next network. In some embodiments, the neural network architecture 140 may include a classical perceptron as an additional layer that reads class information.

[0060] In an embodiment where neural network architectures 140 are stacked, a first neural network architecture 140 can be used as a scanner, allowing a limited number of pixels to cover a larger scene (similar to how a biological organism uses its eyes to focus on one area of ​​the field of view at a time, but synthesizes the entire scene). In order to synthesize the entire scene, the scanned images (or sub-scenes) can be treated as time invariant, even if they were acquired at different times.

[0061] In one example, the principles described herein can be extended or applied to other types of RNNs not specifically mentioned in this disclosure. In another example, the principles described herein can be extended or applied to reinforced learning neural networks. In a further example, the principles described herein can be extended or applied to convolutional neural networks (CNNs).

[0062] For example, in some embodiments, the neural network architecture 140 may additionally or alternatively comprise a convolutional neural network (CNN) or multiple convolutional neural networks. Each CNN may represent an artificial neural network and may be configured to analyze the image 130 and perform deep learning and / or machine learning functions on the image 130. Each CNN may include multiple layers, including, but not limited to, one or more input layers, one or more output layers, one or more convolutional layers (e.g., including learnable filters), one or more ReLU (rectified linear unit) layers, one or more pooling layers, one or more fully connected layers, one or more normalization layers, etc. The configuration of the CNN and their corresponding layers may be configured to enable the CNN to learn and perform various functions for analyzing, interpreting, and understanding the image 130, including any of the functions described in this disclosure.

[0063] Regardless of its configuration, the neural network architecture 140 can be trained to extract robust object representations 165 from the input images 130. In some embodiments, the neural network architecture 140 can also be trained to utilize the object representations 165 to perform one or more computer vision functions. For example, in some cases, the object representations 165 can be utilized to perform an object detection function, which can include predicting or identifying (e.g., using bounding boxes) the location of an object 135 associated with one or more target classes in the image 130. Additionally or alternatively, the object representations 165 can be utilized to perform an object classification function (e.g., which can include predicting or determining whether an object 135 in the image 130 belongs to one or more target semantic classes and / or predicting or determining a label for the object 135 in the image 130) and / or an instance segmentation function (e.g., which can include predicting or identifying the exact location of the object 135 in the image 130 with pixel-level accuracy). The neural network architecture 140 can also be trained to perform other types of computer vision functions.

[0064] The neural network architecture 140 of the computer vision system 150 is configured to generate and output analysis information 160 based on the analysis of the image 130. The analysis information 160 for the image 130 may generally include any information or data associated with analyzing, interpreting, understanding, and / or classifying the image 130 and the object 135 contained therein. In one embodiment, the analysis information 160 may include information or data representative of an object representation 165 extracted from the input image 130. The analysis information 160 may further include orientation information indicating an angle of rotation, orientation, or position of the object 135 contained in the image 130.

[0065] Additionally or alternatively, the analysis information 160 may include information or data indicative of results of computer vision functions performed by the neural network architecture 140. For example, the analysis information 160 may include predictions and / or results associated with performing object detection, object classification, and / or other computer vision functions.

[0066] 1, computer vision system 150 can be stored on or executed by one or more servers 120. In other example systems, computer vision system 150 can additionally or alternatively be stored on or executed by computing device 110 and / or other devices. For example, in some embodiments, computer vision system 150 can be integrated directly into a camera device to enable the camera device to analyze images using the techniques described herein.

[0067] Similarly, computer vision system 150 may also be stored as a local application on computing device 110 or may be integrated with local applications stored on computing device 110 to implement the techniques described herein. For example, in one embodiment, computer vision system 150 may be integrated with (or communicate with) various applications including, but not limited to, facial recognition applications, automated vehicle applications, intelligent transportation applications, surveillance applications, security applications, industrial quality control applications, medical applications, agricultural applications, veterinary applications, image editing applications, social media applications, and / or other applications stored on computing device 110 and / or server 120.

[0068] In some particularly useful applications, the neural network architecture 140 can be integrated with face recognition applications to generate pseudo images to aid in identifying faces or face objects. For example, upon receiving a given image 130 containing a face object, the neural network architecture 140 can robustly generate consistent pseudo images of unknown or deformed shapes (e.g., which can include deformed face objects), which can be used for face recognition purposes. Storage of the actual face object is not required, which can be beneficial both from a technical standpoint (e.g., by reducing storage space usage) and a privacy standpoint.

[0069] In some embodiments where continuous learning by the neural network architecture 140 is not utilized, the neural network architecture 140 can be deployed with a pre-trained weight matrix so that the neural network architecture 140 is immediately available for its assigned application. Additionally, the neural network architecture 140 can also perform additional learning, if desired, even if it is deployed with a pre-trained weight matrix. In some embodiments where no new objects are expected, or where only a small number of objects are expected, the neural network architecture 140 with its learned set of weights can be stored and used directly without any learning (or adaptation) mechanism to accelerate its performance. Alternatively, or additionally, the neural network architecture 140 can be enabled to continually update its weights to account for new objects.

[0070] In some embodiments, one or more computing devices 110 can allow an individual to access the computer vision system 150 over the network 190 (e.g., over the Internet via a web browser application). For example, after a camera device (e.g., which can be directly integrated into the computing device 110 or can be a separate device from the computing device 110) captures one or more images 130, an individual can utilize the computing device 110 to transmit the one or more images 130 to the computer vision system 150 over the network 190. The computer vision system 150 can analyze the one or more images 130 using techniques described in this disclosure. Analysis information 160 generated by the computer vision system 150 can be transmitted over the network 190 to the computing device 110 that transmitted the one or more images 130 and / or to other computing devices 110.

[0071] As illustrated in FIG. 2, the neural network architecture 140 may include a shallow two-layer ANN 200 (e.g., a modified Hopfield network) with a first layer of input nodes 210a-d (which may also be referred to herein as primary layer nodes) and a second layer of representation nodes 220a-e (which may also be referred to herein as discrimination nodes, representation nodes, or secondary layer nodes). Each of the input nodes 210a-d may be connected to each of the representation nodes 220a-e in an all-to-all configuration. In one embodiment, initial feedforward weights between the input nodes 210a-d and the representation nodes 220a-e may be selected based at least in part on the distributed structure of the input data set to minimize the possibility that any two of the representation nodes 220a-e are simultaneously active. Additionally, the representation nodes 220a-e may be connected to each other in the all-to-all configuration using recurrent connections that are inhibitory. The biased connectivity and lateral inhibition in the neural network architecture 140 allow the nodes to encode a structure that uniquely identifies individual objects 135. The sparseness of the object representation 165 of the objects 135 embedded in the image 130 is due to the inhibitory recurrent connections between the representation nodes 220a-e. These inhibitory connections are not present in conventional Hopfield networks that contain excitatory recurrent connections.

[0072] In some embodiments, the two-layer ANN 200 can be configured to be adaptive such that the connectivity between the input layer nodes 210a-d and the representation layer nodes 220a-e can change based on a given input image being processed. This dynamic adaptation of the connections between the input layer nodes 210a-d and the representation layer nodes 220a-e allows the two-layer ANN 200 to optimize the generated object representation 165. The resulting object representation 165 is sparse and the individual representation layer nodes 220a-e of the two-layer ANN 200 are uncorrelated, thereby leading to an efficient encoding of the input pattern. Furthermore, because the two-layer ANN 200 can extract useful structures from the object 135 in the image 130, the resulting object representation 165 is robust to various forms of degradation, corruption, and occlusion.

[0073] In one embodiment, weights between any two nodes are updated using local learning rules. For example, a connection between an input node and a representation node can be strengthened when both nodes are active. When two of the representation nodes 220a-e are simultaneously active, the input connections to these two nodes can be weakened, and an inhibitory weight can be increased when two of the representation nodes 220a-e have the same level of activity. Strengthening the connections between input nodes 210a-d and representation nodes 220a-e is an example of local Hebbian behavior, while weakening any two of the representation nodes 220a-e that are simultaneously active is an example of local non-Hebbian behavior.

[0074] The way these connections are strengthened or weakened can be uniquely modeled using local learning rules at the representation nodes 220a-e to mimic real-world biological cognition processes. In biological systems, the Hebbian learning rule (to store p patterns in a network with N units, the weights that ensure memory of the patterns are

number

[0075] In one embodiment, the two-layer ANN 200 can rapidly represent an image 130 after being exposed to the image 130. For example, the two-layer ANN 200 can accurately capture structural features of inputs including images of symbols from world languages, reaching a learning plateau of performance after fewer than 10 exposures to the symbols. Furthermore, the two-layer ANN 200 is capable of continuous learning. For example, the two-layer ANN 200 can learn to represent a new input type (such as a face) after learning to represent a different input type (such as a symbol from a world language) without "forgetting" how it represented the earlier input type.

[0076] In some embodiments, the number of representation nodes 220a-e included in the neural network architecture 140 can be directly proportional to the number of images 130 or objects 135 desired to be recognized. In such an example, the representation layer 220 can include approximately the same number of nodes as the number of images 130 to be identified. In some embodiments, there can be a 2-fold, 10-fold, or even greater expansion in the number of nodes from the input layer 210 to the representation layer 220. For many applications of the neural network architecture 140, more nodes in each layer will produce better results. There is no upper limit to the total number of nodes comprising the neural network architecture 140. In some embodiments, there can be fewer nodes in the representation 220 or classification (discussed in more detail below) layer than in the input layer 210. For example, the input layer of a two-layer ANN 200 can have 10,000 nodes and in the representation layer and 500 nodes. In another example of a two-layer ANN 200, the input layer 210 can include 10,000 nodes and the representation layer 220 can include 1,000 nodes.

[0077] In some embodiments, slow synaptic weight changes allow for continuous learning from individual examples. In such embodiments, the slowness (relative to conventional image analysis systems) does not cause impairments in overall network connectivity, but allows specific patterns to be encoded. In some embodiments, there is no normalization step at each learning iteration, which can prevent the generation or assignment of negative synaptic weights. Such results are due to slow synaptic weight changes, which are similar to biological systems (e.g., in animal brains, where synaptic weights are never negative).

[0078] The properties of the representation nodes 220a-e in the second layer can be modeled or based on properties of neurons observed in biological systems. For example, certain concepts such as membrane potentials and firing rates derived from biological neural networks or neurons therein can be used to set the attributes of the nodes in the ANN 200. The connections between the (primary) input layer nodes 210a-d and the (secondary) representation layer nodes 220a-e can be represented by a connectivity matrix of the shape of the connectivity matrix, depending on the number of input nodes 210a-d and the number of representation layer nodes 220a-e (and thus need not be symmetric). Meanwhile, the recurrent connections between the representation nodes 220a-e in the second layer can be described by a symmetric matrix. In one embodiment, the connection strength from node i to node j in the representation layer 220 is the same as the connection strength from node j to node i.

[0079] The strength of connections between nodes can be static or can adapt over time. For example, the properties of the nodes can change as the ANN 200 encounters inputs. In some embodiments where the ANN 200 is not specifically adapted to certain types of inputs, the properties of the representation nodes 220a-e in the second layer arise because of their connections to the input nodes 210a-d. Thus, the strength of the recurrent connections can be the similarity of the representation node 220a-e connections to the primary nodes 210a-e. In embodiments where two of the representation nodes 220a-e are similarly connected to input nodes 210a-d in the primary layer, any given input will activate them, and their recurrent interactions will also be similar.

[0080] The ANN 200 can be fully dynamic in some embodiments. For example, the ANN 200 can adapt to inputs not only through changes in the connections between the input nodes 210a-d and the representation nodes 220a-e, but also through updating the iterative connection strengths (between the representation nodes 220a-e). In one embodiment, the ANN 200

number

number

number

number

number

[0081] Nodes in the ANN 200 may exhibit certain non-linear behavior. For example, nodes 220a-e in the representation layer may have a certain threshold, and when the node's value is below that threshold, the node is inactive (or "not firing"). This value may be determined by summing the inputs to the node multiplied by the weights applied to those inputs. After the threshold is reached, the node may respond linearly to its inputs. In some embodiments, this region of linear response may be limited, for example, because the node response saturates at a certain level of activity. Node behavior may be modeled in a number of ways. In some embodiments, the behavior of the representation nodes 220a-e of the ANN 200 is modeled based on biological structures, such as neurons. The behavior of these nodes is determined by certain parameters taken from the biological context, i.e., membrane potential, firing rate, etc. For example, the nodes in the representation layer 220a-e may be modeled using a "leaky integrate-and-fire" model.

[0082] In one embodiment, the goodness or quality of the fit of the ANN 200 can be measured by the difference between the input and its reconstruction obtained from the tuning properties and response values ​​of the representation nodes 220a-e.

number

[0083] The connectivity of the ANN 200 can be updated in a number of ways. For example, the connectivity of the ANN 200 can be updated using the following three step procedure: First, for each state of connectivity, tuning properties are determined. Second, changes in the tuning properties that will reduce the error are calculated from the representation, and finally, proportional changes are made in the connectivity.

[0084] The inability of the ANN 200 to distinguish between different inputs can reduce its effectiveness. In one embodiment, the ANN 200 can be optimized to represent the inputs based on the most useful structure and adapt to different shapes of the inputs, and the initial weights of the ANN 200 can be set to achieve the distinction between inputs that are different from the first input it received. Otherwise, the ANN 200 may not be able to distinguish between the different inputs, leading to a flawed adaptation process and resulting in only selective adaptation.

[0085] In one embodiment, different inputs activate different nodes, and to avoid mapping different inputs to the same expression, the initial weights are set to minimize the probability that any two of expression nodes 220a-e are activated by the same input. This constraint is achieved by deducting the expected value of the variance-covariance matrix of the node's response profile from the identity matrix, i.e., E[V T ]=I, where V is a matrix of representations of the distinct inputs and I is the identity matrix. In an embodiment in which the nonlinearity imposed on the ANN 200 by the function g is ignored, V is modeled by the input matrix and the weight matrix W as V=W T We can approximate it as Y, where Y is the input matrix. The weight matrix W is the weighting matrix for the early nodes (Σ yy Based on the variance-covariance matrix of the response profile of

number

[0086] Perfect knowledge of the inputs is not required. For example, a subsample of the more likely inputs to be encountered can also configure the ANN 200 such that no expected input of the ANN 200 is associated with the same representation. In some embodiments where N is greater than or equal to M, the ANN 200 can generalize by ensuring that η has orthogonal columns (in other words, when the number of representation nodes is greater than the number of first-order nodes).

[0087] In one embodiment in which the connectivity between the primary layer input nodes 210a-d and the representation layer representation nodes 220a-e of the ANN 200 is updated, the update is

number

number

number

[0088] This optimization problem for updating the connectivity between the primary layer input nodes 210a-d and the representation layer representation nodes 220a-e can be solved by employing a gradient descent approach. In this approach, the value of a function is iteratively reduced by updating its variables along its gradient. In other words, for every variable, a value that further reduces the function is found by moving along the negative gradient of the function with respect to the variable. Eventually, a minimum of the function is reached. The gradient descent step is

number

number

[0089]

number

number

number

number

number

[0090]

number

number

number

number

number

number

number

number

number

number

number

[0091] However, the minus sign in front of this update component makes it non-Hebbian in nature. That is, the update reduces all connections between input nodes 210a-d in the primary layer and two similarly active nodes in the representation layer 220. In other words, if two of the representation nodes 220a-e are firing together, their inputs are reduced so that they can be separated. Overall, the updates in connectivity strengthen the connections between simultaneously firing nodes in the primary layer 210 and the representation layer 220, but reduce the likelihood that two of the representation nodes 220a-e will fire together. This process allows the ANN 200 to be gradually tuned to features from multiple inputs presented to it.

[0092] In certain embodiments where updating connections to adapt to new inputs in the manner described above disrupts ANN 200 adaptation to previously encountered inputs, ANN 200 can utilize simultaneous relearning of features from all previous inputs to minimize the effects of such disruption.

[0093] In one embodiment, the ANN 200 can use stochastic gradient descent (SGD) to solve the problem of preventing the ANN from adapting to previously encountered inputs. This is a stochastic approximation to gradient descent optimization. In this method, instead of optimizing the objective function over all training data, the ANN 200 optimizes the function over only a randomly selected subset of the data. To better understand this approach, consider that for any optimization problem, the value of the objective function can be expressed as the sum of the losses for each data point, i.e.,

number

number

number

[0094] In one embodiment, the ANN 200 is designed to update its connectivity as it learns to efficiently represent a finite set of inputs based on the most informative structure. The objective function can be used as a measure of fitness, the optimization variables can be used as a matrix of tuning characteristics, and the training data points can be used as pairs of inputs and their corresponding representations. Since a single input can be a subset of the data points, the SGD method can train the ANN 200 for all inputs presented in succession, although the SGD will not reach an optimum value. The step size can be of any size when using the SGD method. In one embodiment, the step size for a given implementation of the ANN 200 can be determined through an iterative process. The process begins by selecting a very small step size and running a simulation of the ANN 200 for some test input data. Once the weights of the ANN 200 have been adjusted, the output of the ANN 200 can be compared to the optimal output for the inputted test data. The value of the step size can be adjusted upwards until the output of the ANN 200 does not match the input. However, since only a subset of data points are considered while estimating the gradient, taking larger gradient steps in SGD may result in the updated points being very far from the optimum. In one embodiment, only small step sizes are used. The adaptation process may also require that the connectivity be updated to a particular strength in order for the adaptation to be effective (smaller updates in connectivity may be indistinguishable from unadapted connectivity), so that a minimum step size or minimum updates are required. To address this issue, connectivity updates are performed with smaller step sizes and utilize multiple representations of the same input to reach the desired level of adaptation. These types of updates are realistically achievable and provide a way to understand how the frequency of the input affected the adaptation process.

[0095] Unlike certain conventional approaches, such as matrix factorization, which cannot represent inputs that are not included in the input matrix (and may require the use of a separate algorithm for sparse recovery of the inputs), ANN 200 can perform both of these tasks (i.e., solving the sparse recovery problem using SGD and updating the connectivity between the primary layer input nodes 210a-d and the representation nodes 220a-e). ANN 200 can function in two modes. In mode 0, ANN 200 can only perform sparse recovery to generate the desired representation, since the connectivity between the primary 210a-d and the representation nodes 220a-e and the inputs are given as arguments to ANN 200. When functioning in mode 0, no updates on the connectivity are performed. In mode 1, ANN 200 performs both sparse recovery and basis fitting with the initial connectivity, and the inputs are given as arguments to ANN 200. In mode 1, ANN 200 can also generate a sparse representation of the inputs, and the connections between the various nodes are updated using the obtained representation and the corresponding inputs to ensure learning. An ANN 200 operating in mode 1 can learn to represent an initial set of data (such as training set data) very well, but the ANN 200 can also perform well on images 130 that are similar, but not identical, to images contained in the initial (or training) data set. Because the ANN 200 can adapt to new images 130 and employ continuous learning, it can represent images 130 more sparsely and robustly.

[0096] The ANN200 described herein differs from traditional hierarchical assembly models that attempt to account for the increasing complexity of receptive field characteristics along the visual pathway and that later formed the basis of convolutional neural networks. These traditional models assume that neurons in cognitive centers reproduce precise object details. However, precise object image reconstruction is not always necessary for robust representations, and this deeply rooted assumption creates unnecessary complexity in modeling object recognition.

[0097] The ANN 200 described herein does not need to calculate reconstruction errors to evaluate its learning performance. By capturing the dependencies that define the objects 135 and their classes, the ANN 200 can generate remarkably consistent representations of the same object 135 across different conditions. Size, translation, and rotation invariance indicate that the ANN 200 can naturally link together the features that define the object 135 or its class without being ostensibly designed to do so. It allows for a nonlinear transformation of input signals into representation geometries appropriate for identification and differentiation. One aspect of the ANN 200 is that it can generate invariant responses to corrupted inputs, in part because its design employs biological systems for inspiration. Sensory stimuli trigger high-dimensional neural activity that reflects not only the identity of different objects, but also their context, internal brain states, and other sensorimotor activity. The high-dimensional responses can be mapped to low-dimensional manifolds that are object-specific and unperturbed by neural and environmental variability.

[0098] One distinguishing feature of the ANN 200 in comparison with conventional frameworks is that the initial connectivity between the input nodes 210a-d and the representation nodes 220a-e in the discriminative (or representation) layer takes into account the distributed structure of the input data set, ensuring that any two of the representation nodes 220a-e are less likely to fire together for any given input. Furthermore, the learning process does not utilize any labels or require any pre-determined outcomes. It is totally unsupervised, as the representations evolve with exposure to individual images. As such, the recurrent weights do not reflect a pre-determined correlation structure between the representation patterns. Notably, the learning rules are all local, as follows:

number

number

number

[0099] Along with the linear sum of activities to drive the response, the ANN 200 adjusts the connection strengths in an activity-dependent manner.

number

number

[0100] The ANN 200 can filter out noise in the input and extract cleaner structures from the input. The receptive fields of the representation nodes 220a-e of the ANN 200 can generate structures similar to the faces input to the ANN 200 (along with random noise) but are not specific to any input face. The receptive fields can be much less noisy than the input faces at all levels of training, as measured by the average power at the highest spatial frequencies (higher average powers indicated higher noise content).

[0101] ANN200 can have the ability to learn from pure experience and generate consistent representations. ANN200 can achieve promising robustness, defined as consistently representing input patterns it has never seen. For example, ANN200 has the ability to represent face images not in the training set, including unseen images corrupted by Gaussian noise or by occlusion. ANN200 can generate sparse and consistent representations of new faces. The representation of the corrupted input can be nearly identical to the representation of the clean image, even for images with extensive occlusions that are consistently represented. The specificity of ANN200 can be high for all noise levels and corruptions due to occlusions.

[0102] An ANN 200 trained on a particular set of images quickly learns a receptive field (in the representation, or second layer 220) that matches the images. For example, in an ANN 200 trained using codes from the world's languages, the similarity between the receptive field and the code increases quickly as the ANN 200 repeatedly encounters the same character. The specificity of the code representation increases even faster, reaching a learning plateau in fewer than 10 encounters. Thus, the ANN 200 effectively captures the structural features that are maximally informative about the input.

[0103] ANN 200 can learn to represent new input types without compromising its previous discriminatory capabilities. For example, ANN 200 can be trained to represent a fixed set of symbols and subsequently learn faces. Although learning faces after letters may change the receptive field properties of a subset of nodes, for ANN 200 the specificity of the symbol representations before and after learning a different input, such as a face, remained relatively high. ANN 200 can also maintain high specificity of the face representation (or vice versa). In other words, ANN 200 avoids the catastrophic forgetting problem encountered by many other neural network models. ANN 200 can learn from corrupted symbol images 130, such as those in which different parts of the pixels are inverted.

[0104] An ANN 200 can have any number of nodes in its primary layer 210 and its representation layer 220. For example, an ANN 200 can have 256 primary nodes and 500 representation nodes.

[0105] In one embodiment, the ANN 200 is constructed such that it can successfully distinguish between inputs prior to adaptation. The ANN 200 can be constructed in a number of ways to distinguish between inputs prior to adaptation. For example, the ANN 200 can use non-negative uniform connectivity, where the connection strengths between the primary layer input nodes 210a-d and the secondary layer representation nodes 220a-e are chosen as values ​​between 0 and 1. With non-negative uniform connectivity, there is the same probability that the connection strengths will achieve any value, i.e., the connection weights are drawn from a uniform distribution over (0,1). The weights can be normalized such that the length of the weight vector corresponding to any representation node is 1.

[0106] The ANN 200 may also be constructed using normal distribution connectivity, where the weights are derived from a normal distribution with mean 0 and standard deviation 1. The weights may also be normalized to have length 1.

[0107] The ANN 200 can also be constructed with uncorrelated connectivity, again where the weights are normalized to have length one. The uncorrelation can be based on the eigenvectors of the input variance-covariance matrix. In one embodiment, only 150 eigenvectors were utilized as the effective dimensionality of the input space because the variance of the input space along these vectors saturates after 150 dimensions. However, other numbers of eigenvectors can be used to create the input variance-covariance matrix.

[0108] The Frobenius norm of the difference between the correlation and identity matrices can be calculated and used to measure the difference between the two matrices. A lower Frobenius norm indicates better decorrelation. In one embodiment, the Frobenius norm of the difference between the correlation matrix and the identity matrix is ​​lowest for the decorrelation model of connectivity, indicating that it has best been able to decorrelate the nodes. When the input to the ANN 200 comprises 500 images 130, each image 130 can correspond to each of the 500 representation nodes, with each pixel in each image corresponding to each of the first-order nodes.

[0109] The ANN 200 can fit input sets of any number of images. For example, the ANN 200 can fit input sets containing 500, 800, or 1,000 inputs. Each input can be presented iteratively (e.g., up to 100 times) to allow for matching (e.g., using SGD) with inputs presented one at a time in succession (in the order of their randomly selected representations). Changes can be calculated in terms of the initial decorrelated connectivity, which can represent how strongly a particular node of the representation nodes 220a-e is connected to the first-layer nodes 210a-d. These connections can reflect the tuning properties of the representation nodes 220a-e, since an input node (one of the input nodes 210a-d) that is strongly connected to a representation node (of the representation nodes 220a-e) elicits the greatest response in that representation node. In one embodiment, different representation nodes 220a-e are tuned to different structures than the inputs. The distribution of cosine similarities of connectivity changes for different nodes across different states can be used to determine whether connectivity similarity was maintained during repeated encounters with the code. Sustained similarity levels indicate that the distinctiveness of node adjustments remained unchanged. These similarity levels can measure the overall connectivity changes in a particular state, but do not provide information about how connectivity changed for individual nodes across different states.

[0110] In one embodiment, the connectivity structure of the ANN 200 does not change for individual nodes, and the similarity of connectivity to the nodes increases slightly across states and then saturates, indicating that the connections to the individual representation nodes 220a-e change slightly as they repeatedly encounter inputs, reaching a plateau after a certain number of encounters. This can show how the achievement of such a plateau in the connectivity of the nodes eventually reaches saturation. This suggests that in one embodiment of the ANN 200, only the first few encounters of any input change the structure of the connectivity, and the representation of the input changes based on the ANN 200's direct experience, and then saturates. This saturation highlights an important difference between the ANN 200 framework and the classical efficient coding paradigm, where the representation of inputs depends on their overall statistics, not just on direct encounters.

[0111] For some embodiments, low average similarities (<0.5) were observed, indicating that the connections of different nodes changed differently. The average similarities remained consistently small and decreased slightly with state.

[0112] As the number of times ANN 200 encounters an input increases, the structure output by ANN 200 becomes more similar to the structure of the input. In one embodiment, ANN 200 successfully identifies a global, unique structure from the input by repeatedly encountering the same input. However, as the number of distinct inputs increases, the representation nodes 220a-e adjust to a more localized structure.

[0113] The cosine similarity between the change in connectivity and the inputs to the ANN 200 can be measured at different stages. In one embodiment, the similarity increased with the state of the network, but decreased as the number of inputs increased.

[0114] In one embodiment, the representation of the ANN 200 becomes sparser the more inputs it encounters. Furthermore, as the number of inputs increases, the response of the ANN 200 is limited to a smaller number of nodes. Representation efficiency can be quantified in three ways (correlation, kurtosis, and sparseness of the response profile) to highlight the changes that occur while adapting to a diverse number of inputs. These measures can be measured across different numbers of inputs as well as across different states of the ANN 200. In one embodiment, as the ANN 200 experiences more inputs, the representation node 220a-e responses become increasingly non-Gaussian. Increasing the number of input representations can also increase the kurtosis of the node response profile. Both experience and sampling of inputs can increase the representation efficiency of the ANN 200. Correlation between the representation nodes 220a-e can also decrease with more encounters with the same set of inputs (as indicated by the smaller Frobenius norm of the difference between the correlation and identity matrices, and by the L0 and L1 sparseness measures) along with encounters with new inputs. The response of the ANN 200 can become sparser with the number of inputs and with the fit state. Calculation of the node response profile kurtosis can evaluate the efficiency in terms of sparsity of the representation. The node response profile kurtosis increased with the number of inputs and with the ANN 200 network state. The correlation between the nodes can be measured and the Frobenius norm of the difference between the correlation and the identity matrix can be calculated. The norm can also decrease with the number of states and inputs, indicating a trend towards non-correlation. The sparsity of the representation can also show a similar trend for the ANN 200 according to an embodiment. Both the L0 and L1 sparsity measures can decrease with the ANN 200 network state while maintaining a level across the number of inputs. The performance of the ANN 200 according to an embodiment exceeds that obtained through known approaches such as matrix factorization, where the efficiency of the representation drops with increasing inputs.

[0115] The ANN 200 can generate representations that are consistent across all types of corruption and across different network states. For example, after experiencing five different inputs with corruption shapes, the representations are consistent across different shapes of corruption and across different states of the ANN 200. The specificity of the representations for different shapes of corruption can be calculated using the z-scored cosine similarity between the representations of the uncorrupted and corrupted inputs. The specificity can increase slightly with practice, i.e., after encountering the inputs more times for all shapes of corruption (high specificity of the representations has been observed with a slight increase at the 100th state of the network). The representation of the ANN 200 at the 100th state can be sparser than the representation of the ANN 200 at the 50th state. The specificity can decrease as the level of corruption, occlusion, or addition of noise increases. In one embodiment, the consistency of the representations increased as the representation nodes 220a-e of the ANN 200 became more specific by tuning to the unique features from the inputs. ANN 200 does not need to know the entire statistics of the input space to be efficient, and can generate consistent representations of the inputs in a wide variety of situations.

[0116] ANN200 can generalize inputs similarly when it sees different variations of the input. When it experiences a corrupted input (such as an input where 10%-20% of the pixels have changed), the connectivity changes in ANN200 can be very similar to the uncorrupted input, as in the case of adapting to an uncorrupted code. The similarity can vary from input to input, but the maximum similarity observed for any input to ANN200 is high. ANN200 can find and adapt to the consistency that existed across input variants, similar to complex deep or convolutional neural networks that have been shown to function in this way. However, unlike embodiments of ANN200 (including an embodiment with only two layers and learning from 800 examples), these other networks are very complex, include multiple layers, and require a large number of examples.

[0117] FIG. 3 is a diagram illustrating how inputs in an input sequence are adjusted at a representation layer for an ANN 200, according to one embodiment. A series of code images 310a-c can be input sequentially to the input layer input nodes 210a-d of the ANN 200 without delay. The ANN 200 can learn each code in the series of code images 310a-c and reconstruct the code from the output of the representation nodes 220a-e. Between inputs of each code image 310a-c to the ANN 200, the weights between the input nodes 210a-d and the representation nodes 220a-e, or between the representation nodes 220a-e, or both, can be updated. The ANN 200 does not experience catastrophic forgetting. Thus, as each code in the series 310a-c is input, the ANN 200 captures its characteristics and stores them as represented on the sequence of grids 320a-c. The fact that each code occupies its own square in grids 320a-c illustrates that ANN 200 can learn them sequentially without forgetting them. Code grid 330 represents a subset of the learned tuning properties of the representation. Code grid 330 shows that the most informative components of the input code 310 are captured by ANN 200.

[0118] 4 is a diagram illustrating how corrupted inputs contained in an input sequence may be learned by the representation layer 220 for the ANN 200, according to one embodiment. A set of corrupted code shapes 410, which may be generated, for example, by randomly flipping a percentage of the pixels (such as 10% or 20% of the pixels), are input to the input nodes 210a-d of the ANN 200. The set of corrupted code shapes 410 may include about 100 different corrupted shapes for each code. The conditioning characteristics 420 learned by the ANN 200 are clean versions of the input code shapes 410.

[0119] FIG. 5 is a diagram illustrating how the characteristics of an object input with various images are captured in the output of the ANN 200, according to one embodiment. Different object 3D models were rotated in x and y directions to generate different object images, here shown as an example of a human face object 510. A subset of images 520 from all objects can be selected and presented to the ANN 200. Sample alignment characteristics 530 can be learned by the ANN 200 and can include single and multiple image overlays. In this example, two groups of cells 540 emerge from the response of the ANN 200 to the input images 520. One group of cells 540a is specific to the identity of the object, while the other group of cells 540b is specific to the direction and angle of rotation. The output of cells 540a and 540b can be used to identify the object and its rotation, as shown in the columns of the output grid in FIG. 5C.

[0120] 6 is a diagram of a classification network 600 comprising a two-layer ANN connected to a classification layer, according to one embodiment. The first two layers of classification network 600 function similarly to the two layers of two-layer ANN 200 described above. Classification network 600 comprises a first layer of input nodes 610a-d (or first layer nodes), a second layer of identification nodes 620a-e (or representation or second layer nodes), and a third layer of classification nodes 630a-e (or third layer nodes). Nodes 630a-e in the classification layer can receive direct excitatory input from a single node in the identification layer (nodes 620a-e) while also receiving feedforward inhibition in parallel that mirrors the excitatory input from nodes in the input layer (input nodes 610a-d). Nodes in the classification layer 630a-e may also have recurrent excitatory connections and may receive a global inhibitory signal 640 that is imposed on all nodes in the classification layer 630a-e (which helps to limit spurious and / or runaway activity in this layer).

[0121] In one embodiment, the global inhibition 640 is a constant. The value of the global inhibition 640 can be any value that can prevent runaway behavior in the nodes 630a-e of the classification layer. For example, the global inhibition 640 can be a constant, such as 10. This value can be set based on the expected inputs to the classification nodes 630a-e. The excitatory connections between each of the nodes in the discrimination layer 620 and its corresponding node in the classification layer 630 can be a constant, such as 1. The inhibitory weights for the connections between the nodes in the input nodes 610a-d and the nodes in the classification nodes 630a-e can also be constants.

[0122] In some embodiments, the number of nodes in the identification layer 620a-e can be equal to the number of nodes in the classification layer 630a-e. In embodiments with fewer classification nodes 630 than identification nodes 620, the nodes in each layer can be associated with each other by grouping the nodes in each layer and associating those nodes with groups of nodes in the other layer. For example, in a classification network 600 with twice as many nodes in the identification layer 620 than in the classification layer 630, each node in the classification layer 630 can be connected to two nodes in the identification layer 620.

[0123] Learning in the classification network 600 can also be based on local learning rules. Learning for the first two layers (input layers 610a-d and discrimination layers 620a-e) can be achieved using the same techniques as described above for the two-layer ANN 200. Nodes in the third layer (classification layers 630a-e) are boosted when a node in the discrimination layer 620a-e and a node in the classification layer 630a-e are active simultaneously, or when two nodes in the classification layers 630a-e are active simultaneously. In one embodiment, the weights between nodes in the classification layers 630a-e and the input nodes 610a-d and the weights from global inhibition are not changed.

[0124] In one embodiment, the classification network 600 is designed using the principle of maximum dependent capture (MDC), which stipulates that each node (neuron) should capture the maximum information about distinct objects. To achieve this goal, the classification network 600 is designed to be able to distinguish between objects in its initial response. To achieve this, the weights between the input layer input nodes 610a-d and the discrimination layer nodes 620a-e are calibrated to allow distinct inputs to elicit essentially different responses without specific learning. In one embodiment, the initial bias in connectivity is set to minimize the chance of simultaneously coactivating any two of the classification nodes 630a-e, thereby maximizing the discrimination in the initial response of the classification network 600 to various inputs. For example, the connectivity matrix φ, which is a matrix of weights between each node in the input layer 610a-d and each node in the discrimination layer 620a-e, can be set such that the variance-covariance matrix of the response profile of the nodes in the representation layer matches the identity matrix.

[0125] In one embodiment, the nodes in the discrimination layer 620a-e can be modeled as leaky integrate-and-fire neurons with thresholding. For example, the nodes in the discrimination layer 620a-e can be modeled as

number

number

number

number

[0126] The dynamic response of the nodes in the classification layer 630a-e can function similarly to the nodes in the discrimination layer 620a-e, with two main differences. The input to each node in the classification layer (each of the classification nodes 630a-e) has two components: an excitatory input from a node in the discrimination layer 620a-e, and an inhibitory input from the input layer input nodes 610a-d (which can be a weighted inhibitory input from a single one of the input nodes 610a-d, or from some combination of the input nodes 610a-d). Furthermore, the inhibitory iterative connection matrix w is given by w class inhib From w class excit The classification layer w is equal to the one minus class The effective layer dynamics for the classification layers 630a-e are expressed by the equation

number

number

number

[0127] The classification network 600 is

number

number

number

number

[0128] In the classification network 600, the weights between nodes in the discrimination layer 620a-e and nodes in the classification layer 630a-e can be updated based on the activity of the two nodes involved. Recurrent excitatory connections between nodes in the classification layer 630a-e can be initially set to zero, while all nodes in this layer are subject to global inhibition. The weights can then be updated based on the sum of the potentiation effects between any pair of classification nodes 630a-e. For example, when two nodes are both coactive, the potentiation effect for their connection increases. Alternatively, when only one of the two nodes is active at a given time, the potentiation effect for their connection decreases. Finally, when both nodes remain inactive at a given time, the potentiation effect for their connection remains unchanged. The change in potentiation effect between any two nodes i and j of the classification nodes 630a-e, Δp ij is Δp for i=1 and j=1. ij =1, i=1 and j=0, or Δp ​​when i=0 and j=1 ij =-1, and Δp for i=0 and j=0 ij= 0. The connection weight between any two nodes in the classification layer (classification nodes 630a-e) is set to 1 if the sum of all the boost effects after encountering any number of inputs reaches a preset threshold. All other weights remain 0. The boost effect values ​​of all possible connections are reset to zero and the process of updating them starts again. Another way to express this update of weights is as in the equation

number

[0129] The representation function of the classification network 600 maximizes the differences between objects 135 and represents them distinctly. For classification, the classification network 600 can capture shared features, or classes, that distinguish objects 135 from different perspectives. In the classification network 600, distinguishing features of objects 135 of the same type can be linked together using mutual excitation and discriminated from similar features of other categories using inhibition. In vertebrate brains, recurrent excitation and widespread inhibition are widespread in the upper layers of sensory cortex. The design of the classification network 600 draws inspiration from these biological systems by adding a recurrent layer, the classification layer 630 (layer 3), to simulate these circuit motifs (characteristic substructures) and perform the computations for classification. Nodes in this layer receive direct excitatory inputs from the discrimination layer 620 (layer 2) in a columnar, one-to-one fashion. In parallel, the nodes receive feedforward inhibition that mirrors the excitatory inputs from the input layer 610. Nodes in the classification layer 630 may also have recurrent excitatory connections between each other, subject to a global inhibition imposed on all nodes in this layer. The connections between classification nodes 630a-e and discrimination nodes 620a-e may also be adaptive. For example, the learning rule is that when two excitatory nodes are both active, the connections become strong between those nodes (from discrimination to classification, and between classification neurons or nodes). There are no weight changes to connections to or from inhibitory neurons (or nodes).

[0130] This architectural configuration of the classification network 600 allows for the capture of class-specific features from the object 135. First, nodes in the classification layer 630 receive excitatory inputs from the discrimination layer 620 and feedforward inhibition relayed from the input layer 610. This combination conveys the difference between the updated excitatory output and the unupdated inhibitory output to inform the classification layer 630 of the features learned in the discrimination layer 620. Then, lateral excitatory connections between classification nodes 630a-e link correlated features that provide class information. Finally, global inhibition 640 ensures that only nodes that receive sufficient excitatory inputs can be active to reduce spurious and / or runaway activity. The result is that any of the classification nodes 630a-e with mutual excitation display attractor-like activity for class-specific features.

[0131] The classification capabilities of the classification network 600 exceed those of conventional approaches. For example, when classifying objects in the MNIST handwritten digits dataset, training with only 25% of the unlabeled samples results in the receptive field of the classification network 600 being similar to the digits in the discrimination layer 620. Furthermore, the population activity in the classification layer 630 of the classification network 600 exhibits high agreement for the same digit type while maintaining distinction between different classes. The classification network 600 can accurately identify 94% of the digit types when using pooled nodes from the most consistently active nodes of each group. Meanwhile, the most sophisticated existing network models currently achieve accuracies between 85-99%, but they all require some form of supervision. For example, self-supervised networks require digit labeling for initial training.

[0132] Similar to biological brains, the classification network 600 is robust in recognizing and classifying individual symbols, faces, and handwritten digits without being explicitly designed for these tasks. Specifically, in its identification layer 620, the classification network 600 can identify features that uniquely identify an object 135, and in the classification layer 630, it can link those features to form whole class-specific nodes. This last feature allows the classification network 600 to identify three-dimensional objects 135 from images that vary in size, position, and viewpoint. The problem of relating different images to extract the identity of an object is particularly difficult. Various other neural network models require highly sophisticated algorithms with deep convolutional layers and substantial supervision to achieve good performance. However, for the classification network 600, different images of the same object form image classes with shared features, which allows the classification network 600 to capture the shared features of the image classes without being ostensibly designed to do so. In other words, the classification network 600 can learn to consistently represent three-dimensional objects 135 that vary in size, position, and viewpoint.

[0133] The classification network 600 can identify objects 135 from various sizes and positions. For example, after experiencing several short clips of consecutive video frames of objects 135 from various positions and sizes, which may overlap in random clips (portions of film), covering less than 33% in total of the entire animation sequence, the classification network 600 can learn the overlap of specific images and different objects 135 in the input. When analyzing the entire animation sequence (much of which the classification network 600 has not experienced, i.e., less than 67% of all images), the representations of different frames are distinguished in the identification layer 620, and nodes are persistently active (for all objects 135) for most of the animation in the classification layer 630. The overall active nodes are specific to individual objects 135, even when the similarity between some of the objects 135 is high. For the classification network 600, in the representation domain, the overall similarity between images of the same object is significantly higher than the similarity between images of separate objects.

[0134] Generating a representation that is invariant to three-dimensional rotations is a difficult task for existing systems. However, when presented with an animation of a three-dimensional rotation sequence involving training the classification network 600 on a short clip of rotation along the vertical axis, the classification nodes 630a-e are able to show consistent responses to the same object 135 regardless of the representation angle. This is true even for models with highly irregular shapes. For example, for inputs of four-legged animals, the variations in the representations occurred at similar viewpoints, reflecting their common features. Overall, the similarity between different viewpoints of the same object is high, but for the classification network 600, the similarity between different objects is low. Thus, the classification network 600 can generate an invariant identity representation even when it has experienced less than one-third of all possible angles. Furthermore, the classification network 600 has the capability for invariant representations and does not need to encounter all possible variations to represent the object 135 consistently.

[0135] The identity of an object 135 is embedded in the structural relationships between its features. These relationships, or dependencies, can be exploited to encode the object's identity. The classification network 600 maximally captures those dependencies to identify the presence of the object 135 without requiring precise details of the input pattern. Here, the specific configuration of the classification network 600 allows for dependency capture to enable an invariant representation. This design accounts for the increasing complexity of receptive field properties along the visual pathway and is distinct from the hierarchical assembly models that later formed the basis of convolutional neural networks. These models assume that neurons in cognitive centers reproduce precise object details. However, precise object image reconstruction is not necessary for robust representations, and this deeply rooted assumption can create undesirable complexity in modeling object recognition. The classification network 600 does not calculate reconstruction errors to evaluate its learning performance. By capturing the dependencies that define the objects 135 and their classes, it is possible to generate remarkably consistent representations of the same object 135 across different conditions. Size, translation, and rotation invariance indicate that the classification network 600 can naturally link features that together define an object or its class, without being ostensibly designed to do so. It can enable nonlinear transformation of input signals into representational geometries that are suitable for identification and differentiation.

[0136] The classification network 600 illustrates how dependency capture can learn about objects 135 and stably represent them (in a manner similar to biological systems) through local and continuous changes at individual synapses. The two circuit architectures are based on known connectivity patterns. Both designs capture the feature dependencies that define objects 135 and classes, but their connections are different and serve different functions. The discrimination layer 620 makes the individual representations as discriminatory as possible. The classification layer 630 combines class-specific features to highlight and distinguish different object types. This two-pronged representation can produce perceptual distances that are not linearly related to distances in the input space.

[0137] Known networks show improved separation between projections of representations in their final layers, but fail to reproduce the projection rectification observed early in sensory processing in biological systems. For any manifold structure of the population response in the classification network 600, the low-dimensional manifold in the input layer 610 is jagged and occupies a convoluted subspace to rotate a three-dimensional object. The geometry is more organized in the classification layer 620, with some example objects occupying curved or jagged spaces. Nearly all samples are projected onto the rectified hyperplane in the classification layer 630, consistent with their invariant representation by the nodes. At lower curvatures, indicative of manifold rectification, the substantial linearization observed for all shapes of variations in the object 135, and the transformations performed by the classification network 600 to rectify the manifold, allow for perceptual invariance and robustness. This behavior is consistent with recent theories that propose that the geometry of manifolds becomes more separable along multiple sensory processing stages and is corrected at later steps to allow for invariant representation in biological systems.

[0138] The specificity of a representation evaluates how specific the representation of the input is. To estimate the specificity, the pairwise similarities between all representations of all objects are computed to obtain a similarity matrix S. Then, the z-score of the similarity of the input representation to all other representations is computed. In other words,

number

[0139] A power spectrum analysis can be performed to estimate the level of noise in the images 130 and their features learned by the classification network 600. Both the images 130 and the learned images can be Fourier transformed and their logarithms / powers calculated. The two-dimensional logarithms / powers of the images 130 and the learned structures can be radially averaged to obtain a one-dimensional power spectrum. The presence of noise is indicated by higher power in the higher frequencies of the spectrum. A comparison can be performed using the top 20% of frequencies.

[0140] The representations of the different images of a three-dimensional object in the classification layers 630a-e were composed of nodes that were consistently active for all images of the object. The overall consistency of the object representations in the classification layers 630a-e of the classification network 600 can be calculated. To calculate the consistency, the cosine similarity between the representations of successive images of the object 135 can be measured. The variation in the similarity is indicative of the consistency in the representations. Lower variation in the similarity measure means higher consistency and vice versa.

[0141] To estimate the geometry of the manifold structure, we can collect all images of all objects 135 in matrix I. Similarly, we can collect all images of all objects 135 in matrix R. d and R c The representations from the discrimination layers 620a-e and classification layers 630a-e in each can be collected. Principal component analysis can be performed on all three matrices separately, and on all images of individual objects plotted as projections onto the first two principal components. The plot shows a two-dimensional projection of the object manifold. To calculate the curvature of the two-dimensional projection of the manifold, three consecutive points p, p i+1 , and p i+2 The vector points p, p are selected. i+1 , and p i+2 The angle between is given by the equation

number

[0142] FIG. 7 is an illustration showing how the characteristics of an object 135 are captured in the output to the classification network 600, with various images input, according to one embodiment. The animation is rendered as movie frames showing size variation (SF) 730 and position variation (PF) 740. An example of different position variations 721a and 721b is shown for a car on a road in box 720. An example of size variation for a minivan (711a and 711b) is shown in box 710. A short sequence of these frames 730 and 740, where the size variation frames 730 and position variation frames 740 in total typically do not cover more than 33% of the entire sequence, can be randomly selected and fed to the classification network 600. In the identification layer 620, the classification network 600 can capture the complete object shape varying in size and position. Graph 750, comparing similarity scores between same and different objects, shows that the average similarity between representations of frames belonging to the same object (self) is significantly higher than the representation similarity between frames of distinct objects (others).

[0143] The input image 130 to the neural network architecture 140 may include any number of pixels, such as 100×100 pixels. The number of nodes in the identification layer 620 and the classification nodes 630 (when used) may vary. For example, the number of nodes in the identification layer 620 and the classification nodes 630 may vary depending on the number of pixels of the input to the neural network architecture 140. For example, when the input is 100×100 pixels, the number of nodes in the identification layer 620 may be 500 or 1,000. In one embodiment where the input image is 16×16 (in pixels), the size of the identification layer 620 may be 500 nodes. In one embodiment where the input image is 28×28 (in pixels), both the identification layer 620 and the classification layer 630 include 10,000 nodes. When the object image is 100×100 pixels, the size (in both the identification layer 620 and the classification layer 630) may be 1,000, 10,000, or more. Alternatively, the classification layer 630 and the identification layer 620 can have the same or more nodes as the input layer 610. For example, in classification network 600, the input layer 610 can have 784 nodes, and the identification layer 620 and the classification layer 630 can have 10,000 nodes each.

[0144] FIG. 8 illustrates a flow chart of an example method 800 according to certain embodiments. Method 800 is merely an example and is not limited to the embodiments presented herein. Method 800 may be employed in numerous different embodiments or examples not specifically shown or described herein. In some embodiments, the steps of method 800 may be performed in the order presented. In other embodiments, the operations of method 800 may be performed in any suitable order. In still other embodiments, one or more of the steps of method 800 may be combined or skipped. In numerous embodiments, system 100 and / or computer vision system 150 may be configured to perform method 800 and / or one or more of the steps of method 800. In these or other embodiments, one or more of the steps of method 800 may be implemented as one or more computer instructions configured to run on one or more processing devices 201 and configured to be stored on one or more non-transitory computer storage devices 202. Such a non-transitory memory storage device 202 can be part of a computer system, such as system 100 and / or computer vision system 150. The processing device 201 can be similar to or identical to the processing device 201 described above with respect to computer system 100 and / or computer vision system 150.

[0145] In step 810, weights between input layers of the neural network architecture and iterative weights between nodes in the representation layer are initialized. The manner in which the weights are initialized can vary. In one embodiment, the initial weights between nodes in the input layer and nodes in the representation layer can be calculated based on the eigenvectors of the variance-covariance matrix of the inputs. The weights of the connections between nodes in the representation layer are calculated using the formula S=-(W T It can be calculated using

[0146] In step 820, the images contained in the input sequence are input to the nodes of the input layer. In embodiments where the images are composed of pixels, each pixel may be input to a separate node. In other words, the number of input nodes is equal to the number of pixels in the images of the dataset being analyzed. In some embodiments, pixels are input to the input layer without preprocessing, thereby providing the input node with the value of that pixel. Alternatively, the images in the dataset may be preprocessed. For example, the values ​​of each image may be scaled in a range, such as by scaling all image values ​​to be within a range (such as from 0 to 1). A transform, such as a Fourier transform or a wavelet transform, may be performed on the images before inputting the image data to the nodes of the input layer.

[0147] In step 830, initial values ​​for the nodes included in the representation layer are calculated by multiplying the vector of values ​​of the nodes in the input layer in step 820 by a matrix of weights for the connections in the neural network architecture between the nodes in the input layer and the nodes in the representation layer. The first time step 830 is performed, these weights are the initial weights of the ANN, which were calculated in step 810. As additional images are iteratively processed, these weights are updated according to step 850 below.

[0148] In step 840, behavioral models for the nodes in the representation layer are applied to calculate values ​​for the nodes in the representation layer. Various types of behavioral models can be used, including models derived from biological neural networks. For example, the behavior of the nodes in the representation layer of an ANN can be modeled as "leaky integrate-and-fire" neurons. As part of step 840, values ​​from the recurrent connections between the nodes in the representation layer can be used to calculate the values ​​of the nodes in the representation layer. The calculation of the values ​​of the nodes can be performed iteratively until the value of each node reaches a steady state.

[0149] In embodiments in which the neural network architecture corresponds to a classification network having a third layer of nodes, the values ​​of the nodes in the classification layer can be updated by applying a process to the behavioral model as discussed in the paragraph above. For example, the initial values ​​of the nodes in the classification layer can be calculated for each node by summing a) the values ​​of the inputs (multiplied by the excitatory connection weights) from the node in the discrimination (or representation layer), b) the values ​​of the inputs (multiplied by the inhibitory connection weights) from the node in the input layer, and c) the value of the global inhibition applied to all nodes in the classification layer.

[0150] In a neural network architecture that has a classification layer, the number of times any two nodes in the classification layer are active together can be tracked over a given number of inputs. If the number of times any two nodes are active together exceeds a certain threshold, the weight between those nodes can be set to an excitatory value (such as 1). The weight of the connection between nodes in the classification layer that are typically not active together (determined by being below a threshold) can be set to 0.

[0151] In step 850, weights between nodes in the neural network architecture are updated. In one embodiment, the update of the weight matrix for the connections between nodes in the input layer and nodes in the representation layer is performed using a gradient descent approach. The recurrent weights in the representation layer are then updated according to the formula S=-(W T The weights between the nodes in the input layer and the nodes in the representation layer are updated using the weights (WI) between the nodes in the input layer and the nodes in the representation layer.

[0152] In step 860, it is determined whether there are more images in the dataset. If not, the method 800 ends. If so, the method 800 returns to step 820.

[0153] In step 870, the method 800 ends with the neural network architecture being tuned to the input image.

[0154] In some embodiments, the data input to the neural network architecture 140 is not image or visual data. For example, the data being analyzed can be DNA or RNA sequences, audio data, or other sensory data. This data can be "pixellated" or otherwise transformed so that it can be input to the input layer of the neural network architecture 140.

[0155] Neural network architecture 140 has advantages over other known neural networks. Neural network architecture 140 utilizes a learning algorithm that is fundamentally different from existing models and does not rely on error propagation. Neural network architecture 140 can also avoid the credit allocation problem in deep learning. Neural network architecture 140 can produce remarkable results that are comparable to much more complex networks with fewer nodes, fewer parameters, and no requirement on depth. Although this performance cannot match highly sophisticated deep learning models that rely on greater computing power, neural network architecture 140 can also be expanded into complex structures to perform additional tasks with improved performance. Given that it requires much fewer examples to learn and is much more energy efficient, neural network architecture 140 can match or surpass current alternatives.

[0156] As evident from the disclosure herein, the inventive techniques described in this disclosure are rooted in computer techniques that overcome existing problems in known computer vision systems, including problems dealing with extracting robust object representations from images and / or performing computer vision functions. The techniques described in this disclosure provide technical solutions (e.g., solutions that utilize various AI-based neural networking and machine learning techniques) to overcome limitations associated with known techniques. The technology-based solutions feature improvements over existing capabilities and functionality associated with computer vision and machine learning systems by improving the accuracy of computer vision (or machine learning) functions and reducing the information required to perform such functions. Furthermore, in certain embodiments, no storage of reference objects (such as faces or face objects) is required, which can serve to minimize storage requirements and avoid privacy issues. Furthermore, the neural network architectures disclosed herein are less complex and therefore less computationally expensive than other neural networks. Furthermore, no time- and resource-intensive creation and labeling of training set data is required.

[0157] Additionally, the neural network architecture described herein offers the added advantage of being fully interpretable (so-called white-box) and not prone to "catastrophic forgetting" commonly observed in neural networks. These results have profound implications for understanding how biological brains achieve invariant object representations and for developing efficient and robust biologically realistic intelligent networks.

[0158] In one embodiment, a system for extracting object representations from an image comprises one or more processing devices and one or more non-transitory computer readable storage devices storing computing instructions, the computing instructions being configured to execute on the one or more processing devices and to cause the one or more processing devices to perform functions comprising: receiving, at the computing device, an image comprising pixels; and generating, at the computing device, an object representation from the image using a two-layer neural network comprising an input layer of input nodes and a representation layer of representation nodes, where every input node is connected to every representation node through a first set of weighted connections having different values, and every representation node is connected to every other representation node through a second set of weighted connections having different values, and the input nodes of the input layer and the representation layer are connected to each other representation node through a second set of weighted connections having different values. a first set of connection weights associated with a first set of weighted connections between representation nodes in the current layer are selected to minimize the chance of two representation nodes in the representation layer being active simultaneously; a second set of connection weights for the second set of weighted connections are determined such that the weights between any two representation nodes in the representation layer are the same in both directions; input nodes in the input layer receive a first set of values, each of which is associated with one of the pixels of the image; a second set of values ​​for the representation nodes in the representation layer are calculated based at least in part on input received via the first set of weighted connections between the input nodes and the representation nodes and the second set of weighted connections between the representation nodes; and the second set of values ​​for the representation nodes in the representation layer are utilised to generate an object representation for the image.

[0159] In one embodiment, the first set of connection weights associated with the first set of weighted connections are calculated using estimates of eigenvectors of a variance-covariance matrix based on an input matrix created from a vector representation of the image.

[0160] In one embodiment, the learning mechanism continually updates the first set of connection weights as additional images are processed by the two-layer neural network.

[0161] In one embodiment, the learning mechanism includes stochastic gradient descent.

[0162] In one embodiment, the second set of values ​​for the representation nodes in the representation layer and the first set of values ​​for the input nodes in the input layer are all non-negative values.

[0163] In one embodiment, the second set of connection weights for the second set of weighted connections are continually updated based at least in part on changes in the first set of connection weights.

[0164] In one embodiment, the object representation includes data related to the object identification and data related to location information.

[0165] In one embodiment, the second set of weighted connections is inhibitory.

[0166] In one embodiment, the stochastic gradient descent method uses steps with a step size between 0 and 1.

[0167] In one embodiment, a method for extracting an object representation from an image, the method being configured to operate on one or more processing devices and implemented via execution of computing instructions configured to be stored on a non-transitory computer readable medium, comprises receiving, at the computing device, an image comprising pixels; and generating, at the computing device, the object representation from the image using a two-layer neural network comprising an input layer of input nodes and a representation layer of representation nodes, where every input node is connected to every representation node through a first set of weighted connections having different values, every representation node is connected to every other representation node through a second set of weighted connections having different values, and weighted connections between the input nodes of the input layer and the representation nodes of the representation layer. a first set of connection weights associated with the first set of weighted connections are selected to minimize the chance of two representation nodes in the representation layer being active at the same time; a second set of connection weights for the second set of weighted connections are determined such that the weights between any two representation nodes in the representation layer are the same in both directions; input nodes in the input layer receive a first set of values, each of which is associated with one of the pixels of the image; a second set of values ​​for the representation nodes in the representation layer are calculated based at least in part on input received via the first set of weighted connections between the input nodes and the representation nodes and the second set of weighted connections between the representation nodes; and the second set of values ​​for the representation nodes in the representation layer are utilized to generate an object representation for the image.

[0168] In one embodiment, the first set of connection weights associated with the first set of weighted connections are calculated using estimates of eigenvectors of a variance-covariance matrix based on an input matrix created from a vector representation of the image.

[0169] In one embodiment, the learning mechanism continually updates the first set of connection weights as additional images are processed by the two-layer neural network.

[0170] In one embodiment, the learning mechanism includes stochastic gradient descent.

[0171] In one embodiment, the second set of values ​​for the representation nodes in the representation layer and the first set of values ​​for the input nodes in the input layer are all non-negative values.

[0172] In one embodiment, the two-layer neural network includes more representation nodes in the representation layer than input nodes in the input layer.

[0173] In one embodiment, the second set of connection weights for the second set of weighted connections are continually updated based at least in part on changes in the first set of connection weights.

[0174] In one embodiment, the object representation includes data related to the object identification and data related to location information.

[0175] In one embodiment, the second set of weighted connections is inhibitory.

[0176] In one embodiment, a computer program product for extracting an object representation from an image comprises a non-transitory computer readable medium having instructions that cause a computing device to receive, at the computing device, an image comprising pixels; and generate, at the computing device, an object representation from the image using a two-layer neural network having an input layer of input nodes and a representation layer of representation nodes, where every input node is connected to every representation node through a first set of weighted connections having different values, and every representation node is connected to every other representation node through a second set of weighted connections having different values, and where the first set of weighted connections between the input nodes of the input layer and the representation nodes of the representation layer are associated with the first set of weighted connections. a first set of connection weights for the second set of weighted connections are selected to minimize the chance of two representation nodes in the representation layer being active at the same time; a second set of connection weights for the second set of weighted connections are determined such that the weights between any two representation nodes in the representation layer are the same in both directions; input nodes in the input layer receive a first set of values, each of which relates to one of the pixels of the image; a second set of values ​​for the representation nodes in the representation layer are calculated based at least in part on input received via the first set of weighted connections between the input nodes and the representation nodes and the second set of weighted connections between the representation nodes; and the second set of values ​​for the representation nodes in the representation layer are utilised to generate an object representation for the image.

[0177] In one embodiment, the first set of connection weights associated with the first set of weighted connections are calculated using estimates of eigenvectors of a variance-covariance matrix based on an input matrix created from a vector representation of the image.

[0178] In one embodiment, a system for classifying object representations from an image comprises one or more processing devices and one or more non-transitory computer readable storage devices storing computing instructions, the computing instructions being configured to execute on the one or more processing devices and to cause the one or more processing devices to perform functions comprising receiving, at the computing device, an image comprising pixels; and generating, at the computing device, classification data for one or more objects in the image using a three-layer neural network comprising: i) an input layer comprising input nodes; ii) a representation layer comprising representation nodes; and iii) a classification layer comprising classification nodes, wherein every input node is connected to every representation node through a first set of weighted connections having different values, and every representation node is connected to every other representation node through a second set of weighted connections having different values, and wherein a first set of connection weights associated with the first set of weighted connections between the input nodes of the input layer and the representation nodes of the representation layer are configured to minimize the number of times that two representation nodes in the representation layer are simultaneously active. the second set of connection weights for the second set of weighted connections are determined such that the connection weights between any two representation nodes in the representation layer are the same in both directions; the classification nodes of the classification layer are connected to the representation nodes of the representation layer in a one-to-one excitatory manner and to the input nodes of the input layer in a one-to-one inhibitory manner; the classification nodes of the classification layer are connected to each other through a third set of weighted connections such that the connection weights between any two classification nodes in the classification layer are the same in both directions; the classification nodes of the classification layer receive global inhibitory inputs; the input nodes of the input layer receive a first set of values, each of which is associated with one of the pixels of the image; the second set of values ​​for the representation nodes in the representation layer are calculated based at least in part on the inputs received via the first set of weighted connections between the input nodes and the representation nodes and the second set of weighted connections between the representation nodes; and the third set of values ​​for the classification nodes in the classification layer are calculated based at least in part on the inputs received by the classification nodes from the input nodes, the representation nodes, and other classification nodes.Classification data for one or more objects in the image is generated based at least in part on the third set of values.

[0179] In one embodiment, the first set of connection weights associated with the first set of weighted connections are calculated using estimates of eigenvectors of a variance-covariance matrix based on an input matrix created from a vector representation of the image.

[0180] In one embodiment, the learning mechanism continually updates the first set of connection weights as additional images are processed by the three-layer neural network.

[0181] In one embodiment, the learning mechanism includes stochastic gradient descent.

[0182] In one embodiment, the third set of values ​​for the classification nodes in the classification layer, the second set of values ​​for the representation nodes in the representation layer, and the first set of values ​​for the input nodes in the input layer are all non-negative values.

[0183] In one embodiment, the second set of connection weights for the second set of weighted connections are continually updated based at least in part on changes in the first set of connection weights.

[0184] In one embodiment, the classification data comprises identification data associated with at least one object in the image.

[0185] In one embodiment, the second set of weighted connections is inhibitory.

[0186] In one embodiment, the stochastic gradient descent method uses steps with a step size between 0 and 1.

[0187] In one embodiment, a method for classifying object representations from an image, the method being implemented via execution of computing instructions configured to operate on one or more processing devices and configured to be stored on a non-transitory computer readable medium, includes receiving, at the computing device, an image comprising pixels; and generating, at the computing device, classification data for one or more objects in the image using a three-layer neural network comprising: i) an input layer comprising input nodes; ii) a representation layer comprising representation nodes; and iii) a classification layer comprising classification nodes, the three-layer neural network comprising: i) an input layer comprising input nodes; ii) a representation layer comprising representation nodes; and iii) a classification layer comprising classification nodes, the three-layer neural network comprising: i) an input layer comprising input nodes; ii) a representation layer comprising representation nodes; and iii) a classification layer comprising classification nodes, the three-layer neural network comprising: i) an input layer comprising input nodes; a second set of connection weights for the second set are determined such that the connection weights between any two representation nodes in the representation layer are the same in both directions; the classification nodes of the classification layer are connected to the identification nodes of the classification layer in a one-to-one excitatory manner and to the input nodes of the input layer in a one-to-one inhibitory manner; the classification nodes of the classification layer are connected to each other through a third set of weighted connections such that the connection weights between any two classification nodes in the classification layer are the same in both directions; the classification nodes of the classification layer receive global inhibitory inputs; the input nodes of the input layer receive a first set of values, each of which is associated with one of the pixels of the image; the second set of values ​​for the representation nodes in the representation layer are calculated based at least in part on the inputs received via the first set of weighted connections between the input nodes and the representation nodes and the second set of weighted connections between the representation nodes; and the third set of values ​​for the classification nodes in the classification layer are calculated based at least in part on inputs received by the classification nodes from the input nodes, the representation nodes, and other classification nodes;based at least in part on a third set of values.

[0188] In one embodiment, the first set of connection weights associated with the first set of weighted connections are calculated using estimates of eigenvectors of a variance-covariance matrix based on an input matrix created from a vector representation of the image.

[0189] In one embodiment, the learning mechanism continually updates the first set of connection weights as additional images are processed by the three-layer neural network.

[0190] In one embodiment, the learning mechanism includes stochastic gradient descent.

[0191] In one embodiment, the third set of values ​​for the classification nodes in the classification layer, the second set of values ​​for the representation nodes in the representation layer, and the first set of values ​​for the input nodes in the input layer are all non-negative values.

[0192] In one embodiment, the second set of connection weights for the second set of weighted connections are continually updated based at least in part on changes in the first set of connection weights.

[0193] In one embodiment, the classification data comprises identification data associated with at least one object in the image.

[0194] In one embodiment, the second set of weighted connections is inhibitory.

[0195] In one embodiment, the stochastic gradient descent method uses steps with a step size between 0 and 1.

[0196] In one embodiment, a computer program product for classifying object representations from an image comprises a non-transitory computer readable medium containing instructions to cause a computing device to receive, at the computing device, an image comprising pixels; and generate, at the computing device, classification data for one or more objects in the image using a three-layer neural network comprising: i) an input layer comprising input nodes; ii) a representation layer comprising representation nodes; and iii) a classification layer comprising classification nodes, wherein every input node is connected to every representation node through a first set of weighted connections having distinct values, and every representation node is connected to every other representation node through a second set of weighted connections having distinct values, a first set of connection weights associated with the first set of weighted connections between the input nodes of the input layer and the representation nodes of the representation layer selected to minimize a chance that two representation nodes in the representation layer are active at the same time, and a second set of connection weights for the second set of weighted connections selected to minimize a chance that any representation nodes in the representation layer are active at the same time. the connection weights between two representation nodes are determined to be the same in both directions, the classification nodes of the classification layer are connected to the identification nodes of the identification layer in a one-to-one excitatory manner and to the input nodes of the input layer in a one-to-one inhibitory manner, the classification nodes of the classification layer are connected to each other through a third set of weighted connections such that the connection weights between any two classification nodes in the classification layer are the same in both directions, the classification nodes of the classification layer receive global inhibitory inputs, the input nodes of the input layer receive a first set of values, each of which is associated with one of the pixels of the image, a second set of values ​​for the representation nodes in the representation layer are calculated based at least in part on the inputs received via the first set of weighted connections between the input nodes and the representation nodes and the second set of weighted connections between the representation nodes, a third set of values ​​for the classification nodes in the classification layer are calculated based at least in part on inputs received by the classification nodes from the input nodes, the representation nodes, and other classification nodes, and classification data for one or more objects in the image is generated based at least in part on the third set of values.

[0197] In one embodiment, the first set of connection weights associated with the first set of weighted connections are calculated using estimates of eigenvectors of a variance-covariance matrix based on an input matrix created from a vector representation of the image.

[0198] An embodiment may include a computer program product accessible from a computer usable or computer readable medium providing program code for use by or in association with a computer or any instruction execution system. A computer usable or computer readable medium may include any apparatus that stores, communicates, propagates, or ports a program for use by or in association with an instruction execution system, apparatus, or device. The medium may be a magnetic, optical, electronic, electromagnetic, infrared, or semiconductor system (or apparatus or device), or a propagation medium. The medium may include a semiconductor or solid state memory, a magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid magnetic disk, a magneto-optical disk, or the like.

[0199] A data processing system suitable for storing and / or executing program code may include at least one processor coupled directly or indirectly to memory elements through a system bus. The memory elements may include local memory employed during the actual execution of the program code, bulk storage, and cache memory that provides temporary storage of at least some program code to reduce retrievals of the code from bulk storage during execution. Input / output or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) may be coupled to the system either directly or through intervening I / O controllers.

[0200] Network adapters may also be coupled to the system to enable the processing system to become coupled to other data processing systems or remote printers or storage devices through intervening private or public networks. Modems, cable modems, and Ethernet cards are just a few of the currently available types of network adapters.

[0201] Although various novel features of the invention have been shown, discussed, and described as applied to specific embodiments thereof, it should be understood that various omissions, substitutions, and changes in the form and details of the systems and methods described and illustrated may be made by those skilled in the art without departing from the spirit of the invention. In particular, steps in the methods may be performed in different orders in many cases, where appropriate. Those skilled in the art will recognize, based on the above disclosure and an understanding of the teachings of the invention, that the specific hardware and devices that are part of the systems described herein, and the general functions provided and incorporated herein, may vary in different embodiments of the invention. Thus, the description of the system components is for illustrative purposes to facilitate a full and complete understanding and appreciation of the various aspects and functions of specific embodiments of the invention, as implemented in the embodiments of the systems and methods of the invention. Those skilled in the art will recognize that the invention can be practiced in other than the described embodiments, which are presented for illustrative purposes and are not limiting. The above-described embodiment can also be described as follows, but is not limited to the following. [Configuration 1] 1. A system for extracting object representations from an image, comprising: one or more processing units; one or more non-transitory computer readable storage devices storing operational instructions, the operational instructions being executable on the one or more processing units, the one or more processing units being configured to: receiving, at a computing device, an image comprising pixels; configured to cause, at the computing device, to perform a function comprising generating object representations from the images using a two-layer neural network having an input layer of input nodes and a representation layer of representation nodes; every input node is connected to every expression node through a first set of weighted connections having different values, and every expression node is connected to every other expression node through a second set of weighted connections having different values; a first set of connection weights associated with the first set of weighted connections between the input nodes of the input layer and the representation nodes of the representation layer are selected to minimize the likelihood that two representation nodes in the representation layer are simultaneously active; a second set of connection weights for the second set of weighted connections are determined such that the weights between any two representation nodes in the representation layer are the same in both directions; the input nodes of the input layer receive a first set of values, each of the values ​​associated with one of the pixels of the image; a second set of values ​​for the expression nodes in the representation layer are calculated based at least in part on inputs received via the first set of weighted connections between the input nodes and the expression nodes and the second set of weighted connections between the expression nodes; The second set of values ​​for the representation nodes in the representation layer are utilized to generate the object representation for the image. [Configuration 2] The system of configuration 1, wherein the first set of connection weights associated with the first set of weighted connections are calculated using estimates of eigenvectors of a variance-covariance matrix based on an input matrix created from a vector representation of the image. [Configuration 3] 2. The system of claim 1, wherein the learning mechanism continually updates the first set of connection weights as additional images are processed by the two-layer neural network. [Configuration 4] The system of configuration 3, wherein the learning mechanism includes a stochastic gradient descent method. [Configuration 5] The system of configuration 1, wherein the second set of values ​​for the representation nodes in the representation layer and the first set of values ​​for the input nodes in the input layer are all non-negative values. [Configuration 6] 2. The system of claim 1, wherein the second set of connection weights for the second set of weighted connections are continually updated based at least in part on changes in the first set of connection weights. [Configuration 7] 2. The system of claim 1, wherein the object representation includes data related to object identification and data related to location information. [Configuration 8] 2. The system of claim 1, wherein the second set of weighted connections are suppressive. [Configuration 9] 5. The system of claim 4, wherein the stochastic gradient descent method uses steps with a step size between 0 and 1. [Configuration 10] 1. A method for extracting object representations from an image, the method being implemented through execution of operational instructions configured to operate on one or more processing devices and configured to be stored on a non-transitory computer readable medium, the method comprising: receiving, at a computing device, an image comprising pixels; generating, in the computing device, object representations from the images using a two-layer neural network having an input layer of input nodes and a representation layer of representation nodes; every input node is connected to every expression node through a first set of weighted connections having different values, and every expression node is connected to every other expression node through a second set of weighted connections having different values; a first set of connection weights associated with the first set of weighted connections between the input nodes of the input layer and the representation nodes of the representation layer are selected to minimize the likelihood that two representation nodes in the representation layer are simultaneously active; a second set of connection weights for the second set of weighted connections are determined such that the weights between any two representation nodes in the representation layer are the same in both directions; the input nodes of the input layer receive a first set of values, each of the values ​​associated with one of the pixels of the image; a second set of values ​​for the expression nodes in the representation layer are calculated based at least in part on inputs received via the first set of weighted connections between the input nodes and the expression nodes and the second set of weighted connections between the expression nodes; The method of claim 1, wherein the second set of values ​​for the representation nodes in the representation layer are utilized to generate the object representation for the image. [Configuration 11] 11. The method of claim 10, wherein the first set of connection weights associated with the first set of weighted connections are calculated using estimates of eigenvectors of a variance-covariance matrix based on an input matrix created from a vector representation of the image. [Configuration 12] 11. The method of claim 10, wherein the learning mechanism continually updates the first set of connection weights as additional images are processed by the two-layer neural network. [Configuration 13] 13. The method of claim 12, wherein the learning mechanism includes a stochastic gradient descent method. [Configuration 14] 11. The method of claim 10, wherein the second set of values ​​for the representation nodes in the representation layer and the first set of values ​​for the input nodes in the input layer are all non-negative values. [Configuration 15] 11. The method of claim 10, wherein the two-layer neural network includes more representation nodes in the representation layer than input nodes in the input layer. [Configuration 16] 11. The method of claim 10, wherein the second set of connection weights for the second set of weighted connections are continually updated based at least in part on changes in the first set of connection weights. [Configuration 17] 11. The method of claim 10, wherein the object representation includes data related to object identification and data related to location information. [Configuration 18] 11. The method of claim 10, wherein the second set of weighted connections are suppressive. [Configuration 19] 1. A computer program product for extracting object representations from an image, comprising: a non-transitory computer-readable medium containing instructions that are configured to cause a computing device to: receiving, at a computing device, an image having pixels; generating, in the computing device, object representations from the images using a two-layer neural network having an input layer of input nodes and a representation layer of representation nodes; every input node is connected to every expression node through a first set of weighted connections having different values, and every expression node is connected to every other expression node through a second set of weighted connections having different values; a first set of connection weights associated with the first set of weighted connections between the input nodes of the input layer and the representation nodes of the representation layer are selected to minimize the likelihood that two representation nodes in the representation layer are simultaneously active; a second set of connection weights for the second set of weighted connections are determined such that the weights between any two representation nodes in the representation layer are the same in both directions; the input nodes of the input layer receive a first set of values, each of the values ​​associated with one of the pixels of the image; a second set of values ​​for the expression nodes in the representation layer are calculated based at least in part on inputs received via the first set of weighted connections between the input nodes and the expression nodes and the second set of weighted connections between the expression nodes; the second set of values ​​for the representation nodes in the representation layer are utilized to generate the object representation for the image. [Configuration 20] 20. The computer program product of claim 19, wherein the first set of connection weights associated with the first set of weighted connections are calculated using estimates of eigenvectors of a variance-covariance matrix based on an input matrix created from a vector representation of the image. [Configuration 21] 1. A system for classifying object representations from images, comprising: one or more processing units; one or more non-transitory computer readable storage devices storing operational instructions, the operational instructions being executable on the one or more processing units, the one or more processing units being configured to: receiving, at a computing device, an image comprising pixels; The computing device is configured to perform a function comprising generating classification data for one or more objects in the image using a three-layer neural network comprising: i) an input layer comprising input nodes; ii) a representation layer comprising representation nodes; and iii) a classification layer comprising classification nodes; every input node is connected to every expression node through a first set of weighted connections having different values, and every expression node is connected to every other expression node through a second set of weighted connections having different values; a first set of connection weights associated with the first set of weighted connections between the input nodes of the input layer and the representation nodes of the representation layer are selected to minimize the likelihood that two representation nodes in the representation layer are simultaneously active; a second set of connection weights for the second set of weighted connections are determined such that the connection weights between any two representation nodes in the representation layer are the same in both directions; the classification nodes of the classification layer are connected in a one-to-one excitatory manner to the representation nodes of the representation layer and in a one-to-one inhibitory manner to the input nodes of the input layer; the classification nodes of the classification layer are connected to each other through a third set of weighted connections such that the connection weight between any two classification nodes in the classification layer is the same in both directions; the classification nodes of the classification layer receive global inhibitory inputs; the input nodes of the input layer receive a first set of values, each of the values ​​associated with one of the pixels of the image; a second set of values ​​for the expression nodes in the representation layer are calculated based at least in part on inputs received via the first set of weighted connections between the input nodes and the expression nodes and the second set of weighted connections between the expression nodes; a third set of values ​​for the classification nodes in the classification layer are calculated based at least in part on inputs received by the classification nodes from the input nodes, the representation nodes, and other classification nodes; The classification data for one or more objects in the image is generated based at least in part on the third set of values. [Configuration 22] 22. The system of claim 21, wherein the first set of connection weights associated with the first set of weighted connections are calculated using estimates of eigenvectors of a variance-covariance matrix based on an input matrix created from a vector representation of the image. [Configuration 23] 22. The system of embodiment 21, wherein the learning mechanism continually updates the first set of connection weights as additional images are processed by the three-layer neural network. [Configuration 24] 24. The system of claim 23, wherein the learning mechanism includes a stochastic gradient descent method. [Configuration 25] The system of configuration 21, wherein the third set of values ​​for the classification nodes in the classification layer, the second set of values ​​for the representation nodes in the representation layer, and the first set of values ​​for the input nodes in the input layer are all non-negative values. [Configuration 26] 22. The system of claim 21, wherein the second set of connection weights for the second set of weighted connections are continually updated based at least in part on changes in the first set of connection weights. [Configuration 27] 24. The system of claim 23, wherein the classification data comprises identification data associated with at least one object in the image. [Configuration 28] 22. The system of embodiment 21, wherein the second set of weighted connections are suppressive. [Configuration 29] 25. The system of claim 24, wherein the stochastic gradient descent method uses steps with a step size between 0 and 1. [Configuration 30] 1. A method for classifying object representations from an image, the method being accomplished through execution of operational instructions configured to operate on one or more processing devices and configured to be stored on a non-transitory computer readable medium, the method comprising: receiving, at a computing device, an image comprising pixels; generating classification data for one or more objects in the image using a three-layer neural network, the three-layer neural network comprising: i) an input layer comprising input nodes; ii) a representation layer comprising representation nodes; and iii) a classification layer comprising classification nodes; every input node is connected to every expression node through a first set of weighted connections having different values, and every expression node is connected to every other expression node through a second set of weighted connections having different values; a first set of connection weights associated with the first set of weighted connections between the input nodes of the input layer and the representation nodes of the representation layer are selected to minimize the likelihood that two representation nodes in the representation layer are simultaneously active; a second set of connection weights for the second set of weighted connections are determined such that the connection weights between any two representation nodes in the representation layer are the same in both directions; the classification nodes of the classification layer are connected in a one-to-one excitatory manner to the discrimination nodes of the discrimination layer and in a one-to-one inhibitory manner to the input nodes of the input layer; the classification nodes of the classification layer are connected to each other through a third set of weighted connections such that the connection weight between any two classification nodes in the classification layer is the same in both directions; the classification nodes of the classification layer receive global inhibitory inputs; the input nodes of the input layer receive a first set of values, each of the values ​​associated with one of the pixels of the image; a second set of values ​​for the expression nodes in the representation layer are calculated based at least in part on inputs received via the first set of weighted connections between the input nodes and the expression nodes and the second set of weighted connections between the expression nodes; a third set of values ​​for the classification nodes in the classification layer are calculated based at least in part on inputs received by the classification nodes from the input nodes, the representation nodes, and other classification nodes; wherein the classification data for the one or more objects in the image is generated based at least in part on the third set of values. [Configuration 31] 31. The method of claim 30, wherein the first set of connection weights associated with the first set of weighted connections are calculated using estimates of eigenvectors of a variance-covariance matrix based on an input matrix created from a vector representation of the image. [Configuration 32] 32. The method of claim 31, wherein the learning mechanism continually updates the first set of connection weights as additional images are processed by the three-layer neural network. [Configuration 33] 33. The method of claim 32, wherein the learning mechanism includes a stochastic gradient descent method. [Configuration 34] 31. The method of claim 30, wherein the third set of values ​​for the classification nodes in the classification layer, the second set of values ​​for the representation nodes in the representation layer, and the first set of values ​​for the input nodes in the input layer are all non-negative values. [Configuration 35] 31. The method of claim 30, wherein the second set of connection weights for the second set of weighted connections are continually updated based at least in part on changes in the first set of connection weights. [Configuration 36] 33. The method of claim 32, wherein the classification data comprises identification data associated with at least one object in the image. [Configuration 37] 31. The method of claim 30, wherein the second set of weighted connections are suppressive. [Configuration 38] 34. The method of claim 33, wherein the stochastic gradient descent method uses steps with a step size between 0 and 1. [Configuration 39] 11. A computer program product for classifying object representations from images, comprising: a non-transitory computer-readable medium containing instructions that cause a computing device to: receiving, at a computing device, an image having pixels; generating classification data for one or more objects in the image using a three-layer neural network, the three-layer neural network comprising: i) an input layer comprising input nodes; ii) a representation layer comprising representation nodes; and iii) a classification layer comprising classification nodes; every input node is connected to every expression node through a first set of weighted connections having different values, and every expression node is connected to every other expression node through a second set of weighted connections having different values; a first set of connection weights associated with the first set of weighted connections between the input nodes of the input layer and the representation nodes of the representation layer are selected to minimize the likelihood that two representation nodes in the representation layer are simultaneously active; a second set of connection weights for the second set of weighted connections are determined such that the connection weights between any two representation nodes in the representation layer are the same in both directions; the classification nodes of the classification layer are connected in a one-to-one excitatory manner to the discrimination nodes of the discrimination layer and in a one-to-one inhibitory manner to the input nodes of the input layer; the classification nodes of the classification layer are connected to each other through a third set of weighted connections such that the connection weight between any two classification nodes in the classification layer is the same in both directions; the classification nodes of the classification layer receive global inhibitory inputs; the input nodes of the input layer receive a first set of values, each of the values ​​associated with one of the pixels of the image; a second set of values ​​for the expression nodes in the representation layer are calculated based at least in part on inputs received via the first set of weighted connections between the input nodes and the expression nodes and the second set of weighted connections between the expression nodes; a third set of values ​​for the classification nodes in the classification layer are calculated based at least in part on inputs received by the classification nodes from the input nodes, the representation nodes, and other classification nodes; the classification data for the one or more objects in the image is generated based at least in part on the third set of values. [Configuration 40] 40. The computer program product of claim 39, wherein the first set of connection weights associated with the first set of weighted connections are calculated using estimates of eigenvectors of a variance-covariance matrix based on an input matrix created from a vector representation of the image.

Claims

1. 1. A system for extracting an object representation from an input, comprising: one or more processing devices; One or more non-transitory computer readable storage devices storing operational instructions, the operational instructions being executable on the one or more processing units, causing the one or more processing units to: receiving, at a computing device, an input comprising pixelated information; generating, in the computing device, an object representation from the input using a two-layer neural network having an input layer of input nodes and a representation layer of representation nodes; one or more non-transitory computer readable storage devices configured to cause the device to perform functions comprising: every input node is connected to every expression node through a first set of weighted connections having different values, and every expression node is connected to every other expression node through a second set of weighted connections having different values; a weight matrix storing connection weights corresponding to the first set of weighted connections between the input nodes of the input layer and the representation nodes of the representation layer; When both the input node and the expression node are active, the connection weights stored in the weight matrix are strengthened; in response to detecting that two expression nodes are coactive, the connection weights between the input nodes to both expression nodes are weakened; the input nodes of the input layer receive a first set of values ​​corresponding to the pixelated information of the input; a second set of values ​​for the expression nodes in the representation layer is calculated based at least in part on inputs received via (i) the first set of weighted connections between the input nodes and the expression nodes, and (ii) the second set of weighted connections between the expression nodes; The system according to claim 1, wherein the second set of values ​​for the representation nodes in the representation layer is utilized to generate the object representation for the input.

2. a first set of connection weights associated with the first set of weighted connections between the input nodes of the input layer and the representation nodes of the representation layer are selected to minimize the likelihood that two representation nodes in the representation layer are simultaneously active; 2. The system of claim 1 , wherein the first set of connection weights associated with the first set of weighted connections are initially calculated using estimates of eigenvectors of a variance-covariance matrix based on an input matrix created from a vector representation of a selected set of inputs, and the connection weights in the weight matrix are updated as additional inputs are received.

3. 2. The system of claim 1, wherein a learning mechanism continually updates the first set of connection weights as additional inputs are processed by the two-layer neural network.

4. 4. The system of claim 3, wherein the learning mechanism includes a stochastic gradient descent method.

5. 2. The system of claim 1, wherein the second set of values ​​for the representation nodes in the representation layer and the first set of values ​​for the input nodes in the input layer are all non-negative values.

6. a second set of connection weights for the second set of weighted connections are determined such that the weights between any two representation nodes in the representation layer are the same in both directions; in response to detecting that two expression nodes are coactive, the connection weight between the two expression nodes is strengthened; 2. The system of claim 1, wherein the second set of connection weights for the second set of weighted connections are continually updated based at least in part on changes in the first set of connection weights.

7. 2. The system of claim 1, wherein the object representation includes data related to an object identification and data related to location information.

8. 2. The system of claim 1, wherein the second set of weighted connections are inhibitory.

9. 5. The system of claim 4, wherein the stochastic gradient descent uses steps with a step size between 0 and 1.

10. 1. A method for extracting an object representation from an input, the method being configured to operate on one or more processing devices and implemented through execution of operational instructions stored on a non-transitory computer readable medium, the method comprising: receiving, at a computing device, an input comprising pixelated information; generating, at the computing device, an object representation from the input using a two-layer neural network having an input layer of input nodes and a representation layer of representation nodes; every input node is connected to every expression node through a first set of weighted connections having different values, and every expression node is connected to every other expression node through a second set of weighted connections having different values; a weight matrix storing connection weights corresponding to the first set of weighted connections between the input nodes of the input layer and the representation nodes of the representation layer; When both the input node and the expression node are active, the connection weights stored in the weight matrix are strengthened; in response to detecting that two expression nodes are coactive, the connection weights between the input nodes to both expression nodes are weakened; the input nodes of the input layer receive a first set of values ​​corresponding to the pixelated information of the input; a second set of values ​​for the expression nodes in the representation layer is calculated based at least in part on inputs received via (i) the first set of weighted connections between the input nodes and the expression nodes, and (ii) the second set of weighted connections between the expression nodes; The method of claim 1, wherein the second set of values ​​for the representation nodes in the representation layer is utilized to generate the object representation for the input.

11. a first set of connection weights associated with the first set of weighted connections between the input nodes of the input layer and the representation nodes of the representation layer are selected to minimize the likelihood that two representation nodes in the representation layer are simultaneously active; 11. The method of claim 10, wherein the first set of connection weights associated with the first set of weighted connections are initially calculated using estimates of eigenvectors of a variance-covariance matrix based on an input matrix created from a vector representation of a selected set of inputs, and the connection weights in the weight matrix are updated as additional inputs are received.

12. 11. The method of claim 10, wherein a learning mechanism continually updates the first set of connection weights as additional inputs are processed by the two-layer neural network.

13. 13. The method of claim 12, wherein the learning mechanism includes stochastic gradient descent.

14. 11. The method of claim 10, wherein the second set of values ​​for the representation nodes in the representation layer and the first set of values ​​for the input nodes in the input layer are all non-negative values.

15. 11. The method of claim 10, wherein the two-layer neural network includes more representation nodes in the representation layer than input nodes in the input layer.

16. a second set of connection weights for the second set of weighted connections are determined such that the weights between any two representation nodes in the representation layer are the same in both directions; in response to detecting that two expression nodes are coactive, the connection weight between the two expression nodes is strengthened; 11. The method of claim 10, wherein the second set of connection weights for the second set of weighted connections are continually updated based at least in part on changes in the first set of connection weights.

17. 11. The method of claim 10, wherein the object representation includes data related to object identification and data related to location information.

18. 11. The method of claim 10, wherein the second set of weighted connections are inhibitory.

19. 1. A non-transitory computer readable medium storing a computer program for extracting an object representation from an input, the computer program being configured to cause a computing device to: receiving, at a computing device, an input comprising pixelated information; instructions for generating, in the computing device, an object representation from the input using a two-layer neural network having an input layer of input nodes and a representation layer of representation nodes; every input node is connected to every expression node through a first set of weighted connections having different values, and every expression node is connected to every other expression node through a second set of weighted connections having different values; a weight matrix storing connection weights corresponding to the first set of weighted connections between the input nodes of the input layer and the representation nodes of the representation layer; When both the input node and the expression node are active, the connection weights stored in the weight matrix are strengthened; in response to detecting that two expression nodes are coactive, the connection weights between the input nodes to both expression nodes are weakened; the input nodes of the input layer receive a first set of values ​​corresponding to the pixelated information of the input; a second set of values ​​for the expression nodes in the representation layer is calculated based at least in part on inputs received via (i) the first set of weighted connections between the input nodes and the expression nodes, and (ii) the second set of weighted connections between the expression nodes; the second set of values ​​for the representation nodes in the representation layer are utilized to generate the object representation for the input.

20. a first set of connection weights associated with the first set of weighted connections between the input nodes of the input layer and the representation nodes of the representation layer are selected to minimize the likelihood that two representation nodes in the representation layer are simultaneously active; 20. The non-transitory computer-readable medium of claim 19, wherein the first set of connection weights associated with the first set of weighted connections are initially calculated using estimates of eigenvectors of a variance-covariance matrix based on an input matrix created from a vector representation of a selected set of inputs, and the connection weights in the weight matrix are updated when additional inputs are received.

Images