Characterizing activity and encoding and decoding information in recurrent artificial neural networks

Abstract

Methods, systems, and apparatus for characterizing activity in a recurrent artificial neural network and encoding and decoding information, including computer programs encoded on a computer storage medium. In one aspect, a method may include characterizing activity in an artificial neural network. The method is performed by a data processing device and may include identifying a cluster pattern of activity in the artificial neural network. The cluster pattern of activity may encompass a cavity.

Description

Background Art

[0001] This specification relates to characterizing activity in recurrent artificial neural networks. The characterization of activity can be used, for example, to identify decision moments and to encode / decode signals in scenarios such as transmission, encryption, and data storage. It also relates to systems and techniques for encoding and decoding information, and for using the encoded information in various scenarios. The encoded information can represent activity in a neural network (e.g., a recurrent neural network).

[0002] Artificial neural networks are devices inspired by the structure and functionality of biological neural networks. Specifically, they use a system of interconnected constructs called nodes to emulate the information encoding and other processing capabilities of biological neural networks. The arrangement and strength of the connections between nodes in an artificial neural network determine the outcomes of information processing or storage by the artificial neural network.

[0003] Neural networks can be trained to produce a desired signal flow within the network and achieve a desired information processing or information storage result. Typically, training a neural network will change the arrangement and / or strength of connections between nodes during a learning phase. A neural network is considered trained when it achieves sufficiently appropriate processing results for a given set of inputs.

[0004] Artificial neural networks can be used in a wide variety of different devices to perform nonlinear data processing and analysis. Nonlinear data processing does not satisfy the superposition principle, that is, the variable to be determined cannot be written as a linear sum of independent components. Examples of scenarios where nonlinear data processing is useful include systems and techniques in pattern and sequence recognition, speech processing, novelty detection and sequential decision making, complex system modeling, and a wide variety of other scenarios.

[0005] Both encoding and decoding convert information from one form or representation to another. Different representations can provide different features that are more or less useful in different applications. For example, some forms or representations of information (e.g., natural language) can be easier for humans to understand. Other forms or representations can be smaller in size (e.g., "compressed") and easier to transmit or store. Still other forms or representations can intentionally obscure the content of the information (e.g., the information can be encrypted).

[0006] Regardless of the specific application, the encoding or decoding process will generally follow a predefined set of rules or algorithms that establish a correspondence between different forms or representations of information. For example, an encoding process that produces a binary code can assign roles or meanings to individual bits based on their position in a binary sequence or vector. Summary of the Invention

[0007] This specification describes techniques related to characterizing activity in artificial neural networks.

[0008] For example, in one implementation, a method may include characterizing activity in an artificial neural network. The method may be performed by a data processing device and may include identifying a clique pattern of activity in the artificial neural network. The clique pattern of activity encloses a cavity.

[0009] This implementation and other implementations may include one or more of the following features. The method may include defining a plurality of time windows during which the activity of the artificial neural network is responsive to inputs into the artificial neural network. The clique pattern of activity may be identified in each of the plurality of time windows. The method may include identifying a first time window within the plurality of time windows based on a distinguishable likelihood of the clique pattern of activity occurring during the first window. Identifying clique patterns may include identifying directed cliques of activity. Lower dimensional directed cliques that are present within higher dimensional directed cliques may be discarded or ignored.

[0010] The method may include classifying the blob patterns into categories and characterizing the activity according to the number of occurrences of the blob patterns in corresponding categories in the categories. Classifying the blob patterns may include classifying the blob patterns according to the number of points within each blob pattern. The method may include outputting a binary sequence of 0s and 1s from a recurrent artificial neural network. Each number in the sequence may represent whether a corresponding pattern of activity is present in the artificial neural network. The method may include structuring the artificial neural network by reading the numbers output from the artificial neural network and evolving the structure of the artificial neural network. The structure of the artificial neural network may be evolved by iteratively changing the structure, characterizing the complexity of the pattern of activity in the changed structure, and using the characterization of the complexity of the pattern as an indication of whether the changed structure is desired.

[0011] The artificial neural network may be a recurrent artificial neural network. The method may include identifying a decision moment in the recurrent artificial neural network based on determining the complexity of a pattern of activity in the recurrent artificial neural network. Identifying the decision moment may include determining a specific timing of an activity having a complexity that is distinguishable from other activities responsive to the input, and identifying the decision moment based on the specific timing of the activity having the distinguishable complexity. The method may include inputting a data stream into the recurrent artificial neural network and identifying the blob pattern of activity during the input of the data stream. The method may include evaluating whether the activity is responsive to the input into the artificial neural network. The evaluation may include evaluating whether a relatively simple pattern of activity relatively shortly after an input event is responsive to the input, but a relatively complex pattern of activity relatively shortly after the input event is not responsive to the input, and evaluating whether a relatively complex pattern of activity relatively late after the input event is responsive to the input, but a relatively simple pattern of activity relatively late after the input event is not responsive to the input.

[0012] In another implementation, a system may include one or more computers operable to perform operations. The operations may include characterizing activity in an artificial neural network and identifying a clique pattern of activity in the artificial neural network, wherein the clique pattern of activity encompasses a cavity. The operations may include defining a plurality of time windows during which the activity of the artificial neural network is responsive to inputs into the artificial neural network. The clique pattern of activity may be identified in each of the plurality of time windows. The operations may include identifying a first time window within the plurality of time windows based on a distinguishable likelihood of the clique pattern of activity occurring during the first window. Identifying a clique pattern may include discarding or ignoring lower-dimensional directed cliques that are present within higher-dimensional directed cliques. The operations may include structuring the artificial neural network, including reading the digital output from the artificial neural network and evolving the structure of the artificial neural network. The structure of the artificial neural network may be evolved by iteratively changing the structure, characterizing the complexity of the pattern of activity in the changed structure, and using the characterization of the complexity of the pattern as an indication of whether the changed structure is desirable. The artificial neural network may be a recurrent artificial neural network. The operations may include identifying a decision moment in the recurrent artificial neural network based on determining the complexity of a pattern of activity in the recurrent artificial neural network. Identifying the decision moment may include determining a specific time of activity having a complexity distinguishable from other activities responsive to the input, and identifying the decision moment based on the specific time of the activity having the distinguishable complexity. The operations may include inputting a data stream into the recurrent artificial neural network and identifying the blob pattern of activity during the input of the data stream. The operations may include evaluating whether the activity is responsive to the input into the artificial neural network. The evaluation may include evaluating whether a relatively simple pattern of activity relatively shortly after an input event is responsive to the input, but a relatively complex pattern of activity relatively shortly after the input event is not responsive to the input, and evaluating whether a relatively complex pattern of activity relatively late after the input event is responsive to the input, but a relatively simple pattern of activity relatively late after the input event is not responsive to the input.

[0013] As another example, a method for identifying a decision moment in a neural network includes determining a complexity of a pattern of activity in a recurrent artificial neural network, wherein the activity is responsive to an input into the recurrent artificial neural network; determining a particular time of the activity having a complexity distinguishable from other activities responsive to the input; and identifying the decision moment based on the particular time of the activity having the distinguishable complexity.

[0014] As another example, a method for characterizing activity in a recurrent artificial neural network includes identifying a predefined clique pattern of activity in the recurrent artificial neural network. The method is performed by a data processing device. As another example, a method may include outputting a binary sequence of 0s and 1s from the recurrent artificial neural network, wherein each digit in the sequence indicates whether a particular group of nodes in the recurrent artificial neural network exhibits a corresponding pattern of activity.

[0015] As another example, a method for structuring a recurrent artificial neural network may include: characterizing the complexity of patterns of activity that may occur in the recurrent artificial neural network, the recurrent artificial neural network comprising a structured set of nodes and links between the nodes; and evolving the structure of the recurrent artificial neural network to increase the complexity of the patterns of activity. This structuring method may also be used, for example, as part of a method for training the recurrent artificial neural network.

[0016] Other embodiments of these aspects include corresponding systems, apparatus, and computer programs configured to perform the actions of the methods encoded on computer storage devices.

[0017] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. For example, conventional data processing equipment, such as, for example, digital computers and other computers, are programmed to follow a predefined logical sequence when processing information. Therefore, the moment when the computer achieves a result is relatively easy to identify. That is, the completion of the logical sequence embedded in the programming indicates when the information processing is complete and the computer has "reached a decision." The result can be maintained in a relatively long-lived form at the output of the computer's data processor, such as by a memory device, a set of buffers, etc., and can be accessed for a variety of purposes.

[0018] In contrast, as described herein, decision moments in an artificial recurrent neural network can be identified based on characteristics of the dynamic properties of the neural network during information processing. Decision moments in an artificial recurrent neural network can be identified based on characteristics of the functional state of the artificial neural network during information processing, rather than waiting for the artificial neural network to reach a predefined end of a logic sequence.

[0019] Furthermore, features of the dynamics of recurrent artificial neural networks during information processing, including features of activity commensurate with clique patterns and directed clique patterns, can be used in a variety of signaling operations, including signal transmission, encoding, encryption, and storage. In particular, features of activity in recurrent artificial neural networks during information processing reflect the input and can be considered an encoded form of the input (i.e., the "output" of the recurrent artificial neural network during the encoding process). These features can, for example, be transmitted to a remote receiver, which can decode the transmitted features to reconstruct the input or a portion of the input.

[0020] Furthermore, in some cases, the activities in different node groups of a recurrent artificial neural network (e.g., activities consistent with clique patterns and directed clique patterns) can be represented as a binary sequence of 0s and 1s, with each digit indicating whether the activity is consistent with the pattern. Since the activities can be the output of a recurrent artificial neural network in some scenarios, the output of the recurrent artificial neural network can be represented as a vector of binary digits and is compatible with digital data processing.

[0021] Furthermore, in some cases, such characterization of the dynamics of a recurrent artificial neural network can be used before and / or during training to increase the likelihood of complex patterns of activity emerging during information processing. For example, before or during training, the links between nodes in a recurrent artificial neural network can be intentionally evolved to increase the complexity of the activity patterns. For example, the links between nodes in a recurrent artificial neural network can be intentionally evolved to increase the likelihood of, for example, clique and directed clique patterns of activity emerging during information processing. This can reduce the time and effort required to train a recurrent artificial neural network.

[0022] As another example, such characterization of the dynamic properties of a recurrent artificial neural network can be used to determine the degree of completion in the training of the recurrent artificial neural network. For example, a recurrent artificial neural network that exhibits a particular type of ordering in its activity (e.g., clique patterns and directed clique patterns) can be considered to be more highly trained than a recurrent artificial neural network that does not exhibit such ordering. In fact, in some cases, the degree of training can be quantified by quantifying the degree of ordering in the activity of the recurrent artificial neural network.

[0023] For example, a method for identifying a decision moment in a neural network includes: determining a complexity of a pattern of activity in a recurrent artificial neural network, wherein the activity is responsive to an input into the recurrent artificial neural network; determining a particular time of the activity having a complexity distinguishable from other activities responsive to the input; and identifying the decision moment based on the particular time of the activity having the distinguishable complexity.

[0024] As another example, a method for characterizing activity in a recurrent artificial neural network includes identifying clique patterns in the activity of the recurrent artificial neural network. The method is performed by a data processing device.

[0025] As another example, a method may include outputting a binary sequence of 0s and 1s from a recurrent artificial neural network, wherein each digit in the sequence represents whether a particular group of nodes in the recurrent artificial neural network exhibits a corresponding pattern of activity.

[0026] As another example, a method for structuring a recurrent artificial neural network may include: characterizing the complexity of patterns of activity that may occur in the recurrent artificial neural network, the recurrent artificial neural network comprising a structured set of nodes and links between the nodes; and evolving the structure of the recurrent artificial neural network to increase the complexity of the patterns of activity. This structuring method may also be used, for example, as part of a method for training a recurrent artificial neural network.

[0027] Other embodiments of these aspects include corresponding systems, apparatus, and computer programs configured to perform the actions of the methods encoded on computer storage devices.

[0028] Specific embodiments of the subject matter described in this specification can be implemented to achieve one or more of the following advantages. For example, conventional data processing equipment, such as, for example, digital computers and other computers, are programmed to follow a predefined logical sequence when processing information. Therefore, the moment when the computer achieves a result is relatively easy to identify. That is, the completion of the logical sequence embedded in the programming indicates when the information processing is complete and the computer has "reached a decision." The result can be maintained in a relatively long-lived form at the output of the computer's data processor, such as by a memory device, a set of buffers, etc., and can be accessed for a variety of purposes.

[0029] In contrast, as described herein, decision moments in an artificial recurrent neural network can be identified based on characteristics of the dynamic properties of the neural network during information processing. Decision moments in an artificial recurrent neural network can be identified based on characteristics of the functional state of the artificial neural network during information processing, rather than waiting for the artificial neural network to reach a predefined end of a logic sequence.

[0030] Furthermore, features of the dynamics of recurrent artificial neural networks during information processing—including features of activity commensurate with clique patterns and directed clique patterns—can be used in a variety of signaling operations, including signal transmission, encoding, encryption, and storage. In particular, features of activity in recurrent artificial neural networks during information processing reflect the input and can be considered an encoded form of the input (i.e., the "output" of the recurrent artificial neural network during the encoding process). These features can, for example, be transmitted to a remote receiver, which can decode the transmitted features to reconstruct the input or a portion of the input.

[0031] Furthermore, in some cases, the activities in different node groups of a recurrent artificial neural network (e.g., activities consistent with clique patterns and directed clique patterns) can be represented as a binary sequence of 0s and 1s, with each digit indicating whether the activity is consistent with the pattern. Since the activities can be the output of a recurrent artificial neural network in some scenarios, the output of the recurrent artificial neural network can be represented as a vector of binary digits and is compatible with digital data processing.

[0032] Furthermore, in some cases, such characterization of the dynamics of a recurrent artificial neural network can be used before and / or during training to increase the likelihood of complex patterns of activity emerging during information processing. For example, before or during training, the links between nodes in a recurrent artificial neural network can be intentionally evolved to increase the complexity of the activity patterns. For example, the links between nodes in a recurrent artificial neural network can be intentionally evolved to increase the likelihood of, for example, clique and directed clique patterns of activity emerging during information processing. This can reduce the time and effort required to train a recurrent artificial neural network.

[0033] As another example, such characterization of the dynamic properties of a recurrent artificial neural network can be used to determine the degree of completion in the training of the recurrent artificial neural network. For example, a recurrent artificial neural network that exhibits a particular type of ordering in its activity (e.g., clique patterns and directed clique patterns) can be considered to be more highly trained than a recurrent artificial neural network that does not exhibit such ordering. In fact, in some cases, the degree of training can be quantified by quantifying the degree of ordering in the activity of the recurrent artificial neural network.

[0034] As yet another example, in one implementation, a device includes a neural network trained to, in response to a first input, produce an approximation of a first representation of a topological structure in a pattern of activity that occurs in a source neural network in response to the first input, to, in response to a second input, produce an approximation of a second representation of the topological structure in the pattern of activity that occurs in the source neural network in response to the second input, and to, in response to a third input, produce an approximation of a third representation of the topological structure in the pattern of activity that occurs in the source neural network in response to the third input.

[0035] This and other implementations may include one or more of the following features. The topological structures may all include two or more nodes in the source neural network and one or more edges between the nodes. The topological structures may include simplices. The topological structures may enclose a cavity. Each of the first representation, the second representation, and the third representation may represent a topological structure that occurs in the source neural network only at times during which the pattern of activity has a complexity that is distinguishable from the complexity of other activity in response to corresponding ones of the inputs. The device may also include a processor coupled to receive the approximation of the representation generated by the neural network device and to process the received approximation. The processor may include a second neural network that has been trained to process the representation generated by the neural network. Each of the first representation, the second representation, and the third representation may include multi-valued, non-binary numbers. Each of the first representation, the second representation, and the third representation may represent an occurrence of the topological structure without specifying where in the source neural network the pattern of activity occurs. The device may include a smartphone. The source neural network may be a recurrent neural network.

[0036] In another implementation, a device includes a neural network coupled to an input representing a topological structure in a pattern of activity occurring in a source neural network in response to a plurality of different inputs. The neural network is trained to process the representation and produce a responsive output.

[0037] This implementation and other implementations may include one or more of the following features. The topological structures may all comprise two or more nodes in the source neural network and one or more edges between the nodes. The topological structures may comprise simplexes. The representation of the topological structure may represent the topological structure that occurs in the source neural network only at times during which the pattern of activity has a complexity that is distinguishable from the complexity of other activity in response to corresponding ones of the inputs. The apparatus may include a neural network that is trained to produce, in response to a plurality of different inputs, corresponding approximations of the representation of the topological structure in the pattern of activity that occurs in the source neural network in response to the different inputs. The representation of the topological structure may comprise multi-valued, non-binary numbers. The representation of the topological structure may represent an occurrence of the topological structure without specifying where in the source neural network the pattern of activity occurs. The source neural network may be a recurrent neural network.

[0038] In another implementation, a method is implemented by a neural network device and includes: inputting a representation of a topological structure in a pattern of activity in a source neural network, wherein the activity is responsive to input into the source neural network; processing the representation; and outputting a result of the processing of the representation. The processing is consistent with training the neural network to process a different such representation of the topological structure in the pattern of activity in the source neural network.

[0039] This implementation and other implementations may include one or more of the following features. The topological structures may all comprise two or more nodes in the source neural network and one or more edges between the nodes. The topological structures may include simplexes. The topological structures may enclose a cavity. The representation of the topological structure may represent the topological structure that occurs in the source neural network only at times during which the pattern of activity has a complexity that is distinguishable from the complexity of other activity in response to corresponding ones of the inputs. The representation of the topological structure may include multi-valued, non-binary numbers. The representation of the topological structure may represent an occurrence of the topological structure without specifying where in the source neural network the pattern of activity occurs. The source neural network may be a recurrent neural network.

[0040] As yet another example, in one implementation, a device includes a neural network coupled to an input representing a topological structure in a pattern of activity occurring in a source neural network in response to a plurality of different inputs. The neural network is trained to process the representation and produce a responsive output.

[0041] This implementation and other implementations may include one or more of the following features. The topological structures all include two or more nodes in the source neural network and one or more edges between the nodes. The device may include: an actuator coupled to receive the response output from the neural network and act on a real or virtual environment; a sensor coupled to measure characteristics of the environment; and a teacher module configured to interpret the measurements received from the sensor and provide a reward and / or regret to the neural network. The topological structure may include a simplex. The topological structure may enclose a cavity. The representation of the topological structure may represent a topological structure that occurs in the source neural network only at the following times: the pattern of activity during the time has a complexity that is distinguishable from the complexity of other activities in response to corresponding ones of the inputs. The device may include a second neural network trained to produce, in response to a plurality of different inputs, respective approximations of the representation of the topological structure in the pattern of activity that occurs in the source neural network in response to the different inputs. Such a device may further comprise an actuator coupled to receive the response output from the neural network and act on a real or virtual environment, and a sensor coupled to measure a characteristic of the environment. The second neural network may be trained to produce the corresponding approximation at least in part in response to the measured characteristic of the environment. The device may further comprise a teacher module configured to interpret the measurements received from the sensor and provide a reward and / or regret to the neural network. The representation of the topology may comprise a multi-valued, non-binary number. The representation of the topology may represent an occurrence of the topology without specifying where in the source neural network the pattern of activity occurs. The device may be a smartphone. The source neural network may be a recurrent neural network.

[0042] In another implementation, a method implemented by one or more data processing devices may include receiving a training set comprising a plurality of representations of topological structures in patterns of activity in a source neural network, and training the neural network using the representations as input to the neural network or as target answer vectors, the activity being responsive to input into the source neural network.

[0043] This implementation and other implementations may include one or more of the following features. The topologies all include two or more nodes in the source neural network and one or more edges between the nodes. The training set may include multiple input vectors, each of the input vectors corresponding to a corresponding representation in the representations. Training the neural network may include training the neural network using each of the multiple representations as a target answer vector. Training the neural network may include training the neural network using each of the multiple representations as an input. The training set may include multiple reward or regret values. Training the neural network may include reinforcement learning. The topologies may include simplexes. The representation of the topologies may represent topologies that occur in the source neural network only at times during which the pattern of activity has a complexity that is distinguishable from the complexity of other activity in response to corresponding ones of the inputs. The representation of the topologies may include multi-valued, non-binary numbers. The representation of the topologies may represent occurrences of the topologies without specifying where in the source neural network the pattern of activity occurs. The source neural network may be a recurrent neural network.

[0044] The details of one or more implementations described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a schematic illustration of the structure of a recurrent artificial neural network device.

[0046] Figure 2 and Figure 3 is a schematic illustration of the functionality of a looping artificial neural network device within different time windows.

[0047] Figure 4 is a flow chart of a process for identifying decision moments in a recurrent artificial neural network based on characterization of activity in the network.

[0048] Figure 5 is a schematic illustration of patterns of activity that can be identified and used to identify decision moments in a recurrent artificial neural network.

[0049] Figure 6 is a schematic illustration of patterns of activity that can be identified and used to identify decision moments in a recurrent artificial neural network.

[0050] Figure 7 is a schematic illustration of patterns of activity that can be identified and used to identify decision moments in a recurrent artificial neural network.

[0051] Figure 8 is a schematic illustration of a data table that may be used in determining the degree of complexity or ordering in an activity pattern in a recurrent artificial neural network device.

[0052] Figure 9 is a schematic illustration of the time-specific determination of activity patterns with distinguishable complexity.

[0053] Figure 10 is a flow chart of a process for encoding signals using a recurrent artificial neural network based on characterization of activity in the network.

[0054] Figure 11 is a flow chart of a process for decoding signals using a recurrent artificial neural network based on characterization of activity in the network.

[0055] Figure 12 、 Figure 13 and Figure 14 is a schematic illustration of a binary form or representation of a topological structure.

[0056] Figure 15 and Figure 16 An example of how the presence or absence of features corresponding to different bits are not independent of each other is schematically illustrated.

[0057] Figure 17 、 Figure 18 、 Figure 19 、 Figure 20 is a schematic illustration of the representation of the occurrence of topological structures in activities in four different classification systems using neural networks.

[0058] Figure 21 、 Figure 22 is a schematic illustration of an edge device comprising a local artificial neural network that can be trained using a representation of occurrences of a topology corresponding to activity in a source neural network.

[0059] Figure 23 is a schematic illustration of a system in which a local neural network can be trained using a representation of the occurrence of a topological structure corresponding to activity in a source neural network.

[0060] Figure 24 、 Figure 25 、 Figure 26 、 Figure 27 is a schematic illustration of the representation of the occurrence of topological structures in activity in four different systems using neural networks.

[0061] Figure 28is a schematic illustration of a system 0 comprising an artificial neural network that can be trained using a representation of occurrences corresponding to a topology of activity in a source neural network.

[0062] Like reference symbols in the various drawings indicate like elements. DETAILED DESCRIPTION

[0063] Figure 1 1 is a schematic illustration of the structure of a recurrent artificial neural network device 100. A recurrent artificial neural network device 100 is a device that uses a system of interconnected nodes to simulate the information encoding and other processing capabilities of a biological neural network. A recurrent artificial neural network device 100 can be implemented in hardware, software, or a combination thereof.

[0064] The example of a recurrent artificial neural network device 100 includes a plurality of nodes 101, 102, ..., 107 interconnected by a plurality of structural links 110. Nodes 101, 102, ..., 107 are discrete information processing constructs similar to neurons in biological networks. Nodes 101, 102, ..., 107 generally process one or more input signals received via one or more of the links 110 to produce one or more output signals output via one or more of the links 110. For example, in some implementations, nodes 101, 102, ..., 107 may be artificial neurons that weight and sum multiple input signals, pass the sum through one or more nonlinear activation functions, and output one or more output signals.

[0065] Nodes 101, 102, ..., 107 can operate as accumulators. For example, nodes 101, 102, ..., 107 can operate according to an integrate-and-fire model, in which one or more signals are accumulated in a first node until a threshold is reached. After reaching the threshold, the first node fires by transmitting an output signal to a connected second node along one or more of links 110. In turn, the second node 101, 102, ..., 107 accumulates the received signal and, if the threshold is reached, transmits another output signal to another connected node.

[0066] Fabric links 110 are connections capable of transmitting signals between nodes 101, 102, ..., 107. For convenience, all fabric links 110 are herein considered to be identical bidirectional links, which transmit signals from each first one of nodes 101, 102, ..., 107 to each second one of nodes 101, 102, ..., 107 in the same manner as signals are transmitted from the second one to the first one. However, this is not necessarily the case. For example, some or all fabric links 110 may be unidirectional links, which transmit signals from a first one of nodes 101, 102, ..., 107 to a second one of nodes 101, 102, ..., 107, without transmitting signals from the second one to the first one.

[0067] As another example, in some implementations, the structural links 110 can have a variety of properties other than or in addition to directionality. For example, in some implementations, different structural links 110 can carry signals of different amplitudes—resulting in different interconnection strengths between corresponding nodes in nodes 101, 102, ..., 107. As another example, different structural links 110 can carry different types of signals (e.g., inhibitory and / or excitatory signals). Indeed, in some implementations, the structural links 110 can mimic the links between somatic cells in biological systems and reflect at least a portion of the enormous morphological, chemical, and other diversity of such links.

[0068] In the illustrated implementation, the recurrent artificial neural network device 100 is a clique network (or subnetwork) because each node 101, 102, ..., 107 is connected to every other node 101, 102, ..., 107. This is not necessarily the case. Instead, in some implementations, each node 101, 102, ..., 107 may be connected to a proper subset of the nodes 101, 102, ..., 107 (via the same link or a variety of links, as the case may be).

[0069] For clarity of illustration, recurrent artificial neural network device 100 is illustrated as having only seven nodes. Typically, a real-world neural network device will include a significantly larger number of nodes. For example, in some implementations, a neural network device may include hundreds of thousands, millions, or even billions of nodes. Thus, recurrent neural network device 100 may be a small portion (i.e., a subnetwork) of a larger recurrent artificial neural network.

[0070] In biological neural network devices, the accumulation and signal transmission processes require the passage of real-world time. For example, the soma of a neuron integrates inputs received over time, and signal transmission from neuron to neuron requires time, determined by factors such as the speed of signal transmission and the nature and length of the links between neurons. Therefore, the state of a biological neural network device is dynamic and changes over time.

[0071] In artificial recurrent neural network devices, time is artificial and represented using mathematical constructs. For example, rather than requiring the passage of real-world time for a signal to travel from node to node, such signals can be represented using artificial units that are typically unrelated to the passage of real-world time, such as measured by computer clock cycles or other means. However, the state of an artificial recurrent neural network device can be described as "dynamic" because it changes with respect to these artificial units.

[0072] Note that for convenience, these artificial units are referred to herein as "time" units. However, it should be understood that these units are artificial and generally do not correspond to the real-world passage of time.

[0073] Figure 2 and Figure 3 100 within different time windows. Because the state of device 100 is dynamic, the signal transmission activity occurring within a window can be used to represent the functionality of device 100. Such functional illustrations typically show activity in only a small portion of links 110. In particular, because not every link 110 typically transmits a signal within a particular window, not every link 110 is illustrated as actively contributing to the functionality of device 100 in these illustrations.

[0074] exist Figure 2 and Figure 3 In the illustration of FIG, an active link 110 is illustrated as a relatively thick solid line connecting a pair of nodes 101, 102, ..., 107. In contrast, an inactive link 110 is illustrated as a dashed line. This is for illustration purposes only. In other words, the structural connection formed by link 110 exists regardless of whether link 110 is active. However, this formalism highlights the activity and functionality of device 100.

[0075] In addition to schematically illustrating the presence of activity along a link, the direction of the activity is also schematically illustrated. In particular, the relatively thick solid line illustrating an active link 110 also includes an arrow indicating the direction of signal transmission along the link during the relevant window. Generally, the direction of signal transmission within a single window does not decisively limit the link to a unidirectional link with the indicated directionality. Rather, in a first functional instantiation for a first time window, the link can be active in a first direction. In a second functional instantiation for a second window, the link can be active in the opposite direction. However, in some cases, such as, for example, in a recurrent artificial neural network device 100 that includes only unidirectional links, the directionality of the signal transmission will decisively indicate the directionality of the link.

[0076] In a feedforward neural network, information moves in only a single direction (i.e., forward) to an output layer of nodes at the end of the network. A feedforward neural network indicates that a "decision" has been reached and information processing is complete by the propagation of a signal through the network to the output layer.

[0077] In contrast, in a recurrent neural network, the connections between nodes form loops, and the network's activity progresses dynamically without easily identifiable decisions. For example, even in a three-node recurrent neural network, a first node can transmit a signal to a second node, and in response, the second node can transmit a signal to a third node. In response, the third node can transmit a signal back to the first node. The signal received by the first node can be, at least in part, a response to a signal transmitted from that same node.

[0078] Schematic functional examples Figure 2 and Figure 3 This is illustrated in a network only slightly larger than a three-node recurrent neural network. Figure 2 The functional examples shown in may illustrate activities within the first window, and Figure 3 Activity within the immediately following second window can be illustrated. As shown, the collection of signal transmission activity appears to originate in node 104 and progress in a generally clockwise direction through device 100 during the first window. Within the second window, at least some of the signal transmission activity generally appears to return to node 104. Even in this overly simplistic illustration, the signal transmission does not occur in a manner that produces a clearly identifiable output or end.

[0079] When considering, for example, recurrent neural networks of thousands of nodes or more, it is recognized that signal propagation can occur through a vast number of paths, and these signals lack a clearly identifiable "output" location or time. Although the network can be designed to return to a resting state where only background activity occurs or even no signal transmission occurs, the resting state itself does not indicate the outcome of information processing. Regardless of the input, the recurrent neural network always returns to the resting state. Therefore, the "output" or outcome of information processing is encoded in the activity that occurs in the recurrent neural network in response to a specific input.

[0080] Figure 4 4 is a flow chart of a process 400 for identifying decision moments in a recurrent artificial neural network based on characterization of activity in the network. A decision moment is a point in time at which activity in the recurrent artificial neural network indicates the outcome of information processing by the network in response to input. Process 400 can be performed by a system of one or more data processing devices operating according to logic of one or more sets of machine-readable instructions. For example, process 400 can be performed by the same system of one or more computers executing software implementing the recurrent artificial neural network used in process 400.

[0081] At 405, the system performing process 400 receives a notification that a signal has been input into a recurrent artificial neural network. In some cases, the input of the signal is a discrete injection event, where, for example, information is injected into one or more nodes and / or one or more links of the neural network. In other cases, the input of the signal is a flow of information injected into one or more nodes and / or links of the neural network over a period of time. The notification indicates that the artificial neural network is actively processing information and is not, for example, in a quiescent state. In some cases, the notification is received from the neural network itself, such as, for example, when the neural network exits a recognizable quiescent state.

[0082] At 410, the system performing process 400 divides the response activity in the network into a set of windows. Where the injection is a discrete event, the windows can subdivide the time between the injection and the return to a quiescent state into periods during which the activity exhibits variable complexity. Where the injection is an information flow, the duration of the injection (and optionally the time to return to a quiescent state after the injection is complete) can be subdivided into windows during which the activity exhibits variable complexity. Various methods for determining the complexity of an activity are discussed further below.

[0083] In some implementations, the windows all have the same duration, but this is not necessarily the case. Instead, in some implementations, the windows can have different durations. For example, in some implementations, the duration can increase with the increase in time since the discrete injection event has occurred.

[0084] In some implementations, the window can be a continuous series of individual windows. In other implementations, the windows overlap in time so that one window starts before the previous window ends. In some cases, the window can be a moving window that moves in time.

[0085] In some implementations, different durations of windows are defined for different determinations of the complexity of an activity. For example, for an activity pattern that defines activity occurring between a relatively large number of nodes, the window may have a relatively longer duration than a window defined for an activity pattern that defines activity occurring between a relatively small number of nodes. For example, in the scenario of activity pattern 500 ( Figure 5 ), the window defined for identifying activities consistent with pattern 530 can be longer than the window defined for identifying activities consistent with pattern 505.

[0086] At 415, the system performing process 400 identifies patterns in activity within different windows in the network. As discussed further below, patterns in activity can be identified by viewing a functional graph as a topological space with nodes as points. In some implementations, the identified activity patterns are cliques, e.g., directed cliques, in the functional graph of the network.

[0087] At 420, the system performing process 400 determines the complexity of the activity patterns within the different windows. Complexity can be a measure of the likelihood that an ordered pattern of activity will occur within a window. Thus, an activity pattern that appears randomly will be relatively simple. On the other hand, an activity pattern that exhibits a non-random order is relatively complex. For example, in some implementations, the complexity of an activity pattern can be measured using, for example, a simplex count or a Betti number of the activity pattern.

[0088] At 425, the system performing process 400 determines specific times of activity patterns having a distinguishable level of complexity. The specific activity patterns can be distinguishable based on a level of complexity that deviates upward or downward (e.g., from a fixed or variable baseline). In other words, specific times of activity patterns that indicate a non-random sequence of activity with a particularly high level or a particularly low level can be determined.

[0089] For example, where the signal input is a discrete injection event, deviations, such as deviations from a stable baseline or deviations from a curve that is characteristic of the average response of the neural network to a variety of different discrete injection events, can be used to determine the specific timing of distinguishable complex activity patterns. As another example, where the information is input in the form of a stream, large changes in complexity during the streaming period can be used to determine the specific timing of distinguishable complex activity patterns.

[0090] At 430, the system performing process 400 schedules the reading of the output from the neural network based on the specific timing of the distinguishably complex activity pattern. For example, in some implementations, the output of the neural network can be read at the same time that the distinguishably complex activity pattern occurs. In implementations where the complexity deviation indicates a relatively high degree of non-random order in the activity, the observed activity pattern itself can also be treated as the output of the recurrent artificial neural network.

[0091] Figure 5 is an illustration of a pattern 500 of activity that can be identified and used to identify decision moments in a recurrent artificial neural network. For example, in process 400 ( Figure 4 ) at 415 to identify pattern 500.

[0092] Pattern 500 is an illustration of activity in a recurrent artificial neural network. During application of pattern 500, the function graph is viewed as a topological space with nodes as points. Activity in nodes and links that conform to pattern 500 can be identified as ordered, regardless of the identity of the specific nodes and / or links participating in the activity. For example, a first pattern 505 may represent Figure 2 , with point 0 in pattern 505 being node 104, point 1 being node 105, and point 2 being node 101. As another example, the first pattern 505 may also represent Figure 3 The activities between nodes 104, 105, and 106 in the directed clique are represented by point 0 in pattern 505 as node 106, point 1 as node 104, and point 2 as node 105. The order of the activities in the directed clique is also specified. For example, in pattern 505, the activity between points 1 and 2 occurs after the activity between points 0 and 1.

[0093] In the illustrated implementation, pattern 500 is all directed cliques or directed simplexes. In such a pattern, activity originates from a source node that transmits a signal to each other node in the pattern. In pattern 500, such a source node is designated as point 0, and other nodes are designated as points 1, 2, ... . In addition, in a directed clique or simplex, one of the nodes acts as a sink and receives a signal transmitted from each other node in the pattern. In pattern 500, such a sink node is designated as the highest numbered point in the pattern. For example, in pattern 505, the sink node is designated as point 2. In pattern 510, the sink node is designated as point 3. In pattern 515, the sink node is designated as point 3, and so on. Therefore, the activities represented by pattern 500 are sorted in a distinguishable manner.

[0094] Each of the patterns 500 has a different number of points and reflects ordered activity in a different number of nodes. For example, pattern 505 is a two-dimensional simplex and reflects activity in three nodes, pattern 510 is a three-dimensional simplex and reflects activity in four nodes, and so on. As the number of points in the pattern increases, the degree of ordering and complexity of the activity also increases. For example, for a large set of nodes with a certain degree of random activity within the window, some of the activity may coincide with pattern 505 by chance. However, the random activity will gradually become less and less likely to coincide with the corresponding pattern in patterns 510, 515, 520... The presence of activity that coincides with pattern 530 indicates a relatively higher degree of ordering and complexity in the activity than the presence of activity that coincides with pattern 505.

[0095] As previously discussed, in some implementations, windows of different durations may be defined for different determinations of the complexity of an activity. For example, when identifying an activity that matches pattern 530, a window of longer duration may be used than when identifying an activity that matches pattern 505.

[0096] Figure 6 is an illustration of a pattern 600 of activity that can be identified and used to identify decision moments in a recurrent artificial neural network. For example, in process 400 ( Figure 4 ) at 415 to identify pattern 600.

[0097] Like pattern 500, pattern 600 is an illustration of activity in a recurrent artificial neural network. However, pattern 600 deviates from the strict ordering of pattern 500 because not all of pattern 600 are directed cliques or directed simplexes. In particular, patterns 605 and 610 have lower directionality than pattern 515. In fact, pattern 605 completely lacks sink nodes. Nevertheless, patterns 605 and 610 indicate a degree of ordered activity that exceeds that expected by random chance and can be used to determine the complexity of activity in a recurrent artificial neural network.

[0098] Figure 7 is an illustration of a pattern 700 of activity that can be identified and used to identify decision moments in a recurrent artificial neural network. For example, in process 400 ( Figure 4 ) at 415 to identify pattern 700.

[0099] The pattern 700 is a group of directed cliques or directed simplices of the same dimension (ie, having the same number of points) that defines a pattern involving more points than a single clique or simplice and encloses a cavity within the group of directed simplices.

[0100] By way of example, pattern 705 includes six different three-point, two-dimensional patterns 505 that together define a homology class of level 2, while pattern 710 includes eight different three-point, two-dimensional patterns 505 that together define a second homology class of level 2. Each of the three-point, two-dimensional patterns 505 in patterns 705, 710 can be considered to enclose a corresponding cavity. The nth Betti number associated with a directed graph provides a count of such homology classes in a topological representation.

[0101] The activity exemplified by patterns such as pattern 700 exemplifies a relatively high degree of ordering of activity in a network that is unlikely to occur by random chance. Pattern 700 can be used to characterize the complexity of the activity.

[0102] In some implementations, only some patterns of activity are identified and / or some portion of the identified patterns of activity are discarded or otherwise ignored during identification at a decision moment. Figure 5 , activities commensurate with the five-point, four-dimensional simplex pattern 515 inherently include activities commensurate with the four-point, three-dimensional and three-point, two-dimensional simplex patterns 510, 505. For example, Figure 5 Points 0, 2, 3, 4 and points 1, 2, 3, 4 in the four-dimensional simplex pattern 515 are all commensurate with the three-dimensional simplex pattern 510. In some implementations, patterns containing fewer points—and therefore having lower dimensions—can be discarded or otherwise ignored during identification at a decision moment.

[0103] As another example, only some patterns of activity need to be identified. For example, in some implementations, only patterns with an odd number of points (3, 5, 7, ...) or an even number of dimensions (2, 4, 6, ...) are used in identifying decision moments.

[0104] The degree of complexity or order in the activity patterns in a recurrent artificial neural network device within different windows can be determined in a variety of different ways. Figure 8 is a schematic illustration of a data table 800 that can be used in such a determination. Data table 800 can be used to determine the complexity of an activity pattern in isolation or in combination with other activities. For example, in process 400 ( Figure 4 ) uses data table 800 at 420.

[0105] In more detail, table 800 includes a count of the number of occurrences of the pattern during window "N," with counts of activities matching patterns of different dimensions presented in different rows. For example, in the illustrated example, row 805 includes a count of the number of occurrences of activities matching one or more three-point, two-dimensional patterns (i.e., "2032"), while row 810 includes a count of the number of occurrences of activities matching one or more four-point, three-dimensional patterns (i.e., "877"). Because the occurrence of a pattern indicates that the activity has a non-random order, the counts also provide a generalized characterization of the overall complexity of the activity pattern. This can be done, for example, in process 400 ( Figure 4 ) forms a table similar to table 800.

[0106] Although table 800 includes a separate row and separate entry for each type of activity pattern, this is not necessarily the case. For example, one of the multiple counts (e.g., the count of a simpler pattern) can be omitted from table 800 and from the complexity determination. As another example, some implementations may include a single row or entry that includes counts of occurrences of multiple activity patterns.

[0107] although Figure 8 While the number counts are presented in table 800, this need not be the case. For example, the number counts may be presented as a vector (e.g., <2032, 877, 133, 66, 48, ...>). Regardless of how the counts are presented, in some implementations, the counts may be expressed in binary and may be compatible with digital data processing infrastructure.

[0108] In some implementations, the counts of the number of occurrences of a pattern can be weighted or combined to determine the degree or complexity of the ranking, e.g., in process 400 ( Figure 4 ) at 420. For example, the Eulerian characteristic can provide an approximation of the complexity of the activity and is given by the following equation:

[0109] S0-S1+S2-S3+... Equation 1 Among them S n is the number of occurrences of a pattern of n points (i.e., a pattern of dimension n-1). The pattern can be, for example, a directed clique pattern 500 ( Figure 5 ).

[0110] As another example of how the number of occurrences of a pattern can be weighted to determine the degree or complexity of the ranking, in some implementations, the pattern occurrences can be weighted based on the weights of the active links. In more detail, as previously discussed, the strength of the connections between nodes in an artificial neural network can vary, for example, due to the degree of activity of the connections during training. The occurrence of a pattern of activity along a set of relatively strong links can be weighted differently than the occurrence of the same pattern of activity along a set of relatively weak links. For example, in some implementations, the sum of the weights of the active links can be used to weight the occurrences.

[0111] In some implementations, the Euler characteristic or other measure of complexity can be normalized by the total number of patterns matched within a particular window and / or the total number of patterns that a given network can form given its structure. An example of normalization based on the total number of patterns that a network can form is given below in Equations 2 and 3.

[0112] In some implementations, the occurrence of higher-dimensional patterns involving a larger number of nodes can be weighted more heavily than the occurrence of lower-dimensional patterns involving a smaller number of nodes. For example, the probability of forming a directed clique decreases rapidly with increasing dimensionality. In particular, in order to form an n-clique from n+1 nodes, all (n+1)n / 2 edges need to be correctly oriented. This probability can be reflected in the weighting.

[0113] In some implementations, both the dimensionality and directionality of a pattern can be used to weight the occurrence of the pattern and determine the complexity of the activity. Figure 6 Based on the difference in directionality between five-point, four-dimensional pattern 515 and five-point, four-dimensional patterns 605 and 610, the occurrence of five-point, four-dimensional pattern 515 can be weighted more heavily than the occurrence of five-point, four-dimensional patterns 605 and 610.

[0114] An example of using both the directionality and dimensionality of a pattern to determine the degree of sequencing or complexity of an activity can be given by the following equation:

[0115]

[0116] Among them S x active Indicates the number of active occurrences of a pattern of n points and ERN is calculated for an equivalent random network (i.e., a network with the same number of nodes connected randomly). In addition, SC is given by the following equation:

[0117]

[0118] Among them S x silentIndicates the number of occurrences of a pattern of n points when the recurrent artificial neural network is silent and can be considered to represent the total number of patterns that the network can form. In Equations 2 and 3, the pattern can be, for example, a directed clique pattern 500 ( Figure 5 ).

[0119] Figure 9 is a schematic illustration of a specific time determination of an activity pattern with distinguishable complexity. It can be performed in isolation or in combination with other activities. Figure 9 For example, in process 400 ( Figure 4 ) performs this determination at 425.

[0120] Figure 9 905 and 910. Graph 905 illustrates the occurrence of a pattern as a function of time along the x-axis. In particular, each occurrence is schematically illustrated as vertical lines 906, 907, 908, 909. Each row of occurrences may be instances of an activity matching a corresponding pattern or class of patterns. For example, the top row of occurrences may be instances of an activity matching pattern 505 ( Figure 5 ), the occurrence of the second line may be an activity matching pattern 510 ( Figure 5 ), the occurrence of the third line may be an activity matching pattern 515 ( Figure 5 ), and so on.

[0121] Graph 905 also includes dashed rectangles 915, 920, 925 that schematically depict different time windows when activity patterns have distinguishable complexity. As shown, during the windows depicted by dashed rectangles 915, 920, 925, the likelihood that activity in the recurrent artificial neural network matches a pattern indicative of complexity is higher than outside those windows.

[0122] Graph 910 illustrates the complexity associated with these occurrences as a function of time along the x-axis. Graph 910 includes a first peak in complexity 930, which coincides with the window depicted by dashed rectangle 915, and a second peak in complexity 935, which coincides with the window depicted by dashed rectangles 920, 925. As shown, the complexity illustrated by peaks 930, 925 is distinguishable from the complexity of what can be considered a baseline level 940 of complexity.

[0123] In some implementations, the output of the recurrent artificial neural network is read at a time that coincides with the occurrence of an activity pattern of distinguishable complexity. For example, Figure 9 In the illustrative scenario of , the output of the recurrent artificial neural network may be read at peaks 930 , 925 , ie, during the windows depicted by dashed rectangles 915 , 920 , 925 .

[0124] Identifying distinguishable levels of complexity in recurrent artificial neural networks is particularly beneficial when the input is a data stream. Examples of data streams include, for example, video or audio data. Although data streams have a beginning, it is often desirable to process information in a data stream that has no predefined relationship to the beginning of the data stream. By way of example, a neural network can perform object recognition, such as, for example, identifying cyclists near a car. Such a neural network should be able to identify cyclists regardless of when those cyclists appear in the video stream, that is, regardless of the time since the beginning of the video. Continuing with this example, when a data stream is input into an object recognition neural network, any pattern of activity in the neural network will generally exhibit a low or static level of complexity. This low or static level of complexity will be exhibited regardless of the continuous (or near-continuous) input of streaming data into the neural network device. However, when an object of interest appears in the video stream, the complexity of the activity will become distinguishable and indicate the time when the object was identified in the video stream. Therefore, the specific time of the distinguishable level of activity complexity can also serve as a yes / no output regarding whether the data in the data stream meets certain criteria.

[0125] In some implementations, activity patterns of distinguishable complexity can be used to identify not only the specific timing of the output of a recurrent artificial neural network, but also the content of the output of the recurrent artificial neural network. Specifically, the identity and activity of nodes participating in activities commensurate with the activity pattern can be considered the output of the recurrent artificial neural network. Thus, the identified activity pattern can exemplify the results of processing by the neural network, as well as the specific time at which this decision was read.

[0126] The content of a decision can be expressed in a variety of different forms. For example, in some implementations and as discussed in further detail below, the content of a decision can be expressed as a binary vector or matrix of 1s and 0s. Each digit can indicate, for example, whether a pattern of activity exists for a predefined group of nodes and / or a predefined duration. In such an implementation, the content of the decision is expressed in binary and can be compatible with traditional digital data processing infrastructure.

[0127] Figure 101 is a flow chart of process 1000 for encoding a signal using a recurrent artificial neural network based on characterization of activity in the network. Signals can be encoded in a variety of different scenarios, such as, for example, transmission, encryption, and data storage. Process 1000 can be performed by a system of one or more data processing devices that perform operations according to the logic of one or more machine-readable instruction sets. For example, process 1000 can be performed by the same system of one or more computers that execute software for implementing the recurrent artificial neural network used in process 1000. In some instances, process 1000 can be performed by the same data processing device that performs process 400. In some instances, process 1000 can be performed by, for example, an encoder in a signal transmission system or an encoder in a data storage system.

[0128] At 1005, the system performing process 1000 inputs a signal into a recurrent artificial neural network. In some cases, the input signal is a discrete injection event. In other cases, the input signal is streamed into the recurrent artificial neural network.

[0129] At 1010, the system performing process 1000 identifies one or more decision moments in a recurrent artificial neural network. For example, the system may perform process 400 ( Figure 4 ) Identify one or more decision moments.

[0130] At 1015, the system performing process 1000 reads the output of the recurrent artificial neural network. As discussed above, in some implementations, the content of the output of the recurrent artificial neural network is the activity in the neural network that matches the pattern used to identify the decision point.

[0131] In some implementations, a separate "reader node" may be added to the neural network to identify the occurrence of a particular pattern of activity at a particular set of nodes and thereby read the output of the recurrent artificial neural network at 1015. The reader node may fire if and only if the activity at the particular set of nodes meets certain timing (and possibly also magnitude) criteria. For example, to generate a signal at nodes 104, 105, 106 ( Figure 2 、 Figure 3 ) at read mode 505( Figure 5 ) occurs, the reader node may be connected to nodes 104, 105, 106 (or links 110 therebetween). The reader node itself may become active only when a pattern of activity involving nodes 104, 105, 106 (or their links) occurs.

[0132] The use of such reader nodes would eliminate the need to define time windows for the recurrent artificial neural network as a whole. In particular, separate reader nodes can be connected to different nodes and / or multiple nodes (or links between them). Separate reader nodes can be set to have customized responses (e.g., integrating different decay times in the excitation model) to identify different activity patterns. At 1020, the system performing process 1000 transmits or stores the output of the recurrent artificial neural network. The specific actions performed at 1020 can reflect the scenario in which process 1000 is being used. For example, in a scenario where secure or compressed communication is desired, the system performing process 1000 can transmit the output of the recurrent neural network to a receiver that can access the same or similar recurrent neural network. As another example, in a scenario where secure or compressed data storage is desired, the system performing process 1000 can record the output of the recurrent neural network in one or more machine-readable data storage devices for later access.

[0133] In some implementations, the complete output of the recurrent neural network is not transmitted or stored. For example, in implementations where the content of the output of the recurrent neural network is activity in the neural network matching a pattern indicating complexity in the activity, only activity matching a relatively complex or higher dimensional activity may be transmitted or stored. By way of example, referring to pattern 500 ( Figure 5 ), in some implementations, only activities that match patterns 515, 520, 525, and 530 are transmitted or stored, while activities that match patterns 505, 510 are ignored or discarded. In this way, lossy processing allows the amount of data transmitted or stored to be reduced at the expense of the integrity of the encoded information.

[0134] Figure 11 1 is a flow chart of process 1100 for decoding a signal using a recurrent artificial neural network based on characterization of activity in the network. Signals can be decoded in a variety of different scenarios, such as, for example, signal reception, decryption, and reading data from storage. Process 1100 can be performed by a system of one or more data processing devices that perform operations according to the logic of one or more machine-readable instruction sets. For example, process 1100 can be performed by the same system of one or more computers that execute software for implementing the recurrent artificial neural network used in process 1100. In some instances, process 1100 can be performed by the same data processing device that performs process 400 and / or process 1000. In some instances, process 1100 can be performed by, for example, a decoder in a signal reception system or a decoder in a data storage system.

[0135] At 1105, the system performing process 1100 receives at least a portion of the output of the recurrent artificial neural network. The specific actions performed at 1105 can reflect the context in which process 1100 is being used. For example, the system performing process 1100 can receive a transmission signal including the output of the recurrent artificial neural network or read a machine-readable data storage device storing the output of the recurrent artificial neural network.

[0136] At 1110, the system performing process 1100 reconstructs the input of the recurrent artificial neural network from the received output. The reconstruction can be performed in a variety of different ways. For example, in some implementations, a second artificial neural network (recurrent or non-recurrent) can be trained to reconstruct the input to the recurrent neural network from the output received at 1105.

[0137] As another example, in some implementations, a decoder that has been trained using machine learning (including but not limited to deep learning) can be trained to reconstruct the input to the recurrent neural network from the output received at 1105.

[0138] As yet another example, in some implementations, inputs into the same recurrent artificial neural network or into a similar recurrent artificial neural network may be iteratively permuted until the output of the recurrent artificial neural network somewhat matches the output received at 1105 .

[0139] In some implementations, process 1100 may include receiving user input specifying the extent to which the input is to be reconstructed, and in response, adjusting the reconstruction accordingly at 1110. For example, the user input may specify that a complete reconstruction is not required. In response, the system performing process 1100 adjusts the reconstruction. For example, in implementations where the content of the output of the recurrent neural network is activity in the neural network that matches a pattern that indicates complexity in the activity, only outputs that characterize activity that matches relatively complex or higher dimensional activity will be used to reconstruct the input. By way of example, referring to pattern 500 ( Figure 5 ), in some implementations, only the activities that match patterns 515, 520, 525, and 530 can be used to reconstruct the input, while the activities that match patterns 505, 510 can be ignored or discarded. In this way, lossy reconstruction can be performed in selected cases.

[0140] In some implementations, processes 1000 and 1100 can be used for peer-to-peer encrypted communication. Specifically, both the transmitter (i.e., encoder) and the receiver (i.e., decoder) can be configured with the same recurrent artificial neural network. There are several ways to customize a shared recurrent artificial neural network to ensure that a third party cannot reverse engineer it and decrypt the signal, including:

[0141] —Structure of recurrent artificial neural networks

[0142] —functional setup of the recurrent artificial neural network, including node states and edge weights,

[0143] —the size (or dimension) of the pattern, and

[0144] —A small fraction of the pattern in each dimension.

[0145] These parameters can be considered to together ensure multiple layers of transmission security.In addition, in some implementations, the decision moment time point can be used as a key to decrypt the signal.

[0146] Although processes 1000 and 1100 are presented in terms of encoding and decoding a single recurrent artificial neural network, processes 1000 and 1100 can also be applied to systems and processes that rely on multiple recurrent artificial neural networks. These recurrent artificial neural networks can be run in parallel or in series.

[0147] As an example of a serial operation, the output of a first recurrent artificial neural network can be used as the input to a second recurrent artificial neural network. The resulting output of the second recurrent artificial neural network is a twice-encoded (or twice-encrypted) version of the input to the first recurrent artificial neural network. Such a serial arrangement of recurrent artificial neural networks can be useful in situations where different parties have different levels of access to information, for example, in a medical record system where patient identity information may be inaccessible to a party that will use and have access to the rest of the medical record.

[0148] As an example of parallel operation, the same information can be fed into multiple different recurrent artificial neural networks. The different outputs of these neural networks can be used, for example, to ensure that the input can be reconstructed with high fidelity.

[0149] While several implementations have been described, various modifications are possible. For example, while implementations generally imply that activity in a recurrent artificial neural network should match a pattern indicating an ordering, this is not necessarily the case. Instead, in some implementations, activity in a recurrent artificial neural network can be consistent with the pattern without necessarily exhibiting activity that matches the pattern. For example, an increase in the likelihood that a recurrent artificial neural network will exhibit activity that matches the pattern can be considered a non-random ordering of activity.

[0150] As yet another example, in some implementations, different sets of patterns can be customized for use in characterizing activity in different recurrent artificial neural networks. The patterns can be customized, for example, based on their effectiveness in characterizing activity in different recurrent artificial neural networks. The effectiveness can be quantified, for example, based on the size of a table or vector representing occurrence counts of the different patterns.

[0151] As yet another example, in some implementations, the patterns used to characterize activity in a recurrent artificial neural network may take into account the strength of connections between nodes. In other words, the patterns previously described herein treat all signaling activity between two nodes in a binary manner (i.e., activity is present or not). This is not necessarily the case. Instead, in some implementations, commensurate with the pattern may require that activity with a certain level or strength of connections be considered indicative of ordered complexity in the activity of the recurrent artificial neural network.

[0152] As yet another example, the content of the output of a recurrent artificial neural network may include activity patterns occurring outside of a time window within which activity in the neural network has distinguishable levels of complexity. For example, read at 1015 and transmitted or stored at 1020 ( Figure 10 )'s recurrent artificial neural network may include responses to, for example, what occurs in graph 905 ( Figure 9 ). By way of example, the output of a recurrent artificial neural network can characterize only the highest-dimensional patterns of activity, regardless of when those patterns of activity occur. As another example, the output of a recurrent artificial neural network can characterize only patterns of activity surrounding cavities, regardless of when those patterns of activity occur.

[0153] Figure 12 、 Figure 13 and Figure 14 is a schematic illustration of a binary form or representation 1200 of a topological structure, such as, for example, a pattern of activity in a neural network. Figure 12 、 Figure 13 and Figure 14 The topologies illustrated in FIG. 1 all include the same information, namely, an indication of the presence or absence of a feature in the graph. The feature may be, for example, activity in a neural network device. In some implementations, activity is identified based on or during a time period during which activity in the neural network has a complexity that is distinguishable from other activity in response to input.

[0154] As shown, binary representation 1200 includes bits 1205, 1207, 1211, 1293, 1294, 1297, and an additional, arbitrary number of bits (indicated by ellipsis "..."). For didactic purposes, bits 1205, 1207, 1211, 1293, 1294, 1297, ... are illustrated as discrete rectangular shapes that are filled or unfilled to indicate the binary value of the bit. In the schematic illustration, representation 1200 appears to be a one-dimensional vector of bits ( Figure 12 、 Figure 13 ) or a two-dimensional matrix of bits ( Figure 14 ). However, representation 1200 differs from a vector, matrix, or other ordered collection of bits in that the same information can be encoded regardless of the order of the bits—that is, regardless of the position of the individual bits within the collection.

[0155] For example, in some implementations, each individual bit 1205, 1207, 1211, 1293, 1294, 1297, ... can represent the presence or absence of a topological feature - regardless of the location of the feature in the graph. By way of example, refer to Figure 2 , a bit such as bit 1207 may indicate AND mode 505 ( Figure 5 ) regardless of whether the activity occurs between nodes 104, 105, 101 or between nodes 105, 101, 102. Thus, while each individual bit 1205, 1207, 1211, 1293, 1294, 1297, ... can be associated with a particular feature, the location of that feature in the graph need not be encoded, for example, by the corresponding position of that bit in representation 1200. In other words, in some implementations, representation 1200 may merely provide an isomorphic topological reconstruction of the graph.

[0156] Furthermore, in other implementations, it is possible that the positions of the individual bits 1205, 1207, 1211, 1293, 1294, 1297, ... do encode information such as, for example, the position of the feature in the graph. In these implementations, the source graph can be reconstructed using representation 1200. However, such encoding is not necessarily present.

[0157] Given that the presence or absence of a topological feature can be represented regardless of its position in the graph, Figure 1 In , at the beginning of representation 1200, bit 1205 occurs before bit 1207, which occurs before bit 1211. Conversely, in Figure 2 and Figure 3 , the order of bits 1205, 1207, and 1211 in representation 1200—as well as the positions of bits 1205, 1207, and 1211 relative to the other bits in representation 1200—has changed. However, binary representation 1200 remains the same—as does the set of rules or algorithm that defines the process for encoding the information in binary representation 1200. As long as the correspondence between bits and features is known, the positions of the bits in representation 1200 are irrelevant.

[0158] In more detail, each bit 1205, 1207, 1211, 1293, 1294, 1297, etc., individually represents the presence or absence of a feature in a graph. A graph is a set of nodes and a set of edges between these nodes. The nodes can correspond to objects. Examples of objects include, for example, artificial neurons in a neural network, individuals in a social network, etc. Edges can correspond to some relationships between objects. Examples of relationships include, for example, structural connections or activities along these connections. In the context of neural networks, artificial neurons can be connected through structural connections between neurons or through the transmission of information along these structural connections. In the context of social networks, individuals can be connected through "friends" or other relationship connections or through the transmission of information (e.g., posts) along such connections. Thus, edges can characterize relatively long-lasting structural features of a group of nodes or relatively short-lived activities that occur within a defined time frame. Furthermore, edges can be directed or bidirectional. Directed edges indicate the directionality of the relationship between objects. For example, the transmission of information from a first neuron to a second neuron can be represented by a directed edge indicating the direction of the transmission. As another example, in a social network, a relationship connection may indicate that a second user will receive information from a first user, but not that the first user will receive information from the second user. In topological terms, a graph can be expressed as a set of unit intervals [0, 1], where 0 and 1 are identified by corresponding nodes connected by edges.

[0159] The feature whose presence or absence is indicated by bits 1205, 1207, 1211, 1293, 1294, 1297 can be, for example, a node, a group of nodes, a group of nodes in multiple groups of nodes, a group of edges, a group of edges in multiple groups of edges, and / or additional hierarchically more complex features (e.g., a group of multiple groups of nodes in multiple groups of nodes). Bits 1205, 1207, 1211, 1293, 1294, 1297 generally indicate the presence or absence of features at different hierarchical levels. For example, bit 1205 can indicate the presence or absence of a node, while bit 1205 can indicate the presence or absence of a group of nodes.

[0160] In some implementations, bits 1205, 1207, 1211, 1293, 1294, and 1297 can represent characteristics of a graph that have a threshold level of some property. For example, bits 1205, 1207, 1211, 1293, 1294, and 1297 can indicate not only the presence of activity in a set of edges, but also the weighting of this activity above or below a threshold level. The weights can, for example, reflect the training of a neural network device for a specific purpose or can be inherent characteristics of the edges.

[0161] The above Figure 5 、 Figure 6 and Figure 8Features whose presence or absence can be represented by bits 1205, 1207, 1211, 1293, 1294, 1297, ... are illustrated.

[0162] The directed simplexes in the sets 500, 600, and 700 view a function graph or a structure graph as a topological space with nodes as points. A structure or activity involving one or more nodes and links corresponding to the simplexes in the sets 500, 600, and 700 can be represented by bits, regardless of the identity of the specific nodes and / or links participating in the activity.

[0163] In some implementations, only some patterns of structure or activity are identified and / or portions of the identified patterns of structure or activity are discarded or otherwise ignored. Figure 5 , structures or activities commensurate with the five-point, four-dimensional simplex pattern 515 inherently include structures or activities commensurate with the four-point, three-dimensional and three-point, two-dimensional simplex patterns 510, 505. For example, Figure 5 Points 0, 2, 3, 4 and points 1, 2, 3, 4 in the four-dimensional simplex pattern 515 are all commensurate with the three-dimensional simplex pattern 510. In some implementations, simplex patterns that contain fewer points—and therefore have lower dimensions—can be discarded or otherwise ignored.

[0164] As another example, only some patterns of structures or activities need to be recognized. For example, in some implementations, only patterns with an odd number of points (3, 5, 7, ...) or an even number of dimensions (2, 4, 6, ...) are used.

[0165] Return to Figure 12 、 Figure 13 、 Figure 14 , the features whose presence or absence is represented by bits 1205, 1207, 1211, 1293, 1294, 1297, ... may not be independent of each other. By way of explanation, if bits 1205, 1207, 1211, 1293, 1294, 1297 represent the presence or absence of a zero-dimensional simplex, each reflecting the presence or activity of a single node, then bits 1205, 1207, 1211, 1293, 1294, 1297 are independent of each other. However, if bits 1205, 1207, 1211, 1293, 1294, 1297 represent the presence or absence of a higher-dimensional simplex, each reflecting the presence or activity of multiple nodes, then the information encoded by the presence or absence of each individual feature may not be independent of the presence or absence of other features.

[0166] Figure 15An example of how the presence or absence of features corresponding to different bits are not independent of each other is schematically illustrated. In particular, a subgraph 1500 is illustrated that includes four nodes 1505, 1510, 1515, and 1520 and six directed edges 1525, 1530, 1535, 1540, 1545, and 1550. In particular, edge 1525 points from node 1525 to node 1510, edge 1530 points from node 1515 to node 1505, edge 1535 points from node 1520 to node 1505, edge 1540 points from node 1520 to node 1510, edge 1545 points from node 1515 to node 1510, and edge 1550 points from node 1515 to node 1520.

[0167] Represents a single bit in 1200 (e.g., Figure 12 、 Figure 13 、 Figure 14 The filled bit 1207 in 1200 may indicate the presence of a directed three-dimensional simplex. For example, such a bit may indicate the presence of a three-dimensional simplex formed by nodes 1505, 1510, 1515, 1520 and edges 1525, 1530, 1535, 1540, 1545, 1550. The second bit in 1200 (e.g., Figure 12 、 Figure 13 、 Figure 14 The filled bits 1293 in the 1207 bit array may indicate the presence of a directed two-dimensional simplex. For example, such a bit may indicate the presence of a two-dimensional simplex formed by nodes 1515, 1505, 1510 and edges 1525, 1530, 1545. In this simple example, the information encoded by bits 1293 is completely redundant with the information encoded by bits 1207.

[0168] Note that the information encoded by position 1293 may also be redundant with the information encoded by yet another position. For example, the information encoded by position 1293 may be redundant with both the third and fourth positions indicating the presence of an additional directed two-dimensional simplex. Examples of these simplexes are formed by nodes 1515, 1520, 1510 and edges 1540, 1545, 1550, and nodes 1520, 1505, 1510 and edges 1525, 1535, 1540.

[0169] Figure 16Schematically illustrates another example of how the presence or absence of features corresponding to different bits are not independent of each other. In particular, a subgraph 1600 is illustrated that includes four nodes 1605, 1610, 1615, 1620 and five directed edges 1625, 1630, 1635, 1640, 1645. The nodes 1505, 1510, 1515, 1520 and the edges 1625, 1630, 1635, 1640, 1645 generally correspond to the subgraph 1500 ( Figure 15 ) in the subgraph 1500 and the edges 1525, 1530, 1535, 1540, 1545. However, in contrast to the subgraph 1500 in which the nodes 1515, 1520 are connected by the edge 1550, the nodes 1615, 1620 are not connected by an edge.

[0170] Represents a single bit in 1200 (e.g., Figure 12 、 Figure 13 、 Figure 14 The unfilled bit 1205 in 1200 may indicate the absence of a directed three-dimensional simplex (such as, for example, a directed three-dimensional simplex containing nodes 1605, 1610, 1615, 1620). Figure 12 、 Figure 13 、 Figure 14 The presence of a two-dimensional simplex can be indicated by the filled bits 1293 in the representation 1200. An exemplary directed two-dimensional simplex is formed by nodes 1615, 1605, 1610 and edges 1625, 1630, 1645. This combination of filled bits 1293 and unfilled bits 1205 provides information indicating the presence or absence of other features (and the state of other bits) that may or may not be present in the representation 1200. In particular, the combination of the absence of a directed three-dimensional simplex and the presence of a directed two-dimensional simplex indicates that at least one edge is not present in:

[0171] a) A possible directed two-dimensional simplex formed by nodes 1615, 1620, 1610 or

[0172] b) A possible directed two-dimensional simplex formed by nodes 1620, 1605, 1610.

[0173] Therefore, the state of the bit representing the presence or absence of any of these possible simplices is not independent of the state of bits 1205, 1293.

[0174] While these examples have been discussed in terms of features having different numbers of nodes and hierarchical relationships, this is not necessarily the case. For example, a representation 1200 comprising a set of bits corresponding only to the presence or absence of, for example, a three-dimensional simplex is possible.

[0175] Using a single bit to indicate the presence or absence of a feature in a graph yields certain properties. For example, the encoding of information is error-tolerant and provides for "graceful degradation" of the encoded information. In particular, the loss of a particular bit (or group of bits) may increase uncertainty about the presence or absence of a feature. However, the likelihood of the presence or absence of a feature can still be assessed based on other bits indicating the presence or absence of neighboring features.

[0176] Likewise, as the number of bits increases, the certainty regarding the presence or absence of a feature increases.

[0177] As another example, as discussed above, the ordering or arrangement of the bits is irrelevant to the isomorphic reconstruction of the graph represented by the bits. All that is required is a known correspondence between the bits and specific nodes / structures in the graph.

[0178] In some implementations, the pattern of activity in the neural network can be represented in 1200 ( Figure 12 、 Figure 13 and Figure 14 ). Typically, the pattern of activity in a neural network is a result of many characteristics of the neural network, such as, for example, the structural connections between nodes of the neural network, the weights between nodes, and a large number of possible other parameters. For example, in some implementations, the neural network may have been trained prior to encoding the pattern of activity in representation 1200.

[0179] However, regardless of whether the neural network is untrained or trained, the response pattern of activity for a given input can be considered a "representation" or "abstraction" of that input in the neural network. Thus, while representation 1200 may appear to be a straightforward-appearing collection of (in some cases, binary) numbers, each of the numbers may encode a relationship or correspondence between a particular input and the associated activity in the neural network.

[0180] Figure 17 、 Figure 18 、 Figure 19 、 Figure 20Schematic illustrations of the use of representations of occurrences of topological structures in activity in neural networks in four different classification systems 1700, 1800, 1900, and 2000. Classification systems 1700 and 1800 each classify representations of patterns of activity in the neural network as part of the classification of input. Classification systems 1900 and 2000 each classify approximations of representations of patterns of activity in the neural network as part of the classification of input. In classification systems 1700 and 1800, the patterns of activity being represented occur in and are read from a source neural network device 1705 that is part of classification systems 1700 and 1800. In contrast, in classification systems 1900 and 2000, the patterns of activity being approximated occur in a source neural network device that is not part of classification systems 1700 and 1800. However, the approximations of the representations of those patterns of activity are read from an approximator 1905 that is part of classification systems 1900 and 2000.

[0181] In more detail, go to Figure 17 , classification system 1700 includes a source neural network 1705 and a linear classifier 1710. Source neural network 1705 is a neural network device that is configured to receive input and present a representation of the occurrence of topological structure in the activity in source neural network 1705. In the illustrated implementation, source neural network 1705 includes an input layer 1715 that receives input. However, this is not necessarily the case. For example, in some implementations, some or all of the input can be injected into different layers and / or edges or nodes throughout source neural network 1705.

[0182] The source neural network 1705 can be any of a variety of different types of neural networks. Typically, the source neural network 1705 is a recurrent neural network, such as, for example, a recurrent neural network that mimics a biological system. In some cases, the source neural network 1705 can mimic the degree to which the morphology, chemistry, and other characteristics of the biological system are simulated. Typically, the source neural network 1705 is implemented on one or more computing devices (e.g., supercomputers) with a relatively high level of computing performance. In such cases, the classification system 1700 will typically be a decentralized system in which the remote classifier 1710 communicates with the source neural network 1705, for example, via a data communication network.

[0183] In some implementations, source neural network 1705 may be untrained, and the represented activity may be the inherent activity of source neural network 1705. In other implementations, source neural network 1705 may be trained, and the represented activity may reflect this training.

[0184] The representation read from source neural network 1705 may be such as representation 1200 ( Figure 12 、 Figure 13 、 Figure 14 ). The representation 1200 can be read from the source neural network 1705 in a variety of ways. For example, in the illustrated example, the source neural network 1705 includes a "reader node" that reads the pattern of activity between other nodes in the source neural network 1705. In other implementations, the activity in the source neural network 1705 is read by a data processing component that is programmed to monitor the activity of the source neural network 1705 for relatively highly ordered patterns. In other implementations, the source neural network 1705 can include an output layer, for example, when the source neural network 1705 is implemented as a feedforward neural network, and the representation 1200 can be read from the output layer.

[0185] Linear classifier 1710 is a device that classifies an object, that is, a representation of a pattern of activity in source neural network 1705, based on a linear combination of the object's features. Linear classifier 1710 includes input 1720 and output 1725. Input 1720 is coupled to receive a representation of the pattern of activity in source neural network 1705. In other words, the representation of the pattern of activity in source neural network 1705 is a feature vector representing features of the input to source neural network 1705, which are used by linear classifier 1710 to classify the input. Linear classifier 1710 can receive the representation of the pattern of activity in source neural network 1705 in a variety of ways. For example, the representation of the pattern of activity can be received as discrete events or as a continuous stream over a real-time or non-real-time communication channel.

[0186] Output 1725 is coupled to output the classification results from linear classifier 1710. In the illustrated implementation, output 1725 is schematically illustrated as a parallel port having multiple channels. This need not be the case. For example, output 1725 can output the classification results via a serial port or a port having combined parallel and serial capabilities.

[0187] In some implementations, the linear classifier 1710 may be implemented on one or more computing devices with relatively limited computing performance. For example, the linear classifier 1710 may be implemented on a personal computer or a mobile computing device such as a smart phone or tablet computer.

[0188] exist Figure 18In the embodiment of the present invention, classification system 1800 includes source neural network 1705 and neural network classifier 1810. Neural network classifier 1810 is a neural network device that classifies an object, that is, a representation of a pattern of activity in source neural network 1705, based on a nonlinear combination of features of the object. In the illustrated implementation, neural network classifier 1810 is a feed-forward network including an input layer 1820 and an output layer 1825. As with linear classifier 1710, neural network classifier 1810 can receive the representation of the pattern of activity in source neural network 1705 in a variety of ways. For example, the representation of the pattern of activity can be received as discrete events or as a continuous stream over a real-time or non-real-time communication channel.

[0189] In some implementations, neural network classifier 1810 can perform inference on one or more computing devices with relatively limited computing performance. For example, neural network classifier 1810 can be implemented on a personal computer or mobile computing device such as a smartphone or tablet computer, for example, in a neural processing unit of such a device. Like classification system 1700, classification system 1800 will typically be a decentralized system in which remote neural network classifier 1810 communicates with source neural network 1705, for example, via a data communication network.

[0190] In some implementations, the neural network classifier 1810 can be, for example, a deep neural network, such as a convolutional neural network including convolutional layers, pooling layers, and fully connected layers. The convolutional layers can generate feature maps, for example, using linear convolution filters and / or nonlinear activation functions. The pooling layers reduce the number of parameters and control overfitting. The calculations performed by different layers in the image classifier 1820 can be defined differently in different implementations of the image classifier 1820.

[0191] exist Figure 19 , classification system 1900 includes a source approximator 1905 and a linear classifier 1710. As discussed further below, source approximator 1905 is a relatively simple neural network that is trained—at an input layer 1915 or elsewhere—to receive input and output a vector that approximates a representation of the topological structure that appears in the pattern of activity in a relatively complex neural network. For example, source approximator 1905 can be trained to approximate a recurrent source neural network, such as, for example, a recurrent neural network that mimics a biological system and includes a degree of morphological, chemical, and other characteristics of the biological system. In the illustrated implementation, source approximator 1905 includes an input layer 1915 and an output layer 1920. Input layer 1915 can be coupled to receive input data. Output layer 1920 is coupled to output an approximation of a representation of the activity within the neural network device for receipt by input 1720 of the linear classifier. For example, output layer 1920 can output representation 1200 ( Figure 12 、 Figure 13 、 Figure 14 ) of approximately 1200'. In addition, Figure 17 、 Figure 18 The schematically illustrated representation 1200 and Figure 19 、 Figure 20 The approximation 1200′ of the representation 1200 schematically illustrated in FIG. 1 is identical. This is for convenience only. Typically, the approximation 1200′ will differ from the representation 1200 in at least some respects. Despite these differences, the linear classifier 1710 may still classify the approximation 1200′.

[0192] Typically, the source approximator 1905 can perform inference on one or more computing devices with relatively limited computing performance. For example, the source approximator 1905 can be implemented on a personal computer or a mobile computing device such as a smartphone or tablet computer, for example, in a neural processing unit of such a device. Typically, and in contrast to the classification systems 1700, 1800, the classification system 1900 will typically be housed within a single housing, for example, where the source approximator 1905 and the linear classifier 1710 are implemented on the same data processing device or on data processing devices coupled by a hardwired connection.

[0193] exist Figure 20 , classification system 2000 includes a source approximator 1905 and a neural network classifier 1810. An output layer 1920 of source approximator 1905 is coupled to output an approximation 1200' of a representation of activity within the neural network device for receipt by input 1820 of neural network classifier 1810. Neural network classifier 1810 can classify approximation 1200' despite any differences between approximation 1200' and representation 1200. Typically, and like classification system 1900, classification system 1900 will typically be housed within a single housing, e.g., where source approximator 1905 and neural network classifier 1810 are implemented on the same data processing device or on data processing devices coupled by a hardwired connection.

[0194] Figure 212 is a schematic illustration of an edge device 2100 including a local artificial neural network that can be trained using a representation of the occurrence of topological structures corresponding to the activity in a source neural network. In this scenario, the local artificial neural network can be, for example, an artificial neural network that executes entirely on one or more local processors that do not require a communications network to exchange data. Typically, the local processors will be connected via a hardwired connection. In some instances, the local processors can be housed within a single housing, such as a single personal computer or a single handheld, mobile device. In some instances, the local processors can be controlled and accessed by a single individual or a limited number of individuals. In effect, by using a representation of the occurrence of topological structures in a more complex source neural network to train (e.g., using supervised learning or reinforcement learning techniques) a simpler and / or less highly trained, but more unique, second neural network, even individuals with limited computing resources and a limited number of training examples can train a neural network as needed. This reduces storage requirements and computational complexity during training, and conserves resources such as battery life.

[0195] In the illustrated implementation, the edge device 2100 is schematically illustrated as a security camera device that includes an optical imaging system 2110 , image processing electronics 2115 , a source approximator 2120 , a representation classifier 2125 , and a communication controller and interface 2130 .

[0196] The optical imaging system 2110 may include, for example, one or more lenses (or even a pinhole) and a CCD device. Image processing electronics 2115 may read the output of the optical imaging system 2110 and may generally perform basic image processing functions. The communication controller and interface 2130 is a device configured to control the flow of information to and from the device 2100. As discussed further below, among the operations that the communication controller and interface 2130 may perform are transmitting images of interest to other devices and receiving training information from other devices. Thus, the communication controller and interface 2130 may include both a data transmitter and a receiver that may communicate via, for example, a data port 2135. The data port 2135 may be a wired port, a wireless port, an optical port, or the like.

[0197] Source approximator 2120 is a relatively simple neural network that is trained to output a vector that approximates a representation of the topological structure that occurs in the pattern of activity in a relatively complex neural network. For example, source approximator 2120 can be trained to approximate a recurrent source neural network, such as, for example, a recurrent neural network that mimics a biological system and includes a degree of morphological, chemical, and other characteristics of the biological system.

[0198] The representation classifier 2125 is a linear classifier or a neural network classifier that is coupled to receive an approximation of a representation of a pattern of activity in the source neural network from the source approximator 2120 and output a classification result. The representation classifier 2125 can be, for example, a deep neural network, such as a convolutional neural network including convolutional layers, pooling layers, and fully connected layers. The convolutional layers can generate feature maps, for example, using linear convolutional filters and / or nonlinear activation functions. The pooling layers reduce the number of parameters and control overfitting. The calculations performed by different layers in the representation classifier 2125 can be defined differently in different implementations of the representation classifier 2125.

[0199] In some implementations, during operation, the optical imaging system 2110 can generate a raw digital image. The image processing electronics 2115 can read the raw image and typically perform at least some basic image processing functions. The source approximator 2120 can receive the image from the image processing electronics 2115 and perform inference operations to output a vector that approximates a representation of the topological structure that occurs in the pattern of activity in a relatively complex neural network. This approximated vector is input to the representation classifier 2125, which determines whether the approximated vector meets one or more sets of classification criteria. Examples include facial recognition and other machine vision operations. If the representation classifier 2125 determines that the approximated vector meets one of the classification criteria, the representation classifier 2125 can instruct the communication controller and interface 2130 to transmit information about the image. For example, the communication controller and interface 2130 can transmit the image itself, the classification, and / or other information about the image.

[0200] Sometimes, it may be desirable to change the classification process. In these cases, the communication controller and interface 2130 can receive a training set. In some implementations, the training set can include raw or processed image data and a representation of the topological structure that appears in the pattern of activity in a relatively complex neural network. Such a training set can be used to retrain the source approximator 2120, for example, using supervised learning or reinforcement learning techniques. In particular, the representation is used as a target answer vector and represents the desired result of the source approximator 2120 processing the raw or processed image data.

[0201] In other implementations, the training set may include representations of topological structures that appear in patterns of activity in relatively complex neural networks and the desired classifications of those representations of topological structures. Such a training set can be used to retrain the neural network representation classifier 2125, for example, using supervised learning or reinforcement learning techniques. In particular, the desired classification is used as the target answer vector and represents the desired outcome of the representation classifier 2125 processing the representation of the topological structure.

[0202] Regardless of whether the source approximator 2120 or the representation classifier 2125 is retrained, inference operations at the device 2100 can be easily adapted to changing situations and goals without requiring large training data sets and time-intensive and computationally intensive iterative training.

[0203] Figure 22 is a schematic illustration of a second edge device 2200 including a local artificial neural network that can be trained using a representation of the occurrence of a topological structure corresponding to activity in a source neural network. In the illustrated implementation, the second edge device 2200 is schematically illustrated as a mobile computing device such as a smartphone or tablet computer. The device 2200 includes an optical imaging system (e.g., on the back of the device 2200, not shown), image processing electronics 2215, a representation classifier 2225, a communication controller and interface 2230, and a data port 2235. These components may have features and perform operations corresponding to the device 2100 ( Figure 21 ) in the optical imaging system 2110, image processing electronics 2115, representation classifier 2125, communications controller and interface 2130, and data port 2135.

[0204] The illustrated implementation of device 2200 additionally includes one or more additional sensors 2240 and a multi-input source approximator 2245. Sensor 2240 can sense one of a number of characteristics of the environment surrounding device 2200 or of device 2200 itself. For example, in some implementations, sensor 2240 can be an accelerometer that senses acceleration experienced by device 2200. As another example, in some implementations, sensor 2240 can be an acoustic sensor, such as a microphone, that senses noise in the environment of device 2200. Still other examples of sensor 2240 include chemical sensors (e.g., "artificial noses," etc.), humidity sensors, radiation sensors, etc. In some cases, sensor 2240 is coupled to processing electronics that can read the output of sensor 2240 (or other information, such as, for example, a contact list or map) and perform basic processing functions. Thus, different implementations of sensor 2240 can have different "modalities" because the physical parameter that is physically sensed varies from sensor to sensor.

[0205] The multi-input source approximator 2245 is a relatively simple neural network that is trained to output a vector that approximates a representation of the topological structure that occurs in the pattern of activity in a relatively complex neural network. For example, the multi-input source approximator 2245 can be trained to approximate a recurrent source neural network, such as, for example, a recurrent neural network that mimics a biological system and includes a degree of morphological, chemical, and other characteristics of the biological system.

[0206] Unlike source approximator 2120, multi-input source approximator 2245 is coupled to receive raw or processed sensor data from multiple sensors and, based on that data, returns an approximation of a representation of the topological structure that emerges in the pattern of activity in a relatively complex neural network. For example, multi-input source approximator 2245 may receive processed image data from image processing electronics 2215 and, for example, acoustic, acceleration, chemical, or other data from one or more sensors 2240. Multi-input source approximator 2245 may be, for example, a deep neural network, such as a convolutional neural network including convolutional layers, pooling layers, and fully connected layers. The computations performed by different layers in multi-input source approximator 2245 may be dedicated to a single type of sensor data or to multiple forms of sensor data.

[0207] Regardless of the specific organization of the multi-input source approximator 2245, the multi-input source approximator 2245 will return an approximation based on raw or processed sensor data from multiple sensors. For example, processed image data from the image processing electronics 2215 and acoustic data from the microphone sensor 2240 can be used by the multi-input source approximator 2245 to approximate a representation of the topological structure that would emerge in the pattern of activity in a relatively complex neural network receiving the same data.

[0208] Sometimes, it may be desirable to change the classification process at device 2200. In these cases, the communication controller and interface 2230 can receive a training set. In some implementations, the training set can include raw or processed image, sound, chemical, or other data and representations of topological structures that appear in patterns of activity in relatively complex neural networks. Such a training set can be used to retrain the multi-input source approximator 2245, for example, using supervised learning or reinforcement learning techniques. In particular, the representation is used as a target answer vector and represents the desired result of the multi-input source approximator 2245 processing the raw or processed image or sensor data.

[0209] In other implementations, the training set may include representations of topological structures that appear in patterns of activity in relatively complex neural networks and the desired classifications of those representations of topological structures. Such a training set can be used to retrain the neural network representation classifier 2225, for example, using supervised learning or reinforcement learning techniques. In particular, the desired classification is used as the target answer vector and represents the desired outcome of the representation classifier 2225 processing the representation of the topological structure.

[0210] Regardless of whether the multi-input source approximator 2245 or the representation classifier 2225 is retrained, inference operations at the device 2200 can be easily adapted to changing situations and goals without requiring large training data sets and time-intensive and computationally intensive iterative training.

[0211] Figure 23 is a schematic illustration of a system 2300 in which a local neural network can be trained using a representation of the occurrence of a topological structure corresponding to activity in a source neural network. The target neural network is implemented on a relatively simple, less expensive data processing system, while the source neural network can be implemented on a relatively complex, more expensive data processing system.

[0212] System 2300 includes a variety of devices 2305 having local neural networks, a telephone base station 2310, a wireless access point 2315, a server system 2320 and one or more data communication networks 2325.

[0213] Local neural network device 2305 is a device configured to process data using a computationally-relatively-intensive target neural network. As illustrated, local neural network device 2305 can be implemented as a mobile computing device, a camera, an automobile, or any of a wide variety of other appliances, fixtures, and mobile components, as well as different makes and models of devices within each category. Different local neural network devices 2305 can belong to different owners. In some implementations, access to the data processing functionality of local neural network device 2305 will generally be restricted to those owners and / or their designations.

[0214] The local neural network devices 2305 can each include one or more source approximators that are trained to output vectors that approximate representations of topological structures that appear in patterns of activity in a relatively complex neural network. For example, the relatively complex neural network can be a recurrent source neural network, such as, for example, a recurrent neural network that mimics a biological system and includes a degree of morphological, chemical, and other characteristics of the biological system.

[0215] In some implementations, in addition to processing data using the source approximator, the local neural network device 2305 can also be programmed to retrain the source approximator using a representation of the topological structure that emerges in the pattern of activity in the relatively complex neural network as a target answer vector. For example, the local neural network device 2305 can be programmed to perform one or more iterative training techniques (e.g., gradient descent or stochastic gradient descent). In other implementations, the source approximator in the local neural network device 2305 is trainable, for example, by a dedicated training system or by a training system installed on a personal computer that can interact with the local neural network device 2305 to train the source approximator.

[0216] Each local neural network device 2305 includes one or more wireless or wired data communication components. In the illustrated implementation, each local neural network device 2305 includes at least one wireless data communication component, such as a mobile phone transceiver, a wireless transceiver, or both. The mobile phone transceiver is capable of exchanging data with a telephone base station 2310. The wireless transceiver is capable of exchanging data with a wireless access point 2315. Each local neural network device 2305 may also be capable of exchanging data with a peer mobile computing device.

[0217] Telephone base station 2310 and wireless access point 2315 are connected for data communication with one or more data communication networks 2325 and can exchange information with server system 2320 via the network. Therefore, local neural network device 2305 is typically also in data communication with server system 2320. However, this is not necessarily the case. For example, in implementations where local neural network device 2305 is trained by other data processing devices, local neural network device 2305 need only be in data communication with these other data processing devices at least once.

[0218] Server system 2320 is a system of one or more data processing devices that are programmed to perform data processing activities according to one or more sets of machine-readable instructions. The activities may include providing a training set to a training system for mobile computing device 2305. As discussed above, the training system may be within the mobile local neural network device 2305 itself or on one or more other data processing devices. The training set may include a representation of the occurrence of the topological structure corresponding to the activity in the source neural network and the corresponding input data.

[0219] In some implementations, server system 2320 also includes the source neural network. However, this need not be the case, and server system 2320 can receive the training set from another system of data processing devices that implements the source neural network.

[0220] In operation, after server system 2320 receives a training set (from a source neural network found at server system 2320 itself or elsewhere), server system 2320 can provide the training set to a trainer that trains mobile computing device 2305. The source approximator in target local neural network device 2305 can be trained using the training set so that the target neural network approximates the operation of the source neural network.

[0221] Figure 24 、 Figure 25 、 Figure 26 、 Figure 272 is a schematic illustration of a representation of the occurrence of topological structures in activity in four different systems 2400, 2500, 2600, 2700 using a neural network. The systems 2400, 2500, 2600, 2700 can be configured to perform any of a number of different operations. For example, the systems 2400, 2500, 2600, 2700 can perform object localization operations, object detection operations, object segmentation operations, object detection operations, prediction operations, action selection operations, etc.

[0222] The object localization operation locates objects within an image. For example, a bounding box can be constructed around the object. In some cases, object localization can be combined with object recognition, where the located objects are labeled with appropriate names.

[0223] Object detection is the operation of classifying image pixels as belonging to a specific class (e.g., an object of interest) or not belonging to a specific class. Typically, object detection is performed by grouping pixels and forming a bounding box around the pixel group. The bounding box should fit tightly around the object.

[0224] Object segmentation typically assigns a class label to each image pixel. Therefore, object segmentation is performed on a pixel-by-pixel basis and typically requires only assigning a single label to each pixel, rather than a bounding box.

[0225] Predictive operations seek to draw conclusions that are outside the scope of observed data. Although predictive operations can seek to predict future events (e.g., based on information about past and current states), predictive operations can also seek to draw conclusions about past and current states based on incomplete information about those states.

[0226] The action selection operation seeks to select an action based on a set of conditions. The action selection operation has traditionally been broken down into different approaches, such as symbol-based systems (classical programming), distributed solutions, and reactive or dynamic programming.

[0227] Classification systems 2400 and 2500 each perform a desired operation on a representation of a pattern of activity in a neural network. Systems 2600 and 2700 each perform a desired operation on an approximation of a representation of a pattern of activity in a neural network. In systems 2400 and 2500, the patterns of activity being represented occur in and are read from a source neural network device 1705 that is part of systems 2400 and 2500. Conversely, in systems 2400 and 2500, the patterns of activity being approximated occur in a source neural network device that is not part of systems 2400 and 2500. However, the approximation of the representation of those patterns of activity is read from an approximator 1905 that is part of systems 2400 and 2500.

[0228] In more detail, go to Figure 24 , system 2400 includes source neural network 1705 and linear processor 2410. Linear processor 2410 is a device that performs operations based on linear combinations of features of a representation (or an approximation of such a representation) of a pattern of activity in the neural network. The operations can be, for example, object localization operations, object detection operations, object segmentation operations, object detection operations, prediction operations, action selection operations, etc.

[0229] Linear processor 2410 includes an input 2420 and an output 2425. Input 2420 is coupled to receive a representation of a pattern of activity in source neural network 1705. Linear processor 2410 can receive the representation of the pattern of activity in source neural network 1705 in a variety of ways. For example, the representation of the pattern of activity can be received as discrete events or as a continuous stream over a real-time or non-real-time communication channel. Output 2425 is coupled to output the processing results from linear processor 2410. In some implementations, linear processor 2410 can be implemented on one or more computing devices with relatively limited computing performance. For example, linear processor 2410 can be implemented on a personal computer or a mobile computing device such as a smartphone or tablet computer.

[0230] Go to Figure 24 , system 2400 includes source neural network 1705 and linear processor 2410. Linear processor 2410 is a device that performs operations based on linear combinations of features of a representation (or an approximation of such a representation) of a pattern of activity in the neural network. The operations can be, for example, object localization operations, object detection operations, object segmentation operations, prediction operations, action selection operations, etc.

[0231] Linear processor 2410 includes an input 2420 and an output 2425. Input 2420 is coupled to receive a representation of a pattern of activity in source neural network 1705. Linear processor 2410 can receive the representation of the pattern of activity in source neural network 1705 in a variety of ways. For example, the representation of the pattern of activity can be received as discrete events or as a continuous stream over a real-time or non-real-time communication channel. Output 2525 is coupled to output the processing results from linear processor 2410. In some implementations, linear processor 2410 can be implemented on one or more computing devices with relatively limited computing capabilities. For example, linear processor 2410 can be implemented on a personal computer or a mobile computing device such as a smartphone or tablet computer.

[0232] exist Figure 25In the embodiment of the present invention, classification system 2500 includes source neural network 1705 and neural network 2510. Neural network 2510 is a neural network device that is configured to perform an operation based on a nonlinear combination of features of a representation of a pattern of activity in the neural network (or an approximation of such a representation). The operation can be, for example, an object localization operation, an object detection operation, an object segmentation operation, a prediction operation, an action selection operation, etc. In the illustrated implementation, neural network 2510 is a feedforward network including an input layer 2520 and an output layer 2525. As with linear processor 2410, neural network 2510 can receive the representation of the pattern of activity in source neural network 1705 in a variety of ways.

[0233] In some implementations, neural network 2510 can perform inference on one or more computing devices with relatively limited computing performance. For example, neural network 2510 can be implemented on a personal computer or mobile computing device such as a smartphone or tablet computer, for example, in a neural processing unit of such a device. Like system 2400, system 2500 will generally be a decentralized system in which remote neural network 2510 communicates with source neural network 1705, for example, via a data communication network. In some implementations, neural network 2510 can be, for example, a deep neural network, such as a convolutional neural network.

[0234] exist Figure 26 , system 2600 includes source approximator 1905 and linear processor 2410. Despite any differences between approximation 1200' and representation 1200, processor 2410 may still perform operations on approximation 1200'.

[0235] exist Figure 27 , system 2700 includes source approximator 1905 and neural network 2510. Despite any differences between approximation 1200' and representation 1200, neural network 2510 can still perform operations on approximation 1200'.

[0236] In some implementations, the systems 2600, 2700 may be implemented on edge devices such as, for example, edge devices 2100, 2200 ( Figure 21 、 Figure 22 In some implementations, the systems 2600, 2700 may be implemented as systems (such as the system 2300 ( Figure 23 )) is part of a system in which a local neural network can be trained using a representation of the occurrence of a topological structure corresponding to activity in a source neural network.

[0237] Figure 28is a schematic illustration of a reinforcement learning system 2800 that includes an artificial neural network that can be trained using a representation of the occurrence of a topological structure corresponding to activity in a source neural network. Reinforcement learning is a type of machine learning in which the artificial neural network learns from feedback about the consequences of actions taken in response to decisions made by the artificial neural network. A reinforcement learning system moves from one state to another in an environment by performing actions and receiving information that characterizes the new state and a reward and / or regret that characterizes the success (or lack thereof) of the action. Reinforcement learning seeks to maximize the total reward (or minimize the regret) through the learning process.

[0238] In the illustrated implementation, the artificial neural network in the reinforcement learning system 2800 is a deep neural network 2805 (or other deep learning architecture) trained using reinforcement learning methods. In some implementations, the deep neural network 2805 can be a local artificial neural network (such as the neural network 2510 ( Figure 25 、 Figure 27 )) and implemented locally on, for example, a car, airplane, robot, or other device. However, this is not necessarily the case, and in other implementations, the deep neural network 2805 can be implemented on a system of networked devices.

[0239] In addition to the source approximator 1905 and the deep neural network 2805, the reinforcement learning system 2800 also includes an actuator 2810, one or more sensors 2815, and a teacher module 2820. In some implementations, the reinforcement learning system 2800 also includes one or more additional data sources 2825.

[0240] Actuator 2810 is a device that controls a mechanism or system that interacts with environment 2830. In some implementations, actuator 2810 controls a physical mechanism or system (e.g., the steering of a car or the positioning of a robot). In other implementations, actuator 2810 may control a virtual mechanism or system (e.g., a virtual game board or investment portfolio). Thus, environment 2830 may also be physical or virtual.

[0241] Sensors 2815 are devices that measure characteristics of environment 2830. At least some of these measurements characterize the interaction between the controlled mechanism or system and other aspects of environment 2830. For example, when actuator 2810 operates a car, sensor 2815 may measure one or more of the car's speed, direction, and acceleration, the car's proximity to other features, and the responses of other features to the car. As another example, when actuator 2810 controls an investment portfolio, sensor 2815 may measure the value and risk associated with the portfolio.

[0242] Typically, both the source approximator 1905 and the teacher module 2820 are coupled to receive at least some of the measurements obtained by the sensors 2815. For example, the source approximator 1905 may receive the measurement data at the input layer 1915 and output an approximation 1200′ of a representation of the topological structure that emerges in the pattern of activity in the source neural network.

[0243] The teacher module 2820 is a device configured to interpret the measurements received from the sensor 2815 and provide a reward and / or regret to the deep neural network 2805. Rewards are positive and indicate successful control of the mechanism or system. Regret is negative and indicates unsuccessful or less than optimal control. Typically, the teacher module 2820 also provides a characterization of the measurements and the reward / regret for reinforcement learning. Typically, the characterization of the measurements is an approximation (such as approximation 1200') of a representation of the topological structure that appears in the pattern of activity in the source neural network. For example, the teacher module 2820 can read the approximation 1200' output from the source approximator 1905 and pair the read approximation 1200' with a corresponding reward / regret value.

[0244] In various implementations, reinforcement learning does not occur in real time in the system 2800 or during active control of the actuator 2810 by the deep neural network 2805. Instead, training feedback can be collected by the teacher module 2820 and used to reinforce training when the deep neural network 2805 is not actively instructing the actuator 2810. For example, in some implementations, the teacher module 2820 can be remote from the deep neural network 2805 and only in intermittent data communication with the deep neural network 2805. Regardless of whether reinforcement learning is intermittent or continuous, the deep neural network 2805 can be evolved, for example, to optimize reward and / or reduce regret using information received from the teacher module 2820.

[0245] In some implementations, system 2800 also includes one or more additional data sources 2825. Source approximator 1905 may also receive data from data source 2825 at input layer 1915. In these instances, approximation 1200' will result from processing both sensor data and data from data source 2825.

[0246] In some implementations, data collected by one reinforcement learning system 2800 can be used for training or reinforcement learning of other systems (including other reinforcement learning systems). For example, the characterization of the measurements and the reward / regret values ​​can be provided by the teacher module 2820 to a data exchange system that collects such data from various reinforcement learning systems and redistributes the data among them. Furthermore, as discussed above, the characterization of the measurements can be an approximation of a representation of the topological structure that appears in the pattern of activity in the source neural network, such as approximation 1200'.

[0247] The specific operations performed by reinforcement learning system 2800 will of course depend on the specific operating scenario. For example, in a scenario where source approximator 1905, deep neural network 2805, actuator 2810, and sensor 2815 are part of a car, deep neural network 2805 can perform object localization and / or detection operations while steering the car.

[0248] In cases where the data collected by the reinforcement learning system 2800 is used for training other systems or implementing reinforcement learning, the reward / regret values ​​and approximations 1200′ that characterize the state of the environment when performing object localization and / or detection operations can be provided to the data exchange system. The data exchange system can then distribute the reward / regret values ​​and approximations 1200′ to other reinforcement learning systems 2800 associated with other vehicles for use in reinforcement learning at these other vehicles. For example, reinforcement learning can be used to improve object localization and / or detection operations at a second vehicle using the reward / regret values ​​and approximations 1200′.

[0249] However, the operations learned at the other vehicle need not be identical to the operations performed by the deep neural network 2805. For example, an approximation 1200′ based on a reward / regret value based on travel time and input from sensor data characterizing an unexpectedly wet road at a location identified, for example, by the GPS data source 2825, may be used for route planning operations at another vehicle.

[0250] The embodiments of the operations and themes described in this specification can be implemented in digital electronic circuits, or in computer software, firmware or hardware, including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of them. The embodiments of the themes described in this specification can be implemented as one or more computer programs, that is, one or more modules of computer program instructions, which are encoded on a computer storage medium for execution by a data processing device or to control the operation of a data processing device. Alternatively or additionally, the program instructions can be encoded on an artificially generated propagation signal (e.g., a machine-generated electrical signal, optical signal or electromagnetic signal), which is generated to encode information for transmission to a suitable receiver device for execution by a data processing device. The computer storage medium can be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of them or included therein. In addition, although a computer storage medium is not a propagation signal, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagation signal. The computer storage medium can also be, or be included in, one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices).

[0251] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0252] The term "data processing apparatus" encompasses all kinds of apparatus, devices, and machines for processing data, including, by way of example, a programmable processor, a computer, a system on a chip, or a plurality or combination of the foregoing. The apparatus may include dedicated logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or a combination of one or more of these. The apparatus and execution environment may implement a variety of different computing model infrastructures, such as network services, distributed computing, and grid computing infrastructures.

[0253] A computer program (also referred to as a program, software, software application, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files that store one or more modules, subroutines, or portions of code). A computer program may be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[0254] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform actions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can be implemented as, special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

[0255] Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory, or both. The essential elements of a computer are a processor for performing actions in accordance with instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data, or be operatively coupled to receive data from one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data, or to transfer data to one or more mass storage devices (e.g., magnetic, magneto-optical, or optical disks) for storing data, or both. However, a computer need not have such devices. Furthermore, a computer may be embedded in another device such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few. Devices suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and storage devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0256] To provide for interaction with a user, embodiments of the subject matter described herein may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with the user; for example, feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, voice, or tactile input. Additionally, a computer may interact with a user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0257] Although this specification includes many specific implementation details, these details should not be interpreted as limiting the scope of any invention or the content that may be claimed, but should be understood as a description of the features of the specific embodiments of specific inventions. Certain features described in the scene of a separate embodiment in this specification may also be implemented in combination in a single embodiment. Conversely, a plurality of features described in the scene of a single embodiment may also be implemented individually or in any suitable sub-combination in a plurality of embodiments. In addition, although features may be described above as working in certain combinations and even initially claimed, one or more features from the claimed combination may be deleted from the combination in some cases, and the claimed combination may point to a variant of a sub-combination or a sub-combination.

[0258] Similarly, although operations are described in a particular order in the accompanying drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in a continuous order, or that all illustrated operations be performed to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. In addition, the separation of the various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0259] Thus, particular implementations of the subject matter have been described. Other implementations are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve the desired results. Additionally, the processes depicted in the accompanying figures do not necessarily require the specific order shown, or sequential order, to achieve the desired results. In some implementations, multitasking and parallel processing may be advantageous.

[0260] A number of embodiments have been described. However, it will be appreciated that various modifications may be made. For example, although representation 1200 is a binary representation in which each bit individually represents the presence or absence of a feature in the graph, other representations of information are possible. For example, a vector or matrix of multi-valued, non-binary numbers may be used to represent, for example, the presence or absence of a feature and possibly other characteristics of these features. An example of such a characteristic is the weight of the edges of the activities that constitute the feature.

[0261] Accordingly, other implementations are within the scope of the following claims.

Claims

1. A method performed by a data processing device, the method comprising: An input video or audio data signal is encoded by characterizing signal transmission activity in a recurrent artificial neural network and outputting the characterization of the signal transmission activity, the signal transmission activity in the recurrent artificial neural network being responsive to the input video or audio data signal, wherein in the recurrent artificial neural network, nodes operate as accumulators, the method being performed by a data processing device and comprising identifying a blob pattern of the signal transmission activity of the recurrent artificial neural network, wherein the blob pattern of the signal transmission activity surrounds a cavity and the characterization of the signal transmission activity indicates the presence or absence of the blob pattern without encoding the position of the blob pattern in a graph of the recurrent artificial neural network.

2. The method of claim 1 , wherein the method further comprises defining a plurality of time windows during which the signal transmission activity of the recurrent artificial neural network responds to the input video or audio data signal, wherein the cluster pattern of signal transmission activity is identified in each of the plurality of time windows. 3 . The method of claim 2 , wherein the method further comprises identifying a first time window within the plurality of time windows based on a distinguishable likelihood of the blob pattern of signal transmission activity occurring during the first time window. The method of claim 1 , wherein identifying cluster patterns comprises identifying directed clusters of signaling activity. The method of claim 4 , wherein identifying directed clusters comprises discarding or ignoring lower-dimensional directed clusters that exist within higher-dimensional directed clusters.

6. The method according to claim 1, further comprising: classifying the clique patterns into categories; as well as The signal transmission activity is characterized according to a number of occurrences of the blob pattern in corresponding ones of the categories. The method of claim 6 , wherein classifying the clique patterns comprises classifying the clique patterns according to the number of nodes within each clique pattern.

8. The method of claim 1 , further comprising outputting a binary sequence of 0s and 1s from the recurrent artificial neural network, wherein each digit in the sequence represents whether a corresponding pattern of signaling activity in three or more nodes in the artificial neural network is present.

9. The method according to claim 8, further comprising: The recurrent artificial neural network is structured to include reading the number output from the recurrent artificial neural network, and Evolving the structure of the recurrent artificial neural network, wherein evolving the structure of the recurrent artificial neural network comprises: iteratively changing the structure, Characterize the complexity of the patterns of signaling activity in the altered structure, and The characterization of the complexity of the pattern is used as an indication of whether a changed structure is desirable.

10. The method according to claim 1, wherein The method further comprises: Identifying a decision moment in the recurrent artificial neural network based on a determination of the complexity of a pattern of signal transmission activity in the recurrent artificial neural network, wherein a decision moment is a point in time at which the signal transmission activity in the recurrent artificial neural network indicates a completed result of information processing performed by the recurrent artificial neural network in response to the input video or audio data signal, the identification of the decision moment comprising determining a specific time of signal transmission activity having a complexity distinguishable from other signal transmission activity responsive to the input video or audio data signal, and The decision moment and the time-specific reading of the output from the recurrent artificial neural network are identified based on the time-specific identification of the signal transmission activity with a distinguishable complexity.

11. The method of claim 10, further comprising inputting the input video or audio data signal into the recurrent artificial neural network as a data stream and identifying the blob pattern of signal transmission activity during input of the data stream.

12. The method of claim 1 , further comprising evaluating whether the signal transmission activity is responsive to the input video or audio data signal, the evaluating comprising: assessing that a simpler pattern of signal transmission activity occurring earlier after an input event is responsive to the input video or audio data signal but that a more complex pattern of signal transmission activity occurring earlier after the input event is not responsive to the input video or audio data signal; as well as evaluating that the more complex pattern of signal transmission activity occurring later after the input event is responsive to the input video or audio data signal but the simpler pattern of signal transmission activity occurring later after the input event is not responsive to the input video or audio data signal, The simpler pattern of signal transmission activity involves more nodes than the more complex pattern of signal transmission activity.

13. An encoder comprising one or more computers operable to perform operations including the method according to any preceding claim.

14. A signal transmission system comprising the encoder according to claim 13.

15. A data storage system comprising the encoder according to claim 13.

Images