Distance metrics and clustering in recurrent neural networks

Abstract

Distance metrics and clustering in recurrent neural networks. For example, a method includes, in response to an input of first data into a recurrent artificial neural network, determining whether an active topological pattern of a set of topological patterns occurs in the recurrent artificial neural network; and based on the active topological pattern determined to occur in response to the input of the first data, determining a distance between the first data and a second data or a reference.

Description

Technical Field

[0001] This invention relates to distance metrics and clustering in recurrent neural networks, and more specifically to defining distance metrics in relation to the processing results of recurrent neural networks and using these distance metrics for clustering. Background Technology

[0002] Artificial neural networks are devices inspired by the structure and function of biological neural networks. Specifically, artificial neural networks use interconnected systems called nodes to simulate the information encoding and other processing capabilities of biological neural networks. The arrangement and strength of the connections between nodes in an artificial neural network determine the results of information processing or storage performed by the network.

[0003] Neural networks can be trained to generate a desired signal flow within the network and achieve desired information processing or storage results. Generally, training a neural network involves altering the arrangement and / or strength of connections between nodes during the learning phase. Training is directed towards achieving certain processing results. These results should be consistent with a set of examples (i.e., the training set). A neural network can be considered trained when it achieves sufficiently appropriate processing results for a given array of inputs.

[0004] One purpose of training neural networks is clustering. Clustering groups of objects makes objects in one group or cluster more similar to each other by some metric than objects in other groups or clusters. Clustering is part of many data processing pipelines—including, for example, bioinformatics, data mining, data compression, statistical data analysis, and machine learning. Due to the importance of clustering, various clustering algorithms have been developed, and neural networks have been trained to cluster inputs. These algorithms and neural networks are more or less useful in a variety of different contexts. However, it should be noted that there is no objectively "correct" clustering result. More precisely, the desired clustering result depends on what one hopes to achieve.

[0005] A classic example of a neural network trained to perform clustering is the self-organizing map. Competitive learning is used to train the self-organizing map to produce a low-dimensional, discretized representation of the input training samples. This representation is also called a "map". Because training is the basis of the processing performed by the neural network, these networks are generally unable to handle data that deviates in form or type from the data in the training set. Summary of the Invention

[0006] The method and apparatus of this invention use active topological patterns, which appear even in untrained recurrent artificial neural networks, to define a distance metric. The distance metric can be useful, for example, when clustering different inputs. Active topological patterns typically occur over a time period and can be represented in a set of numbers. At least some of these active topological patterns are inherently present due to the structure of the recurrent artificial neural network. Therefore, even untrained recurrent artificial neural networks can be used to cluster different inputs.

[0007] Furthermore, even if some of the active topological patterns appear in an untrained recurrent artificial neural network (RNN), the RNN does not necessarily remain completely untrained. Specifically, the RNN itself can be trained to synthesize patterns that appear in the untrained state. Patterns that appear in the untrained state do not necessarily have to be output from the RNN and fed into a separate classifier. In fact, the same RNN—in which the active topological patterns appear even in the untrained state—can be trained to perform clustering activities.

[0008] In one aspect, methods, systems, and apparatuses for clustering inputs are described—including computer programs encoded on a computer storage medium. For example, one method includes: determining whether active topological patterns in a set of topological patterns occur in the recurrent artificial neural network in response to input to the first data; and determining a distance between the first data and second data or a reference based on the active topological patterns determined to occur in response to the input to the first data. Corresponding systems and apparatuses—including computer programs encoded on a computer storage medium—are also possible.

[0009] In another aspect, one method includes: receiving information about a topological pattern of an activity occurring in a recurrent artificial neural network in response to input to a plurality of different elements; calculating a distance between the different elements based on the occurrence of the topological pattern of the activity; and adjusting one or more characteristics of the recurrent artificial neural network to increase or decrease the distance. Corresponding systems and apparatus—including computer programs encoded on a computer storage medium—are also possible.

[0010] These and other aspects may include one or more of the following features. It can be determined whether the topological pattern of activities in the set occurs in the recurrent artificial neural network in response to input to the second data. The distance between the first data and the second data is determined by comparing the topological pattern of activities determined to occur in response to the input to the first data with the topological pattern of activities determined to occur in response to the input to the second data. This distance may be determined between the first data and the centroid of a cluster or a cluster boundary.

[0011] The occurrence of the active topological pattern can be represented in a set of binary or multi-valued numbers, each number indicating whether a corresponding topological pattern has occurred. A distance metric can be used to determine the distance, treating a first subset and a second subset of the active topological patterns in the set differently. For example, the first subset of topological patterns can be weighted to indicate distance more strongly than the second subset. The first and second data can be input data of multiple categories. The first subset of topological patterns only needs to include topological patterns appearing in a region of the recurrent artificial neural network primarily perturbed by a single category of input data. Each category of input data can originate from a different sensor. The single category of input data only needs to originate from the first of the sensors. The recurrent artificial neural network can be trained, and the second subset of topological patterns only needs to include topological patterns appearing in a region reflecting the training of the recurrent artificial neural network. The recurrent artificial neural network can also be untrained. The complexity of the topological patterns in the second subset of topological patterns can be higher than the complexity of the topological patterns in the first subset. At least some of the topological patterns in the first subset may be included in the topological patterns in the second subset. It can be repeatedly determined whether the topological patterns of activities in the set occur in the recurrent artificial neural network in response to input data to the recurrent artificial neural network, and the input data—including the first and second data—can be clustered based on the distance between the data. The distance is determined by comparing the corresponding topological patterns of activities determined to occur in response to the input data. The topological patterns of activities may include simplex patterns of activity occurrence. For example, the simplex pattern may enclose a cavity. A specific timing of activities having complexity distinguishable from other activities responding to the input can be determined, and the topology can be identified based on the specific timing of activities having said distinguishable complexity.

[0012] The distance between elements can be determined by comparing topological patterns of activities identified as occurring in response to input of a first element among the elements with topological patterns of activities identified as occurring in response to input of a second element among the elements. The distance can be calculated using a distance metric that differently processes a first subset and a second subset of topological patterns of activities in the set. The distance calculation can weight the first subset of topological patterns to indicate distance more strongly than the second subset. The complexity of the topological patterns in the second subset of topological patterns can be lower than the complexity of the topological patterns in the first subset. Information identifying the occurrence of topological patterns of activities can include a first binary vector for the first element and a second binary vector for the second element. The distance can be calculated by computing the distance between the binary vectors. The recurrent artificial neural network can be untrained.

[0013] Details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the invention will become apparent from the specification and drawings. Attached Figure Description

[0014] Figure 1 This is a schematic representation of one implementation of an artificial neural network system that abstracts and clusters data from multiple different sensors.

[0015] Figure 2 and Figure 3 It is a representation of patterns of activity that can be recognized and “read” from neural networks to generate sets of numbers.

[0016] Figure 4 It is a graph that represents the occurrence of a pattern as a function of time.

[0017] Figure 5 It is a schematic representation of a method for inputting data from different sensors into a neural network and processing the input data.

[0018] Figure 6 This is a schematic representation of how a recurrent neural network can be trained to produce a set of numbers, which is tailored to achieve a desired distance metric in a specific context.

[0019] Figure 7 It is a schematic representation of using a set of numbers to calculate a distance metric, where each number represents the presence or absence of a corresponding topological pattern of activity in a recurrent neural network.

[0020] The same reference numerals in the various figures indicate the same elements. Detailed Implementation

[0021] The biological brain processes different types of input—including, for example, visual, audio, and proprioceptive input. Despite the diversity of inputs, the biological brain can extract different attributes from them and respond accordingly, such as engaging in appropriate movement or conceptual abstraction. Neuroscientists have discovered a rich variety of subsystems in the brain involved in these processes. These include, for example, specific cells in the visual system that encode edges with specific orientations; "place cells" in the hippocampus that become active when an animal is in a precise spatial location; and cells that even encode abstract but selective concepts (the so-called "grandmother cells"). Furthermore, these subsystems can encode temporal relationships, spatial relationships, conceptual relationships, and / or hierarchical relationships. In short, a portion of the innate function of the biological brain can be considered as a combination of specific clusters and non-specific clusters—some of which are innate and some of which are learned.

[0022] In some cases, recurrent artificial neural networks (RNNs) can be constructed to embody at least some of the properties of a biological brain. Such RNNs can incorporate innate, specific structures of the real brain and encode nonlinear projections useful for different types of computation and clustering. These specific structures may include, for example, structures that perform pre-clustering and approximate clustering of the inputs and reduce the dimensionality of the input data. These specific structures may also include, for example, structures that perform direct and specific clustering of specific concepts—without prior learning of those concepts. As previously mentioned, the RNN itself can be trained to synthesize patterns in these structures, and these patterns emerge without training for subsequently learned clustering.

[0023] Therefore, unlike mathematical clustering algorithms that require an explicit definition of the distance function, a predefined distance function is unnecessary for such recurrent artificial neural networks (RNNs). Instead, the distance function is implicitly encoded by the wiring of the nodes and their functional properties. In fact, different nodes and different structures of an RNN can encode different distance functions. As a result, some structures or states of an RNN can cluster the input in a manner similar to hierarchical clustering, while other structures or states can—simultaneously—cluster the input in a manner similar to density-based clustering.

[0024] Even in untrained recurrent artificial neural networks, the topological patterns of activity thus represent clusters of different inputs. A topological pattern represents the state of the recurrent artificial neural network in response to input perturbations. Topological patterns filter out unnecessary or redundant characteristics of the response and make abstractions useful for clustering accessible. As discussed further below, in some cases, decision moments can be identified to further ensure that unnecessary or redundant characteristics of the response are filtered out and not used for clustering.

[0025] Topological patterns can be read from recurrent artificial neural networks as, for example, a set of (generally binary) numbers, each of which represents the presence or absence of a corresponding topological pattern of activity in the network. Each of the numbers represents a specific abstraction performed by the network on the input data. As discussed further below, each abstraction can be arbitrarily complex. Thus, a single number can encode an arbitrarily complex abstraction, and a set of numbers can convey a set of abstractions, each with an arbitrary level of complexity.

[0026] Furthermore, the set of numbers allows distance metrics to be defined. In other words, the distance between two different sets of numbers that arise in response to different inputs can be used to define the distance between those inputs. For example, a distance metric can be as simple as counting the number of simplexes of certain types and calculating, for example, the Hamming distance. As another example, a distance metric can be relatively more complex, such as the earth mover distance calculated over different simplex orders and their topographic appearance.

[0027] Figure 1 This is a schematic representation of one implementation of an artificial neural network system 100 that abstracts and clusters input data 50. The neural network system 100 includes a set of inputs 105, a recurrent neural network 110 itself, a set of outputs 115, and a distance calculator 125 dedicated to calculating the distance or (non)similarity between the abstractions of data 50 output by the outputs 115.

[0028] In the illustrated embodiment, the recurrent neural network 110 receives data 50 from multiple different sensors via input 105. Sensors may be, for example, transducers that convert different physical properties into data or devices that simply sense data, such as, for example, devices that sense the content of a document or data stream. In other embodiments, the recurrent neural network 110 receives data 50 from a single sensor. In any case, the abstraction of data 50 by the neural network 110 can be read from output 115 as, for example, a set of (generally binary) numbers 120, each representing the presence or absence of a corresponding topological pattern of activity in the neural network 110 in response to the input data 50. These response patterns of activity represent a specific abstraction performed by the neural network 110 on the input data 50. As discussed further below, the abstraction can be arbitrarily complex. Thus, a single number can encode an arbitrarily complex abstraction, and a set of numbers can convey a set of abstractions, each with an arbitrary level of complexity.

[0029] Furthermore, the topological patterns of the activities—and their representation in the number 120—can be “general”: in a sense, they do not depend on the source of the data input into the neural network. More precisely, the topological patterns of the activities express abstract properties of the data 50 input into the neural network 110—regardless of the source of that data.

[0030] More specifically, the data 50 can originate from different sensors and have different formats. For example, some types of data 50 (e.g., video or audio data) can change relatively quickly over time, while other types of data 50 (e.g., still images or temperature) can change relatively slowly or not at all. Despite the different sources and formats, the neural network 110 can still abstract characteristics from the data. For example, the neural network 110 can abstract: - Physical characteristics (e.g., color, shape, orientation, speed). - Category (e.g., car, cat, dog), and / or - Abstract qualitative characteristics (e.g., "alive" vs. "dead", "smooth" vs. "rough", "living" vs. "inanimate", "hot" vs. "cold", "open" vs. "closed").

[0031] The topological patterns of the activity—and their representation in the digital 120—are the responses of the neural network 110 to diverse data 50. Typically, multiple topological patterns of the activity will emerge in response to a single input, whether the input is discrete (e.g., a still photograph or a single reading from a transducer measuring physical parameters) or continuous (e.g., a video or audio stream).

[0032] Sometimes, neural network 110 will respond to inputs from data 50 originating from different sensors using one or more of the same topological patterns, even if other topological patterns are different. For example, neural network 110 can respond to a temperature reading or a still image of a desert using a topological pattern representing a qualitative assessment of "heat," even if other topological patterns are also part of the response to each input. Similarly, neural network 110 can respond to the end of a musical piece or a still image of a plate with debris using a topological pattern representing a qualitative assessment of "completed," even if other topological patterns are also part of the response to each input. Therefore, sometimes the same characteristics can be abstracted from data with different sources and different formats.

[0033] Sometimes, the neural network 110 will respond to inputs from data 50 originating from different sensors using one or more topological patterns that represent the characteristics of the data from those sensors, either synthesized or fused. In other words, a single such pattern can represent an abstraction of the same characteristics present in different types of data. Generally, the fusion or synthesis of data from different sensors will work to result in the emergence of such patterns or an increase in the intensity of the activity of such patterns. In other words, data from different sensors can serve as “corroborating evidence” of the same characteristics present in diverse input data.

[0034] In some cases, a topological pattern representing the synthesis or fusion of characteristics from different sensors only emerges when certain characteristics are present in the data from different sensors. The neural network 110 can effectively act as an AND gate, requiring certain characteristics from the data from different sensors to produce certain patterns of activity. However, this is not always the case. Instead, the magnitude of the activity forming a pattern can be increased, or the specific duration of the activity can be shortened, in response to data from different sensors. In fact, the topological pattern of the activity—and its representation in digit 120—represents an abstraction of the characteristics of data 50 in a very rich state space. In other words, the topological pattern of the activity and its representation are not necessarily predefined “outcomes” of processing the input data, in a sense that, for example, yes / no classification is a predefined outcome produced by a classifier, a set of related inputs is a predefined outcome produced by a clustering device, or a prediction is a predefined outcome produced by a prediction model. More precisely, the topological pattern is an abstraction of the characteristics of the input data in a richer state space. While the state space can sometimes include abstractions such as yes / no classification, it is not limited to those predefined outcomes.

[0035] Furthermore, topological patterns can abstract only a portion of the characteristics of the input data (e.g., a specific region of an image, a specific moment in a video or audio stream, or a specific detail of the input such as a pixel), rather than the entirety of the input data. Therefore, the abstracted state space is not limited to predefined types of results (e.g., classification, clustering, or prediction), nor is it limited to an abstraction of the entire input data. More precisely, topological patterns are tools that allow for the reading of processing performed by high-dimensional, nonlinear, cyclic dynamic systems (i.e., neural networks 110). Topological patterns extract correlations from data from different sensors appearing in neural networks 110—including correlations that fuse the data into a more complete "whole."

[0036] When input data 50 is received, neural network 110 will respond with a certain activity. This activity will include: - Activities that are incompatible with the defined topology pattern, and - Indeed, activities that are consistent with the defined topology pattern.

[0037] Activities in neural network 110 that are incompatible with the defined topological pattern can, in some cases, be incorrect or incomplete abstractions of the characteristics of the input data 50. Conversely, activities in neural network 110 that are indeed compatible with the topological pattern can abstract different characteristics of the input data 50. Each of these abstracted characteristics may be more or less useful, depending on the application. By restricting the number 120 to a representation of certain topological patterns, both incorrect or incomplete abstractions and abstractions of characteristics irrelevant to a particular application can be "filtered out" from the number 120.

[0038] In the illustrated embodiment, data 50 includes one or more of the following: sound data 60 from, for example, a microphone; still image data 62 from, for example, a still camera; video data 64 from, for example, a video camera; and temperature data 66 from, for example, a temperature sensor. This is for illustrative purposes only. Data 50 does not necessarily include one or more of the following: sound data 60, still image data 62, video data 64, and temperature data 66. Furthermore, data 50 may include one or more of various other different types of data—including, for example, pressure data, chemical composition data, acceleration data, electrical data, position data, etc. As discussed further below, data 50 from a sensor may undergo one or more processing actions before being input into the neural network 110. Examples of such processing actions include, for example, nonlinear processing in an artificial neural network device.

[0039] As discussed above, the topological patterns that emerge in neural network 110 are abstractions of the characteristics of input data 50 within a rich state space. If data 50 is restricted to originating from a small number of sensors, neural network 110 may not be able to abstract the data from those sensors in some way. For example, neural network 110 may not be able to abstract temperature data 66 itself into patterns of activity corresponding to spatial characteristics such as shape or orientation. However, when data from different sensors is input into neural network 110, the perturbations induced by the diverse input data 50 meet each other and can collectively influence the activity within neural network 110. As a result, neural network 110 can abstract input data 50 into different or more deterministic patterns of activity.

[0040] For example, there may be a degree of uncertainty associated with the presence or absence of a pattern. If data 50 includes data from a diverse range of sensors, both the diversity and certainty of the patterns can increase as data 50 from different sensors is synthesized or fused in the neural network 110. By analogy, a passenger sitting in a train at a train station can look out the window and see an adjacent train that appears to be moving. The same passenger can also, for example, feel forward pressure from their seat. The fusion or synthesis of this information increases the passenger's certainty that their train is moving, rather than an adjacent train. When the neural network receives diverse input data 50, the perturbations induced by that data can be collectively abstracted into different or more deterministic patterns of activity.

[0041] The ability of the neural network 110 to process input data 50 from diverse sensors also provides a degree of robustness for abstracting that data. For example, one sensor in a group might become inaccurate or even malfunction, while the neural network 110 can still continue abstracting the data 50 from the other sensors. Often, the neural network 110 abstracts the data 50 from the other sensors into the same pattern of activity that would occur if all sensors functioned as designed. However, in some cases, the determinism of those abstractions may decrease. Even if such problems arise, the abstraction can continue.

[0042] In the illustrated implementation, input 105 is schematically represented as a well-defined input layer of nodes, each passively relaying the input to one or more locations in neural network 110. However, this is not necessarily the case. For example, in some implementations, one or more of inputs 105 may scale a portion or all of the input data or perform other preliminary processing before the data is transmitted to neural network 110. As another example, data 50 may be injected into different layers and / or edges or nodes throughout neural network 110, i.e., without such a formal input layer. For example, a user may specify that data will be injected into specific nodes or links distributed throughout network 110. As another example, neural network 110 is not necessarily limited to receiving input in a known, previously defined manner (e.g., always injecting the first bit into the first node, the second bit into the second node, and so on). Instead, a user may specify that certain bits of the data will be injected into edges rather than nodes, and the order of injection need not follow the order in which the bits appear, or a combination of these and other parameters. However, for convenience, the representation of input 105 as an input layer will be maintained herein.

[0043] Neural network 110 is schematically illustrated as a recurrent neural network. In a recurrent neural network, the connections between nodes form a directed graph along a time series, and the network exhibits time-dynamic behavior. Generally, neural network 110 is a relatively complex neural network modeled on biological systems. In other words, neural network 110 itself can model the degree to which the morphology, chemistry, and other properties of biological systems are represented. Generally, neural network 110 modeled on biological systems is implemented on one or more computing devices with relatively high computational performance.

[0044] Compared to, for example, conventional feedforward neural networks, the neural network 110 modeled on biological systems can display background or other activities that are not responsive to the input data 50. In fact, activity can exist in such a neural network 110 even in the absence of input data 50.

[0045] However, when input data 50 is received, the recurrent neural network 110 will be perturbed. Since the response of such a neural network 110 to the perturbation may depend in part on the state of the neural network 110 when input data 50 is received, the response of such a neural network 110 to the input data 50 may also depend on the background or other activities already present in the neural network 110.

[0046] However, even though such activity in a neural network is not only in response to the input of data 50, but also in response to the input data 50, the number 120 can represent the presence or absence of a topological structure in a pattern of activity in response to the input data, even in relatively complex recurrent neural networks modeled on biological systems.

[0047] In the illustrated embodiment, output 115 is schematically represented as a multi-node output layer. However, output 115 need not be a multi-node output layer. For example, output node 115 could be a single “reader node” that identifies the occurrence of a specific pattern of activity at a particular set of nodes in neural network 110 and thus reads the output of neural network 110. A reader node is activated if and only if the activity at a particular set of nodes meets a specific time criterion (and possibly amplitude or other). For example, output node 115 could be connected to a set of nodes in neural network 110 and the presence or absence of a topology could be indicated based on, for example, the activity level of each individual node across a corresponding threshold activation level, a weighted sum of the activity levels of those nodes across a threshold activation level, or a nonlinear combination of the activity levels of those nodes across a threshold activation level.

[0048] The output received via output 115 is a set of numbers 120, each number representing the presence or absence of a corresponding pattern of activity in neural network 110—which can be implemented as a relatively complex neural network mimicking the characteristics of biological systems. Set 120 is only schematically illustrated, and set 120 can be, for example, a one-dimensional vector of numbers, a two-dimensional matrix of numbers, or other sets of numbers. Generally, the numbers in set 120 will be binary and indicate the presence or absence of a pattern of activity in a yes / no manner. However, this is not always the case. Instead, in some embodiments, the numbers representing 120 will be multi-valued. Values ​​can represent characteristics of the presence or absence of a corresponding pattern of activity in neural network 110. For example, values ​​can indicate the intensity of the activity or the statistical probability that a particular pattern of activity actually exists. By way of example, activity with relatively large amplitude or occurring within a relatively short time window can be considered to indicate that a particular operation has been performed or is very likely to have been performed. In contrast, activity with relatively small amplitude or occurring over a relatively long time period can be considered to indicate that a particular operation has not been performed.

[0049] The information in set 120 is represented holographically: in a sense, information about the value of a single digit is distributed across the values ​​of the other digits in set 120. In other words, a random subset of the digits in set 120 also contains information about the abstraction performed by neural network 110 on input 50, only at a lower resolution than would exist if all digits in set 120 were present. As discussed further below, different topological patterns have different degrees of complexity. Some relatively more complex patterns can include relatively less complex patterns, and simple patterns can be assembled into more complex patterns. Furthermore, a relatively high level of abstraction represented by some digits can be related to other abstractions represented by other digits to some extent. For example, a digit representing the occurrence of an abstract qualitative characteristic like "alive" can be related to a digit representing a qualitative characteristic like "having life." Therefore, information about the occurrence of some topological patterns inherently includes some information about the occurrence of other topological patterns.

[0050] For convenience, the remainder of this application will refer to the set of 120 numbers as binary bits, and the accompanying drawings will illustrate them as such. However, it should be understood that in all cases, the set of 120 numbers can also be multi-valued to encode aspects of the abstraction performed by the network.

[0051] The distance calculator 125 is a device specifically designed to calculate the distance or (non)similarity between different abstractions of data 50 output by output 115. The distance calculator 125 can use any of a variety of different methods to calculate this distance. For example, the distance calculator 125 can calculate Hamming distance, bulldozer distance, Jaccard similarity measure, or other distance measures. An investigation of distance measures applicable to binary eigenvectors can be found in the publication “Asurvey of Binary Similarity and Distance Measures” by Sung-Seok Choi et al. (Journal of Systemics, Cybernetics and Informatics, pp. 43–48 (2010)). An example selection of appropriate distance measures can be found in Wijaya et al. (BMC Bioinformatics 17:520 (2016)). The contents of both are incorporated herein by reference.

[0052] The distance calculator 125 includes an input 135 and an output 140. The input 135 is coupled to receive a representation of the patterns of activity in the recurrent neural network 110, i.e., a set of numbers 120. The distance calculator 125 can receive the set of numbers 120 in various ways. For example, the numbers in the set 120 can be received as a set of discrete events or as a continuous stream on a real-time or non-real-time communication channel.

[0053] Output 140 is coupled to output a measure of distance relative to the set of numbers 120 received from neural network 110. For example, output 140 may output the measured distance between two sets of numbers 120 that have been output from neural network 110, the measured distance between a set of numbers 120 that has been output from neural network 110 and a set of numbers that has been output from another neural network 110, and / or the measured distance between the set of numbers 120 that has been output from neural network 110 and a reference—such as, for example, the centroid of a cluster or cluster boundary.

[0054] In some implementations, the distance calculator 125 is integrated into the neural network 110. In other words, the neural network 110 can directly output a distance metric. The distance metric can measure, for example, the distance between two different sets of inputs to the neural network 110, the evolution of a single set of inputs to the neural network 110, or the distance between input data and a reference. In some implementations, the distance metric output from the neural network 110 can be passed to another application and / or used to train the neural network 110. The distance metric output from the distance calculator 125 can be used in a variety of different contexts. For example, the distance metric can be used to evaluate generated clusters, determine the optimal number of clusters, identify correlations between attributes within clusters, and estimate the degree of clustering in a dataset. For example, distance metrics can be used to compute the Davies-Bouldin index, the Dunn index, the silhouette coefficient, and the Hopkins statistic.

[0055] The distance metric output from the distance calculator 125 can be useful in these and other contexts. For high-dimensional data, distance becomes less useful for parsing elements and clusters. Specifically, as the number of dimensions increases, all elements in the space appear relatively (un)similar. In other words, when the number of dimensions approaches infinity, the difference between the distance between the nearest neighbor and the distance between the farthest neighbor approaches zero.

[0056] However, the number 120, representing the presence or absence of a corresponding topological pattern in response to the activity in the input data 50, can be valuable for computing distance metrics for high-dimensional data. As a thresholding problem, even the simplest abstraction represented in the number 120 inherently reduces the dimensionality of the input data. Therefore, the number 120 inherently defines the distance between even high-dimensional input data based on the topological patterns of the activity that occur in the recurrent neural network 110 in response to the input data 50.

[0057] Furthermore, as discussed above, the topological patterns of an activity can represent different levels of abstraction. For example, the abstraction represented by the occurrence of a first topological pattern might require the simultaneous occurrence of two other topological patterns representing other "lower-level" abstractions. Thus, a set of numbers can convey a set of abstraction properties that are useful for calculating distance metrics based on diverse levels of abstraction.

[0058] Different levels of abstraction also allow entirely new distance metrics to be defined. For example, a single number 120 indicating that different elements are abstracted to the same relatively high level of abstraction could be a stronger indicator of similarity compared to a large number of numbers 120 representing the abstraction of input data to various relatively low levels of abstraction. This contrasts with other methods, such as Hamming distance, which inherently and similarly handle all information in a feature vector.

[0059] In some implementations, the recurrent neural network 110 can be trained to achieve certain processing results. Generally, training teaches the recurrent neural network 110 to achieve higher levels of abstraction based on combinations of lower-level abstractions. Thus, the measurement of the distance between two elements can be defined by training and training a specific distance metric. The user does not need to have prior knowledge or understanding of the dimensions of the input data related to the processing result or the resulting distance metric. Specifically, the training process itself can customize the recurrent neural network 110 to achieve the desired processing result and provide abstractions related to distance computation.

[0060] Figure 2 It is possible to obtain information from neural network 110 ( Figure 1 It identifies and "reads" to generate a representation of pattern 400 of activities in set 120.

[0061] Pattern 400 is a representation of activity in a recurrent artificial neural network. To read Pattern 400, the functional graph is viewed as a topological space with nodes as points. Activities in the nodes and links corresponding to Pattern 400 can be identified as ordered, regardless of the identity of the specific node and / or link participating in the activity. In the illustrated implementation, Pattern 400 is entirely a directed clique or directed simplice. In such a pattern, activity originates from a source node that transmits signals to every other node in the pattern. In Pattern 400, such a source node is designated as point 0, and the other nodes are designated as points 1, 2, ... Furthermore, in a directed clique or simplex, one of the nodes acts as a sink and receives signals transmitted from every other node in the pattern. In Pattern 400, such a sink node is designated as the highest-numbered point in the pattern. For example, in Pattern 405, the sink node is designated as point 2. In Pattern 410, the sink node is designated as point 3. In Pattern 415, the sink node is designated as point 3, and so on. Therefore, the activities represented by pattern 400 are ordered in a distinguishable manner.

[0062] Each of the patterns in pattern 400 has a different number of points and reflects ordered activities in different numbers of nodes. For example, pattern 405 is a two-dimensional simplex and reflects activities in three nodes, pattern 410 is a three-dimensional simplex and reflects activities in four nodes, and so on. As the number of points in a pattern increases, the ordering and complexity of the activities also increase. For example, for a large set of nodes with a certain level of random activity within a window, some of these activities may coincidentally correspond to pattern 405. However, the random activity will gradually become less and less likely to correspond to the corresponding patterns in patterns 410, 415, 420, ... Therefore, the existence of activities corresponding to pattern 430 indicates a relatively higher ordering and complexity in the activities compared to the existence of activities corresponding to pattern 405.

[0063] Windows of different durations can be defined to represent different activities of varying complexity. For example, a longer duration window can be used to identify activities of pattern 430 compared to identifying activities of pattern 405.

[0064] Figure 3 It is possible to obtain information from neural network 110 ( Figure 1 It identifies and "reads" to generate a representation of the active pattern 300 of the binary digit set 120.

[0065] Pattern 300 is a group of directed cliques or directed simplexes of the same dimension (i.e., having the same number of points), which defines a pattern involving more points than a single clique or simplex and encloses a cavity within the group of directed simplexes.

[0066] By way of example, pattern 305 includes six distinct three-point, two-dimensional pattern 405 that together define a level two homology class, while pattern 310 includes eight distinct three-point, two-dimensional pattern 405 that together define a level two homology class. Each of the three points in patterns 305, 310, and 405 can be considered to enclose a corresponding cavity. The nth Betti number associated with the directed graph provides a count of such homology classes in the topological representation.

[0067] Activities represented by patterns such as pattern 300 indicate a relatively high degree of ranking in a network where such activities are unlikely to occur by random chance. Pattern 300 can be used to characterize the complexity of such activities.

[0068] In some implementations, only some patterns of activity are identified and / or a portion of the identified activity's patterns is discarded or otherwise ignored. For example, see reference... Figure 2 The activities commensurate with the five-point, four-dimensional simplex mode 415 inherently include activities commensurate with the four-point, three-dimensional, and three-point, two-dimensional simplex modes 410 and 405. For example, Figure 2 Points 0, 2, 3, 4 and points 1, 2, 3, 4 in the four-dimensional simplex pattern 415 correspond to the three-dimensional simplex pattern 410. In some implementations, patterns comprising fewer points—and therefore having lower dimensions—may be discarded or otherwise ignored. As another example, it may be necessary to identify only some patterns of activity. For example, in some implementations, only patterns with an odd number of points (3, 5, 7, ...) or an even number of dimensions (2, 4, 6, ...) are identified. Although only some patterns are identified, information about the activity in the neural network can still be represented holographically, i.e., at a lower resolution than when all patterns are identified and / or represented in the output.

[0069] As discussed above, the pattern of activity in response to input data 50 represents a specific operation of arbitrary complexity performed by neural network 110 on that input data 50. In some implementations, the complexity of the operation will be reflected in the complexity of the topological pattern. For example, an operation or abstraction represented by a five-point, four-dimensional simplex pattern 415 may be more complex than an operation or abstraction represented by four-point, three-dimensional, and three-point, two-dimensional simplex patterns 410, 405. In such cases, the numerical representation of the presence of activity indicates the execution of a set of operations or abstractions in neural network 110, each of which has an arbitrary level of complexity.

[0070] Figure 4 It is a specific, schematic representation of an activity pattern with distinguishable complexity at a particular time. Figure 4 The determination of the representation in the middle can be used as a basis for neural network 110 ( Figure 1 The activity pattern is identified or "read" to generate a portion of the binary digit set 120 for execution.

[0071] Figure 4 This includes charts 605 and 610. Chart 605 shows the occurrence of patterns as a function of time along the x-axis. Specifically, each occurrence is schematically represented by vertical lines 606, 607, 608, and 609. The occurrence in each row can be an active match of a corresponding pattern or an instance of a class of patterns. For example, the occurrence in the top row could be an active match of pattern 405 (…). Figure 2 ) instances, the occurrence of the second line can be an active matching pattern 410 ( Figure 2 ) instances, the occurrence of the third line can be an active matching pattern 415 ( Figure 2 Examples of ), and so on.

[0072] Figure 605 also includes dashed rectangles 615, 620, and 625 schematically depicting different time windows when the activity patterns have distinguishable complexity. As shown, during the windows depicted by dashed rectangles 615, 620, and 625, the probability of the activity matching patterns indicating complexity in the recurrent artificial neural network is higher than outside those windows.

[0073] Chart 610 illustrates the complexity associated with these occurrences as a function of time along the x-axis. Chart 610 includes: a first peak of complexity 630, which coincides with the window depicted by dashed rectangle 615; and a second peak of complexity 635, which coincides with the windows depicted by dashed rectangles 620 and 625. As shown, the complexity represented by peaks 630 and 635 is distinguishable from the complexity at a baseline level 640 that can be considered as complexity.

[0074] In some implementations, the time at which the output of a recurrent artificial neural network is read coincides with the occurrence of activity patterns with distinguishable complexity. For example, in Figure 4 In the exemplary context, the output of the recurrent artificial neural network can be read at peaks 630 and 635, that is, during the window period depicted by dashed rectangles 615, 620, and 625.

[0075] In some implementations, not only the content of activity patterns with distinguishable complexity but also the specific time of said activity patterns can be output from a recurrent artificial neural network. Specifically, not only the identity and activity of nodes participating in activities commensurate with the activity pattern, but also the specific time of the activity pattern can be considered the output of the recurrent artificial neural network. Therefore, the identified activity pattern and the specific time at which this decision will be made can represent the result of processing performed by the neural network.

[0076] Figure 5 This is a schematic representation of a method for inputting data 50 from different sensors into a neural network 110 and processing the input data. In the illustrated embodiment, different subsets 105', 105'', 105''' of the network input 105 are dedicated to receiving different types of input data. For example, a first subset 105' may be dedicated to receiving a first type of input data (e.g., data from a first sensor), while a second subset 105'' may be dedicated to receiving a second type of input data (e.g., data from a second sensor).

[0077] In some implementations, corresponding “regions” 505, 510 of the neural network 110 receive different kinds of input data from different subsets 105', 105'', 105''' of the network input 105. For example, in an illustrative example, regions 505, 510 are shown as spatially discrete sets of nodes and edges, with relatively few node-to-node connections between each region. This is not necessarily the case. Rather, the nodes and edges of each region 505, 510 may be spatially distributed within the neural network 110, but still receive a specific kind of input data.

[0078] Regardless of the node distribution in each region 505, 510, the processing in each region 505, 510 is primarily—but not necessarily exclusively—perturbed by the received type of input data. The degree of perturbation can be measured based on the activity occurring in a region with or without the corresponding type of input data. For example, a region primarily perturbed by the first type of input data can respond to that first type of input data in roughly the same way, regardless of whether other types of input data simultaneously perturb the network 110. The processing and abstraction performed by each region 505, 510 are primarily influenced by the received type of input data. However, the topological patterns of activity occurring in each region 505, 510 can be read as a set of numbers 120. The same applies to other regions of the recurrent neural network 110.

[0079] This is schematically represented in the neural network system 500 by separately specifying different subsets 115', 115'', and 115''' of the network output 115. Specifically, subset 115' may be dedicated to outputting numbers representing the topological patterns of activity appearing in region 505 of the neural network 110, while subset 115''' may be dedicated to outputting numbers representing the topological patterns of activity appearing in region 5100 of the neural network 110. However, subset 115'' outputs numbers that do not exist in either region 505 or 510. In fact, the numbers output in subset 115'' can represent the fusion of abstract representations and processing results appearing in regions 505 and 510, or further abstracted to a higher level of complexity.

[0080] For example, a given number in subset 115'' can appear if and only if both one or more numbers in subset 115' and one or more numbers in subset 115''' have certain values. Therefore, the number in subset 115'' can represent any higher level of abstraction—both the abstractions generated in regions 655 and 660 and the input data itself.

[0081] When different regions are primarily perturbed by a single type of input data, the processing in those regions can be customized based on the nature of the input data. For example, the depth of connections and the topology of network loops can be customized based on the input data. In recurrent neural networks modeled on biological systems, neuronal dynamics and synaptic plasticity can also be customized based on the input data. Customization, for example, captures different time scales. For instance, processing in regions customized to handle input data of a relatively rapidly changing type (e.g., video or audio data) can be faster than processing in regions customized to handle input data of a relatively slowly changing type or that does not change at all.

[0082] Furthermore, when different regions of a recurrent neural network are primarily perturbed by a single type of input data, humans are more likely to attribute the representations appearing in the recurrent neural network to specific input data. A representation appearing in a particular region can be attributed to the type of input data that primarily perturbed that region. Once a representation appearing in a particular region is attributed, the higher-level and more complex abstractions that arise in response to that representation can also be more easily understood.

[0083] Furthermore, training can be targeted at the parts of the recurrent neural network that are not primarily perturbed by a single type of input data; that is, at the parts of the recurrent neural network that fuse the processing results of regions primarily perturbed by a single type of input data. In fact, the regions primarily perturbed by a single type of input data will generate a general representation of the input data—not only for use from the output of the recurrent neural network, but also for use with further abstractions and other operations of the recurrent neural network.

[0084] Figure 6 This is a schematic representation of how a recurrent neural network 110 can be trained to produce a set of numbers 120, which is customized to achieve a desired distance metric in a specific context.

[0085] For example, the digits can be customized based on the type of input that will be received by the neural network 110. Customization, for example, can ensure that the neural network 110 is sufficiently sensitive to certain differences in different input data and / or sufficiently insensitive to other differences in the input data. In practice, the digits can be customized to ensure that different sets of digits 120 are sufficiently close to or far from each other (e.g., a “crowding distance” between different sets of digits 120 is desirable).

[0086] In addition to the recurrent neural network 110 and the distance calculator 125, the illustrated implementation of the neural network system 100 also includes a comparator 710 and a training unit 715.

[0087] Comparator 710 is a device configured to compare the distance metric output 140 of distance calculator 125 with a desired output 730. This comparison can produce a cost function or other metric of the proximity of the distance metric output 140 to the desired output 730.

[0088] Training unit 715 is a device configured to modify one or more properties of recurrent neural network 110 to reduce the differences between them based on a metric of how close the distance metric output 140 is to the desired output 730. For example, training unit 715 may add or remove nodes or links from neural network 110, change the weights of links, change the topological pattern represented in digit set 120, or otherwise modify recurrent neural network 110. In implementations where recurrent neural network 110 is a relatively complex neural network modeling a biological system, training unit 715 may modify, for example, the morphology, chemistry, or other properties of the model.

[0089] Training a recurrent neural network 110 to produce a set of digits 120—which produces a desired distance metric—can be beneficial in a variety of different contexts. For example, as discussed above, the neural network 110 can be trained to achieve a desired sensitivity to certain characteristics of the input data. Customizations can be performed, for example, to improve the adversarial resistance of the neural network 110 by making it insensitive to adversarial attacks. As another example, customizations can be performed to ensure that the neural network 110 clusters the input data into a desired number of clusters. As yet another example, customizations can be performed to ensure that the set of digits 120 includes digits relevant to a particular downstream processing—such as, for example, classification or other supervised learning applications. As yet another example, customizations can be performed to adjust the activity threshold in the neural network 110. For example, the threshold can be adjusted based on the amount of “noise” in the environment, where noise is input data that potentially contributes to the activity level in the neural network 110. By adjusting the threshold, the neural network 110 can be made insensitive to noise.

[0090] Figure 7 This is a schematic representation of using a set of 120 numbers to compute a distance metric, each of which represents the presence or absence of a corresponding topological pattern of activity in the recurrent neural network 110. As shown, the set of 120 numbers includes different subsets 115', 115'', 115''' of the network output 115 of the recurrent neural network 110, which are dedicated to outputting in different regions—such as regions 505, 510, 515 (…). Figure 5 The representations appearing in the diagram are as follows: For example, in Figure 5, subsets 115' and 115''' can be used to output relatively "low-level" abstract representations, while subset 115'' can be used to output relatively "high-level" abstract representations.

[0091] As schematically indicated by the dashed lines, the distance calculator 125 can handle low-level and high-level abstractions differently during the calculation of the distance metric. This different handling can be implemented in various different ways. For example, the higher-level abstraction can be weighted more heavily than the lower-level abstraction. As another example, a first distance calculation can be used for the low-level abstraction, while a second distance calculation is used for the high-level abstraction. As yet another example, the calculation of the distance metric can be performed in a multi-step process. For example, in the first step, the high-level abstraction can be used, while the low-level abstraction can be used in the second step. In practice, the weights or other parameters used in calculating the distance metric for the low-level abstraction can be adjusted based on the distance calculated for the high-level abstraction in the first step.

[0092] In this way, not only can a single recurrent neural network generate different levels of abstraction, but the computation of distance metrics can also be adapted to those different levels by processing them differently. This could lead to new methods for computing distance metrics, even for high-dimensional data.

[0093] Treating different levels of abstraction differently can be beneficial in many contexts. For example, different levels of abstraction can be used to pre-cluster large amounts of data or to progressively cluster data step by step, where clustering begins with coarse, unrefined clusters that are progressively refined in successive operations.

[0094] The embodiments of the operations and subjects described in this specification may be implemented as digital electronic circuits, or as computer software, firmware, or hardware—including the structures disclosed in this specification and their structural equivalents—or as a combination of one or more of these. Embodiments of the subjects described in this specification may be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a computer storage medium for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Optionally or additionally, program instructions may be encoded on artificially generated propagating signals (e.g., machine-generated electrical, optical, or electromagnetic signals) generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus. The computer storage medium may be a computer-readable storage device, a computer-readable storage substrate, a random or serial access memory array or device, or a combination of one or more of these, or may be included therein. Furthermore, although the computer storage medium is not a propagating signal, it may be a source or destination of computer program instructions encoded in artificially generated propagating signals. Computer storage media can also be one or more separate physical components or media (e.g., multiple CDs, disks, or other storage devices) or included therein.

[0095] The operations described in this specification can be implemented as operations performed by a data processing apparatus on data stored on one or more computer-readable storage devices or received from other sources.

[0096] The term "data processing apparatus" encompasses all kinds of devices, apparatuses, and machines for processing data, including, by way of example, programmable processors, computers, systems-on-a-chip, or a combination thereof. The apparatus may include special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, cross-platform runtime environments, virtual machines, or combinations thereof. The apparatus and execution environment can implement a variety of different computing model infrastructures, such as network services, distributed computing, and grid computing infrastructures.

[0097] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language—including compiled or interpreted languages, declarative or procedural languages—and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for use in a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program may be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), a single file dedicated to the program in question, or multiple coordination files (e.g., a file storing portions of one or more modules, subroutines, or code). A computer program can be deployed to be executed on a single computer or on multiple computers located at one site or distributed across multiple sites and interconnected via a communication network.

[0098] The processes and logic flows described in this specification can be executed by one or more programmable processors, which execute one or more computer programs to perform actions by manipulating input data and generating outputs. The processes and logic flows can also be executed by special-purpose logic circuitry (e.g., FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits)), and the apparatus can also be implemented as special-purpose logic circuitry (e.g., FPGAs or ASICs).

[0099] Processors suitable for executing computer programs include, by way of example, both general-purpose and special-purpose microprocessors, as well as any one or more processors in any kind of digital computer. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The basic components of a computer are a processor for performing actions according to instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or be operatively coupled to receive data from or transfer data to one or more mass storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or both. However, a computer does not necessarily need to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, GPS receiver, or portable storage device (e.g., a Universal Serial Bus (USB) flash drive), to name just a few. Suitable devices for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM discs. The processor and memory can be supplemented by or incorporated into dedicated logic circuitry.

[0100] To provide interaction with the user, embodiments of the subject matter described herein can be implemented on a computer having a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to allow interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0101] While this specification contains numerous specific implementation details, these details should not be construed as limiting the scope of any invention or potentially claimed content, but rather as descriptions of features specific to particular embodiments of a particular invention. Certain features described in this specification within the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, multiple features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments. Furthermore, although features may be described above as functioning in certain combinations and even initially claimed in this way, in some cases one or more features from the claimed combination may be removed from the combination, and the claimed combination may refer to a sub-combination or a variation of a sub-combination.

[0102] Similarly, although operations are described in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or in a sequential order, or requiring all illustrated operations to be performed to obtain the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of the various system components in the embodiments described above should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0103] Therefore, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and the desired result can still be obtained. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to obtain the desired result. In some embodiments, multitasking and parallel processing may be advantageous.

[0104] Several embodiments have been described. However, it should be understood that various modifications can be made. Therefore, other embodiments are within the scope of the appended claims.

Claims

1. A method executed by a data processing device, the method comprising: Determine whether a simplex topological pattern of signal transmission activity in a set of topological patterns occurs in the recurrent artificial neural network in response to input of first data, wherein the first data includes image data, video data, temperature data, pressure data, chemical composition data, acceleration data, or position data; Determine whether the simplex topological pattern of signal transmission activity in the set occurs in the recurrent artificial neural network in response to input of second data, wherein the second data includes image data, video data, temperature data, pressure data, chemical composition data, acceleration data, or position data, wherein the first data and the second data are diverse data and originate from different sensors; as well as The distance between the first data and the second data is determined by comparing the simplex topology pattern that is determined to be a signal transmission activity in response to the input of the first data with the simplex topology pattern that is determined to be a signal transmission activity in response to the input of the second data.

2. The method according to claim 1, wherein: The occurrence of the simplex topology pattern of the signal transmission activity is represented in a first set and a second set of binary or multi-valued numbers, each number indicating whether a corresponding topology pattern occurs in response to the input of corresponding data in the first data and the second data.

3. The method of claim 2, wherein the information in the first set and the second set is holographically represented because information about the value of a single number in either the first set or the second set is distributed across the values ​​of other numbers in the same set.

4. The method of claim 1, wherein a distance metric is used to determine the distance, the distance metric processing a first subset of active simplex topological patterns in the set and a second subset of active simplex topological patterns in the set differently, wherein the processing weights the first subset of the simplex topological patterns to indicate distance more strongly than the second subset of the simplex topological patterns.

5. The method according to claim 1, wherein: The distance is determined using a distance metric that treats a first subset of active simplex topological patterns in the set differently from a second subset of active simplex topological patterns in the set; and The first subset of the simplex topology patterns includes only the simplex topology patterns that appear in a region of the recurrent artificial neural network, primarily due to a first data perturbation originating from the first sensor.

6. The method according to claim 1, wherein: The distance is determined using a distance metric that treats a first subset of active simplex topological patterns in the set differently from a second subset of active simplex topological patterns in the set. The recurrent artificial neural network is trained; and The second subset of the simplex topological patterns includes only the simplex topological patterns that appear in a region of the recurrent artificial neural network, reflecting the training process.

7. The method of claim 1, wherein the recurrent artificial neural network is untrained.

8. The method according to claim 1, wherein: The distance is determined using a distance metric that treats a first subset of active simplex topological patterns in the set differently from a second subset of active simplex topological patterns in the set. The complexity of the simplex topological patterns in the second subset of the simplex topological patterns is higher than that in the first subset.

9. The method according to claim 1, wherein: The distance is determined using a distance metric that treats a first subset of active simplex topological patterns in the set differently from a second subset of active simplex topological patterns in the set. At least some of the simplex topological patterns in the first subset are included in the simplex topological patterns in the second subset.

10. The method according to claim 1, further comprising: Repeatedly determine whether the simplex topological pattern of signal transmission activities in the set occurs in the recurrent artificial neural network in response to the input of data to the recurrent artificial neural network; as well as The input data is clustered based on the distance between the data, wherein the distance is determined by comparing corresponding simplex topological patterns that are identified as signal transmission activities occurring in response to the input of the data.

11. The method according to any one of the preceding claims, wherein determining whether the simplex topology mode of the signal transmission activity occurs comprises: Determine whether a directed simplex topological mode of signal transmission activity occurs.

12. The method of claim 1, wherein the simplex topological pattern surrounds the cavity.

13. A method for training a recurrent artificial neural network, the method comprising: Receive information identifying the occurrence of a directed simplex topological pattern in response to inputs to multiple different elements in a recurrent artificial neural network, wherein each of the different elements includes image data, video data, temperature data, pressure data, chemical composition data, acceleration data, or position data. The distance between the different elements is calculated based on the occurrence of the directed simplex topological pattern of the signal transmission activity occurring in the recurrent artificial neural network. as well as Adjust one or more properties of the recurrent artificial neural network to increase or decrease the distance between the different elements.

14. The method of claim 13, comprising: The distance between the first element and the second element is determined by comparing a directed simplex topological pattern that is determined to be a signal transmission activity in response to an input of the first element with a directed simplex topological pattern that is determined to be a signal transmission activity in response to an input of the second element.

15. The method of claim 13, wherein a distance metric is used to calculate the distance, the distance metric differently processing a first subset of directed simplex topological patterns of signal transmission activity in a set of topological patterns and a second subset of directed simplex topological patterns of signal transmission activity in the set.

16. The method of claim 15, wherein the distance calculation weights the first subset of the directed simplex topological pattern to indicate distance more strongly than the second subset of the directed simplex topological pattern.

17. The method of claim 15, wherein the complexity of the directed simplex topological patterns in the second subset of the directed simplex topological patterns is lower than the complexity of the directed simplex topological patterns in the first subset.

18. The method of claim 13, wherein: The information for identifying the occurrence of a directed simplex topological pattern of signal transmission activity includes a first binary vector for the first element and a second binary vector for the second element; Calculating the distance includes calculating the distance between binary vectors.

19. The method of claim 13, wherein the recurrent artificial neural network is untrained.

20. The method of claim 13, wherein: The occurrence of the directed simplex topological mode of the signal transmission activity is represented in a corresponding set of binary or multi-valued numbers for each element, wherein each number in each set indicates whether a corresponding directed simplex topological mode has occurred.

21. The method of claim 20, wherein the information in the first set of the set is holographically represented because information about the value of a single number in the first set is distributed across the values ​​of other numbers in the first set.

Images