An action recognition method and system based on unsupervised neural network LBRI

Abstract

This invention discloses an action recognition method and system based on the unsupervised neural network LBRI. The method includes the following steps: S1: Acquire action information and extract feature values; S2: Transmit the feature values ​​to the LBRI model via the feature bus FB, wherein the feature values ​​are a feature value sequence TSQ input over time, and the feature value sequence TSQ is classified into a permutation class ATar; S3: The LBRI model memorizes new permutation classes ATar and recognizes old permutation classes ATar, thus solving the action recognition problem. Compared with existing technologies, this invention is more biomimetic, saves more storage space, and has higher recognition efficiency in action recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to a motion recognition method and system based on unsupervised neural network LBRI. BACKGROUND

[0002] In the field of computer motion recognition, recurrent neural network and long short-term memory network are often used for motion recognition, wherein the recurrent neural network (RNN) is a recursive neural network with sequence data as input, recursion in the evolution direction of the sequence and all nodes (recurrent units) connected in a chain.

[0003] The long short-term memory network (LSTM) is a time recurrent neural network, which is specially designed to solve the long-term dependence problem of general RNN (recurrent neural network), and all RNNs have a chain form of repeated neural network modules. In the standard RNN, the repeated structure module has only a very simple structure, such as a tanh layer.

[0004] However, there is no method that can imitate and efficiently recognize motion in the prior art. SUMMARY

[0005] The present application aims to provide a motion recognition method and system based on unsupervised neural network LBRI, to solve the technical problem of how to improve the efficiency of motion recognition.

[0006] The purpose of the present application is achieved by adopting the following technical solution: a motion recognition method based on unsupervised neural network LBRI, comprising the following steps:

[0007] S1: obtaining motion information and extracting feature values;

[0008] S2: transmitting the feature values to the LBRI model through the feature bus FB, wherein the feature values are a sequence of feature values TSQ input over time, and the sequence of feature values TSQ is classified into an arrangement class ATar;

[0009] S3: the LBRI model memorizes new arrangement class ATar and recognizes old arrangement class ATar, solving the problem of motion recognition.

[0010] Further, the LBRI model remembers a number of arrangement classes ATar, and when an arrangement class ATar that has been remembered is input to the LBRI model, the LBRI model recognizes it; when a new arrangement class ATar is input to the LBRI model, the LBRI model memorizes it.

[0011] Further, the receptor receptor in the receptor-brain model RB receives a stimulus from the outside world and extracts a feature value therefrom.

[0012] Further, the receptor receptor includes a CNN that removes a fully connected layer.

[0013] Further, the LBRI model includes a long short-term memory network framework LSTMF module, a biological instinct neural network training module, a recurrent associative neural network RANN module, and an interpretable sequence spiking neural network ISNN module.

[0014] Further, the long short-term memory network framework LSTMF module includes a short-term memory submodule, a long-term memory submodule, and a biological instinct submodule, wherein the short-term memory submodule is used to process and unconditionally record incoming feature values in chronological order; the long-term memory submodule is used to reason about incoming feature values; and the biological instinct submodule is used to identify recurring, regular sequences of feature values and generate new long-term memories.

[0015] Further, the biological instinct neural network training module includes an Rdtc biological instinct submodule and an Lgen biological instinct submodule, the Rdtc biological instinct submodule is used to periodically search short-term memories and determine whether there are recurring regular segments therein, classify and point them out; the Lgen biological instinct submodule is used to generate long-term memories, and if the Rdtc detects recurring regular segments, new long-term memories are generated based on these segments.

[0016] Further, the recurrent associative neural network RANN module includes a differential recurrent neural network DRNN submodule, an object memory submodule, and a sequence memory submodule, the differential recurrent neural network DRNN submodule is used to find the trend of feature value changes; the object memory submodule is used to classify feature value memories by class; and the sequence memory submodule is used to simulate the biological short-term memory characteristic of remembering the latest and forgetting the oldest.

[0017] Furthermore, the interpretable sequence spiking neural network (ISNN) module includes an inference igear submodule and a non-spiking neuron filter submodule. The inference igear submodule is used to simulate the permutation class ATAr; the non-spiking neuron filter submodule is used to receive the input of the feature value bus FBR and output it to the inference igear submodule.

[0018] An action recognition system based on the unsupervised neural network LBRI includes an LBRI model, which is used to solve the problem of action recognition.

[0019] The beneficial effects of the present invention are as follows: compared with the prior art, the present invention is more biomimetic, saves more storage space, and has higher recognition efficiency in terms of action recognition. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0021] Figure 1 This is a flowchart of the present invention;

[0022] Figure 2 Flowchart of input and output for 3D CNN;

[0023] Figure 3 The input-output flowchart is shown below;

[0024] Figure 4 Here is a block diagram of the Recurrent Associative Neural Network (RANN) module.

[0025] Figure 5 Accumulate the same type of schematic diagrams for the object memory submodule;

[0026] Figure 6 For a structure of the same type in the object memory submodule;

[0027] Figure 7 This is a schematic diagram of the SMC structure.

[0028] Figure 8 Here is a diagram of an SMC queue with a sum of 6.

[0029] Figure 9 for Figure 8 Schematic diagram of a circular queue after expansion;

[0030] Figure 10 For sequence instances;

[0031] Figure 11 For Figure 10 Abstract diagram;

[0032] Figure 12 For sequence schematic diagram;

[0033] Figure 13 For Figure 12 Abstract diagram;

[0034] Figure 14 For igear submodule schematic diagram;

[0035] Figure 15 For igear abstract diagram;

[0036] Figure 16 For non-pulse neuron filter submodule schematic diagram;

[0037] Figure 17 For igear structure schematic diagram of the mth and the m+1th member;

[0038] Figure 18 For data making flow chart;

[0039] Figure 19 Flow chart for making long-term memory through data set;

[0040] Figure 20 For the general schematic diagram of LBRI model. DETAILED DESCRIPTION

[0041] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0042] It should be noted that: similar labels and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0043] Some embodiments of the present application will be described in detail below in connection with the drawings. The following embodiments and features in the embodiments can be combined with each other without conflict.

[0044] Embodiment 1:

[0045] Referring to Figure 1 A motion recognition method based on unsupervised neural network LBRI, comprising the following steps:

[0046] S1: Obtain action information and extract feature values;

[0047] S2: The feature values ​​are transmitted to the LBRI model via the feature bus FB. The feature value is the feature value sequence TSQ input over time. The feature value sequence TSQ is classified into the permutation class ATar.

[0048] S3: The LBRI model memorizes new permutation class ATar and recognizes old permutation class ATar, thus solving the problem of action recognition.

[0049] The mathematical basis of the LBRI model, the types of problems it can handle, and its inputs and outputs:

[0050] Input basis: Sequence. A sequence, as defined in conventional mathematics, refers to a regular arrangement of elements. For any sequence A, the nth (n∈N) element in A... * A member is the nth element counted sequentially from the beginning, denoted as A(n). num (Using the superscript num as a label), we call A(n). num With A(n+1) num Adjacent members are called member indices (n). Different members are separated by commas (,). In sequence A, members from A(n)... num Starting from A(n+k) num The sequence ends (the sequence itself has no fewer than n+k members, n∈N). * The part of sequence A that contains k ∈ N is called a subsequence of sequence A, denoted as A'ss. n,n+k (A's subsequence, with the starting index as the subscript). A permutation is a combination of several adjacent or non-adjacent members of sequence B, arranged according to their order in sequence B. A permutation of B is also a sequence; for example, in the sequence B1,6,7,8,3,4, the 2nd, 3rd, and 5th members, 6,7,3, form a permutation. The order in which members appear in a permutation must be the same as the order in which elements appear in its sequence; for example, 3,7,6 is not a permutation. A permutation of B is denoted as B'ar. A (B's arrange), where A is the sequence formed by arranging the members' numbers in ascending order. The above 6, 7, 3 is denoted as B'ar. 2,3,5 Arrangement B'ar A The encompassed subsequence refers to the subsequence of sequence B that starts from the first member of the permutation and ends with the last member of the permutation. The function cs(B'ar) A (calculate sub sequence) is used to obtain B'ar A Functions that encompass subsequences. For example, B'ar 2,3,5 The included subsequence is cs(B'ar)2,3,5 ) = B'ss 2,5 . The lengths of a sequence, a subsequence, and a permutation are all equal to the number of members they contain. Let C be a permutation or a subsequence of sequence B, and B(n) num be the m-th member of C. Then B(n) num is called the parent mapping member of C(n) num , and C(n) num is the child mapping member of B(n) num . For example, the parent mapping member of B'ar 2,3,5 (3) num is B(5) num .

[0051] Problems that the LBRI model can handle: The input of LBRI is the same as that of a spiking neural network (SNN). As time goes by, eigenvalues are continuously input into LBRI through the Feature Bus (FB). LBRI will continuously respond to the input like a living being. The input eigenvalues are the time sequence TSQ (Time sequence) of eigenvalues input over time. TSQ has an additional property compared to an ordinary sequence, which is the time difference between adjacent members. TSQ can be represented by a binary tuple <s, t>, where s is an arbitrary sequence with length n, and t is the time difference vector corresponding to s, with dimension n - 1. Any m-th position (m < n) of t represents the incoming time difference between s(m) num and s(m + 1) num on the FB. For any members s(a) num and s(b) num (a < b) in the TSQ sequence s, define the incoming time difference between them on the FB as s(a) num , s(b) num ). t Define the one-dimensional functions pred and posn that accept a member of a sequence. pred is to find the incoming time difference between the input and its previous member (if any), and posn is to find the incoming time difference between the input and its next member (if any), that is

[0052] pred(s(b) num ) = (s(b - 1) num , s(b) num ) t , posn((b) num ) = (s(b) num , s(b + 1) num ) t

[0053] In a sequence of actions (TSQ), some members are useless noise, while others form several classes of regular sequences that need to be identified. These arrangements are called ATars (attention arranges). An ATar represents a class of TSQs; for example, running represents a class of action sequences. The LBRI model remembers several ATars. When it encounters a remembered ATar, it recognizes it; when it encounters a new ATar, it determines that a new ATar exists in the TSQ and memorizes it. The problem that LBRI solves is memorizing new ATars and recognizing old ATars. This problem is known as the Attention Arrangement Recognition Problem, a typical sub-problem of which is action recognition.

[0054] The input and output of the LBRI model involve a receptor-brain model (RB). In the RB, the receptor receives stimuli from the outside world (such as an incoming image) and extracts feature values. The receptor then passes these feature values ​​to the brain via a feature bus (FB), where the brain processes the feature values ​​and produces the result. LBRI plays the role of the brain in this model. Many receptor models already exist, such as CNNs with several convolutional and pooling layers removed.

[0055] The following will use CNNs as an example to illustrate the differences between the LBRI model and traditional deep learning inputs and outputs. Traditional CNNs use fully connected layers as the brain; in this example, LBRI replaces fully connected layers as the brain, while the previous convolutional and pooling layers act as receptors, similar to the eyes of a living organism—providing the brain with feature values ​​in the form of neural impulses. In action recognition, a traditional CNN takes multiple images as input, performs one inference, and produces an output (this is called a 3D CNN; its input and output can be found in [reference needed]). Figure 2 Instead of using the LBRI model to replace the fully connected layers of the CNN, images are fed in sequentially according to their capture time (see flowchart). Figure 3 The LBRI model continuously reacts and produces outputs (similar to SNNs). The above comparison is just one example; the LBRI model can be embedded not only into CNNs, but also handle features from other sources (not just images, but also audio) as long as the receptor changes. The input and output of LBRI are very similar to SNNs, due to the structure of Recurrent Associative Neural Networks (RANNs) and Interpretable Sequence Spike Neural Networks (ISNNs), which will be discussed later.

[0056] The LBRI model includes a Long Short-Term Memory (LSTMF) network framework module, a biological instinct neural network training module, a Recurrent Association Neural Network (RANN) module, and an Interpretable Sequence Spike Neural Network (ISNN) module.

[0057] The Long Short-Term Memory Frame (LSTMF) module is the framework of the LBRI model, which is divided into three sub-modules: short-term memory sub-module, long-term memory sub-module, and biological instinct sub-module.

[0058] The Short-Term Memory (STM) submodule is responsible for processing and unconditionally recording the incoming feature values ​​in chronological order (i.e., processing and remembering every input feature). STM has a storage limit; once this limit is reached, if a new feature value is input, the oldest memory value (the earliest arriving feature value) will be forgotten to make room for the newest input feature value.

[0059] The long-term memory submodule is responsible for inference on the input feature values. It contains a large number of long-term memories, each corresponding to a class of ATars. Its function is: if a class of ATars already exists in the long-term memory module (learned), then when a complete TSQ belonging to this ATars is input, the long-term memory can recognize this TSQ and generate a specific signal: recall. This process is called inference. How this function is implemented will be explained later.

[0060] The Biological Instinct submodule is a training method for the neural network in the LBRI model. It has two main functions: 1) Rdtc (repetition detect), which identifies recurring, regular sequences of feature values; and 2) Lgen (long-term memory generator), which generates new long-term memories (new neural networks). It's important to emphasize Rdtc. Rdtc periodically scans short-term memory, and if a new ATa is suspected, it generates a specific signal: "detected".

[0061] The general logic of LSTMF is as follows: the short-term memory submodule and the long-term memory submodule operate in parallel. When a feature value is input, the short-term memory processes and stores it, while the long-term memory directly receives it for inference. Rdtc periodically scans the short-term memory. If no detected value is generated during this period, the recently input sequences are considered irregular, unworthy of learning, or already learned, and the LBRI model will not generate any additional behavior. However, if a detected value appears, the LBRI model will consider that the recently input feature value may contain a new ATa. Then, LBRI will invoke the biologically instinctive Lgen function to create several new long-term memories based on Rdtc and add them to the long-term memory submodule. It should be noted that initially, the long-term memory submodule does not contain any long-term memories. The LBRI model continuously accumulates long-term memories during the continuous input of feature values, and these long-term memories can then be used for inference. Therefore, the LBRI model is an unsupervised learning model.

[0062] The Biological Instinct training method module describes a theory and biomimetic training method, its origins and principles. Living organisms possess an innate, function-mapping ability, such as instinctively determining whether food is edible upon seeing it. This ability is called "Biological Instinct." Biological instinct is a mapping, taking input and producing output; it is a function symbolizing intuition. Structures that achieve this function mapping can be used to simulate biological instinct. In the LBRI model, two biological instincts are simulated: the Rdtc biological instinct submodule and the Lgen biological instinct submodule.

[0063] Rdtc (Repetition Detection): Rdtc fits the biological instinct: given a set of sequences as input, humans can determine whether they contain repetitive segments, classify them, and identify them. Rdtc periodically retrieves short-term memory and determines whether sequences within it contain repetitive segments. The Rdtc part takes short-term memory as input and produces output in a specific format. The specific output format will be mentioned later.

[0064] Lgen (Long-term memory generator): The biological instinct that Lgen fits: After learning new things, the structure of neurons in the human brain undergoes changes to varying degrees and aspects. These changes have a certain functional relationship with the input. Long-term memory is generated by a network similar to SNN, and the process of generating new long-term memories corresponds to new changes between neurons. The number of parameters of neurons and the weights of connections in the new network are obtained by the biological instinct corresponding to Lgen. The input of Lgen is the output of Rdtc, and the output of Lgen is the new long-term memory. The details of Rdtc and Lgen will be described later; both are functional structures that map functions.

[0065] The Recurrent Associated Neural Network (RANN) module proposes the hypothesis of "separation of memory objects from memory sequences." When a person remembers a sequence A, such as when they see an action, they do not memorize every single member of A. Instead, they categorize the members into several classes already existing in their memory, transforming A into a sequence of different classes. Each class is memorized separately, storing specific details. However, for time-series memories, only the order in which these objects appear is remembered, without memorizing the specific details. This viewpoint is called the object-sequence separation hypothesis. The RANN module includes a Differential Recurrent Neural Network (DRNN) submodule, an object memory submodule, and a sequence memory submodule. The latter two submodules apply the object-sequence separation hypothesis. A block diagram of the RANN module can be found [link to diagram]. Figure 4 .

[0066] The Differential Recurrent Neural Network (DRNN) submodule is a type of RNN that processes the feature values ​​passed from the First Bus (FB) and outputs them to the Feature Bus after RNN (FBR), a general-purpose feature bus within the LBRI. The dimension of the feature values ​​passed by the FBR is FBR. d FBR every FBR t A new feature value is input, which remains consistent with the initial flow (FB). The purpose of the DRNN is to find the trend of feature value changes. It believes that the current feature value represents a state that depends not only on itself but also on previously input feature values. For example, if a ball is moving in a straight line or a curve, capturing an image at a certain moment cannot determine whether it is moving in a straight line or a curve; it needs to be combined with previous images. The DRNN is equivalent to differentiating a sequence to determine the current state of the sequence.

[0067] An RNN can be used as a DRNN under two conditions: poor memory and robustness to noise. 1. Poor memory: An RNN should only remember recently input feature value sequences. Regardless of the previous input feature value sequences, as long as the recently input feature value sequences are similar, the RNN should have similar hidden states and outputs. 2. Robustness to noise: If the recently input feature value sequences are similar, even if noise makes them not exactly the same, the RNN should still have similar hidden states and outputs. The standard for feature value sequence similarity should be determined by the user. A specially designed training set should be used to train the RNN to achieve the above objectives. The DRNN part can also be implemented using structures similar in function to RNNs, such as LSTM models.

[0068] The object memory submodule implements the object part of the object sequence separation hypothesis. It receives feature values ​​from the FBR and employs a clustering approach. Object memory divides feature values ​​into clusters. Two feature value vectors that are close in distance are considered to belong to the same cluster. Initially, object memory does not contain clusters; as feature values ​​are continuously passed in, object memory accumulates a large number of clusters (see [link to documentation]). Figure 5 For the structure of the same type in the object memory submodule, please refer to [reference needed]. Figure 6 Centroid is the centroid of the cluster and is a vector with dimension FBR. d When a feature value's distance from the centroid is less than a user-defined threshold (tolerance), the feature value is classified into this cluster. `weight` is the cluster's weight; it's a number with a range of (0, wmax). `wmax` is a user-defined constant representing the frequency of recently passed feature values ​​belonging to this cluster; its initial value is a constant within the range (0, wmax), the determination of which is explained below. `average` represents the mean distance between feature values ​​belonging to the cluster and the centroid; it's a number, and the larger the mean, the larger the average; its initial value is 0. `variance` represents the variance between feature values ​​belonging to the cluster and the centroid; it's a number, and the larger the variance, the larger the variance; its initial value is 0. When a new feature value A comes from the FBR, the object memory compares it with all clusters in memory. If the centroid distance between A and a cluster is greater than the tolerance, then A is determined not to belong to that cluster.

[0069] We will now discuss three scenarios: 1. There is no cluster in the object memory; 2. There is a cluster in the object memory, but A does not belong to any cluster; 3. There is a cluster in the object memory, and A belongs to several clusters.

[0070] Scenario 1: Create a new cluster, use A as its centroid, and assign initial values ​​to weight, average, and variance.

[0071] Scenario 2: Create a new cluster, using A as its centroid, and assign initial values ​​to weight, average, and variance. There are two options for the initial weight value: first, all clusters can use user-defined constants as initial values; second, create a function structure that uses the centroid initial value to determine the initial weight value. This function structure can be implemented using an ANN or similar mechanism, with the aim of prioritizing certain features, allowing LBRI to focus more attention on specific types of features. This function structure needs to be trained by the user as needed.

[0072] Scenario 3: Select the cluster with the closest centroid to A as the class to which A belongs. If multiple clusters have centroids that are very close to A, select the cluster with the largest weight as the class to which A belongs. Then recalculate the centroid, average, and variance values ​​of the cluster to which A belongs, and increase the weight value accordingly.

[0073] The specific changes in the cluster parameters during the above steps are explained below: the centroid, average, and variance values ​​for each cluster are only updated in case 3, and remain unchanged at other times. The calculation method for centroid, average, and variance in case 3 is as follows:

[0074] centroid′=centroid+(centroid-A)×f(weight)×μ

[0075] average′=average×(1-frac a (weight))+dis(centroid,A)×frac a (weight)

[0076] variance′=variance×(1-fracv (weight))+|dis(centroid,A)-average′|×frac v (weight)

[0077] Where centroid, average, and variance are the old values ​​before calculation, and centroid', average', and variance' are the new values ​​after calculation. μ is the learning step size, a normal quantity given by the user; its value should be determined through extensive experimentation in specific application scenarios. f(x) is a function that satisfies the following conditions:

[0078] 1) The domain is (0, wmax) and the range is (0, e), where e is a normal value given by the user.

[0079] 2) It is continuous and monotonically decreasing over its domain;

[0080] dis(x, y) is a function that calculates the distance between vectors x and y.

[0081] frac a (x), frac v (x) is a function that satisfies the following conditions:

[0082] 1) The domain is (0, wmax) and the range is (0, 1);

[0083] 2) It is continuous and monotonically decreasing over its domain.

[0084] The cluster weight will change in two situations: 1. Over time, 2. When a situation occurs.

[0085] 1. As time passes, Weight decreases by the value of dw every dt seconds, where...

[0086] dw = g(weight)dt

[0087] Where g(x) can be any function that satisfies the following conditions:

[0088] 1) The domain is [0, wmax]

[0089] 2) g(0)=0, g(wmax)=0, g(x)<0.

[0090] 3) There is a constant b in (0, wmax) depending on the situation, and g(b) is the minimum value of the function g(x).

[0091] 4) The function is continuous on its domain and monotonically decreasing on (0, b] and monotonically increasing on [b, wmax).

[0092] The above parameters and functions are provided by the user as needed.

[0093] 2. If scenario 3 occurs, the weight should increase by a constant. It is a constant between (0, wmax) that depends on the situation and is called the ascending step size, which is generally much smaller than wmax.

[0094] if If the value is greater than wmax, then weight is forced to be equal to wmax - α, where α is a constant greater than 0, depending on the situation, and α << wmax.

[0095] The above explains the specific changes in centroid and weight. It's worth noting that although f(x), g(x), and frac... a (x), frac v (x) Any function that meets the conditions can be chosen, but the choice of different functions will have a significant impact on LBRI. The choice of function should be determined through extensive experimentation in specific application scenarios. When the weight value of a cluster is less than fgt, i.e., it has not appeared for too long, LBRI will delete it. This process is called forgetting, where fgt > 0 and fgt << wmax. It can be seen that as long as FBR is provided with feature values, they will definitely belong to a certain cluster after passing through object memory. The above parameters and functions are given by the user according to the situation.

[0096] The sequence memory submodule implements the sequence part in the object sequence separation theory. It does not remember the feature values ​​transmitted from FBR, but rather remembers the cluster to which the feature values ​​belong.

[0097] Sequence memory is a finite-length, cluster-based short-term memory (TSQ). It employs a circular queue-like memory mechanism to mimic the biological short-term memory's characteristic of remembering the most recent information and forgetting the oldest. It is composed of a large number of sequence memory cells (SMCs).

[0098] First, let's discuss the structure of a sequence memory (SMC). An SMC has two pointers: one pointing to the next SMC, called the nextpointer (nptr); and the other pointing to a cluster, called the associate pointer (aptr). We will now discuss the structure of sequence memory in two parts.

[0099] 1. The structure of SMCs: SMCs form a circular queue, with each SMC using an nptr pointer to point to the next one. An SMC can only be pointed to by one other SMC, and they are linked end-to-end. Additionally, there is a head pointer (header) pointing to a specific SMC (see [reference needed]). Figure 7 The number of Short-Term Memory (SMCs) is fixed; the more SMCs there are, the more they can remember. Initially, the header points to any SMC. Each SMC corresponds to a feature value passed in by the FBR, but the SMC does not record the specific feature value. The header points to the SMC corresponding to the most recently passed feature value; this SMC is called the hsmc. When a new feature value is passed in, it will be associated with the next SMC of the current hsmc, and the header will point to the next SMC of the hsmc. This process is called hsmc iteration. For example, if SMC0 points to SMC1 and SMC1 points to SMC2, initially SMC0 is the hsmc. Feature values ​​A and B are passed in sequentially. When A is passed in, A will correspond to SMC1, and the header will point to SMC1, making SMC1 the current hsmc. When B is passed in, B will correspond to SMC2, and the header will point to SMC2, making SMC2 the current hsmc. When the entire SMC circular queue is full, meaning all SMCs in the queue have been passed through the HSMC once, HSMC iteration will still occur if a new feature value arrives. At this point, the SMC representing the oldest memory will correspond to the latest feature value. In simpler terms, the oldest memory will make room for the newest memory.

[0100] 2. The structure of an SMC pointing to a cluster, where one SMC corresponds to one eigenvalue, is implemented using the aptr. When eigenvalue A is passed in from the FBR, the object memory outputs the cluster to which it belongs. Simultaneously, the hsmc in the sequence memory iterates, generating a new hsmc and setting its aptr to point to the cluster to which A belongs. If an SMC has never been an hsmc, its aptr is set to null. Similarly, if the cluster pointed to by an SMC is deleted due to its small weight, its aptr is also set to null.

[0101] In summary, object memory uses its cluster structure to record the details of feature values, while sequence memory uses a circular queue (SMC) to record cluster sequences. The process of finding a cluster pointed to by the SMC is called association.

[0102] The structure and behavior of RANN have been explained. The following section describes the Biological Instinct function related to RANN: repetition detection.

[0103] The purpose of the DTC function is to retrieve the TSQ (Short Memory Queue) without repeating ATAs corresponding to the stored short-term memory. The SMC (Short Memory Queue) circular queue is expanded into a sequence of length `sum`. The next SMC after `hsmc` is the first member, `hsmc` is the `sum`-th member, and the SMC being pointed to is located after the SMC pointing to it. Then, the clusters associated with the SMCs are arranged in this order to form a TSQ, called `cw` (current window), representing the sequence of feature values ​​within the short-term memory. The length of `cw` is `sum`, and its elements are all clusters with a time difference of FBR between adjacent members. t An SMC queue graph with a sum of 6, for example. Figure 8 As shown, its circular queue, after being expanded, can be found in [reference needed]. Figure 9 Then the cw sequence is in the following order: cluster2, cluster3, cluster3, cluster2, cluster1, cluster1; its time difference vector is [FBR t FBR t FBR t FBR t FBR t ].

[0104] This paper proposes the Same Position Resemblance hypothesis. In two permutations that humans perceive as similar, the degree of similarity is determined not only by fundamental properties of the permutation such as its length, but also, and perhaps more importantly, by the impressions people have of each member within the permutation. For example... Figure 10 As shown, although sequences A and B have different lengths, and B and C have the same length, the subjects still perceived A and B as having a higher similarity than B and C. Figure 11 As shown. Although figures A and B are of different lengths and are not similar figures (i.e., proportionally enlarged and reduced figures), and the proportions of figures B and C are closer than those of A and B, the subjects still considered A and B to be more similar than B and C. The theory of similar-positional feature approximation posits that people extract a general structural feature value from objects, and that people perceive objects with similar structures as more similar. Among things with similar structures, some elements are referred to as being of similar position, such as... Figure 12 and Figure 13As shown, subjects perceived the elements connected by dashed lines in the above AB structure as having similar positions within similar structures. The co-positional feature approximation hypothesis states that the human brain generates similar feature values ​​for elements with similar positions within structures it perceives as similar. These feature values ​​are called positional feature values ​​and have the following characteristics: only elements with similar positions within similar structures can generate similar feature values; otherwise, the feature values ​​generated by any two elements should be far apart. (Even feature values ​​of different positions in similar structures are far apart.) If a and b are members from two identical or different sequences and their positional feature values ​​are similar, then a is said to be equivalent to b. In the first example, the third member of A is equivalent to the third member of B, and the sixth and seventh members of A and the sixth member of B are mutually equivalent. The evaluation criteria for structural similarity are given by the user; to ensure that the LBRI model has equivalent standards, a special training set must be created to train the Rdtc.

[0105] In a cw sequence, clusters with high weights will attract more attention. If permutations of similar structures formed by some high-weight clusters in a cw sequence appear repeatedly, these permutations are said to belong to the same class ATar. Now, for any class ATar: B in a cw sequence, for any cw'ar A If a subsequence belongs to B, it is called a derived permutation of B. The subsequences encompassed by a derived permutation of B are called derived subsequences of B. A derived sequence must contain at least one derived subsequence. If subsequence a is a derived subsequence of B, then a belongs to B. A class of ATars must exist on a cw for it to be considered as having a subsequence cw with multiple non-overlapping subsequences, high member weights, and similar structures. The structure of an ATars is a similar structure. Even if some low-weight clusters are mixed into a derived subsequence of B, the subsequence still belongs to the derived subsequences of B, and the low-weight clusters are considered noise. Valid members (high weight) in a derived subsequence of B are called valid, and noise members (low weight) are called noise. The length or members of a derived subsequence or derived subsequence of B are not necessarily the same. The necessary and sufficient condition for determining whether a subsequence of a cw belongs to B is whether the permutation formed by its valid members is structurally similar to B. Note that the eigenvalues ​​of cw members are different from the positional eigenvalues. Eigenvalues ​​refer to the eigenvalues ​​of the centroid records of cw member clusters, while positional eigenvalues ​​are only the eigenvalues ​​generated by members based on their preceding and following structures.

[0106] The goal of training Rdtc is for it to identify structurally similar, class-alike subsequences from cw according to its own criteria. This requires creating a customized training set as needed. However, the given structural similarity criteria must adhere to the following constraints:

[0107] 1. Complete set of equivalence classes

[0108] An equivalence class is a set of all mutually equivalent members (regardless of whether they come from the same TSQ). Members in different equivalence classes are not equivalent. Given a class ATAr with several equivalence classes, let any class B ATAr have n equivalence classes. Any derived permutation b of B must satisfy the following condition: for each of the n equivalence classes, every member of b must belong to one of these classes, and for any one of these n equivalence classes, b must have at least one member belonging to it. Derived subsequences of B may not necessarily satisfy this condition due to noise. For any set Ba containing several derived permutations of class B ATAr, and any member c of any permutation in Ba, the binary function eql(Ba,c) can be defined to find the set of all members in all sequences of Ba that belong to the same equivalence class as c. The input to eql is a TSQ set and any member belonging to any sequence in this set.

[0109] 2. Approximate time difference between adjacent equivalence classes

[0110] Members belonging to different equivalence classes appear in a given permutation of an ATaar class in an ordered manner. An ATaar equivalence class can be formed into an equivalence class sequence in order. Suppose that in class B, there are adjacent nth and (n+1th)th equivalence classes, and in any permutation b of B, two members b1 and b2 belong to the nth and (n+1th)th equivalence classes respectively. Then (b1, b2) t It must be within a certain range. This range is called the adjacent class range. In simple terms, if a person has memorized a set of actions A, when the next action in A appears too early or too late, the person will think that what they are seeing is not action A.

[0111] Rdtc achieves its functionality by segmenting a cw sequence into different subsequences belonging to different classes and labeling them. More than one class of ATar may exist within the same cw sequence. Users should train Rdtc according to their specific needs, but Rdtc must meet exclusion constraints.

[0112] Due to exclusion restrictions, longer subsequences in the cw can only be classified into one subsequence of class ATar at most. That is, in the cw partitioned by Rdtc, no two derived subsequences can have large-scale overlap, regardless of whether they belong to the same class. Large-scale overlap between two subsequences means that the ratio of the number of members in both subsequences to the length of either subsequence exceeds a certain upper limit threshold 'a', the specific value of 'a' being determined by the user based on their needs. The illustration shows... This violates the exclusion restriction in the partitioning method. The ratio of the number of public members to the length of subsequence B reaches [a certain value]. Since the number exceeds 'a', A and B largely overlap. Rdtc should not make divisions that violate exclusion constraints in most cases; this requires users to create special training sets for training.

[0113] The input specifications for Rdtc are as follows: Rdtc accepts a total of sum vectors of the same dimension as input, called input subvectors. Each member of cw corresponds to an input subvector. The sum input subvectors are concatenated into an input matrix according to the order of the cw sequence and then input to Rdtc. For any n <sum,cw(n) num The input sub-vector corresponds to the vector in the nth column of the input matrix. The input sub-vectors have FBR d +1 dimension, formed by concatenating the centroid vectors of the cw member clusters with the weights. The first FBR... d The dimension is centroid, and the last dimension is weight. In summary, the input to Rdtc is a (FBR) d The matrix is ​​(input matrix) of (+1)×sum. If the aptr of smc is empty, then its input subvector is set to zero.

[0114] The output of Rdtc is also a matrix, composed of sum output subvectors corresponding to each member of cw, concatenated in the order specified in cw. Each output subvector consists of three parts: judge, head, and position. judge and head are two numbers, located at the first two positions of the output subvector, while position is pos. d A dimensional vector, pos d The value is given by the user based on the specific circumstances and is located at the end of the output subvector. d Therefore, the output subvector has a total of posd+2 dimensions, while the output of Rdtc is a single (pos... d The output matrix is ​​a matrix of (+2)×sum. The values ​​of the judge and head positions should be in the range (0,1). Rdtc divides the cw sequence using the judge and head positions, where the position eigenvalues ​​are the positional eigenvalues ​​of the generated cw members. Suppose that two subsequences A and B are divided from the cw. If A and B belong to the same class ATar, then the position vectors of members with similar positions in the output matrix should also be close in distance. Rdtc should satisfy the following output constraints:

[0115] Serialize the entire cw into an input matrix, input it into Rdtc, and obtain the output matrix. If there exists at least one class ATar, then:

[0116] 1. In the output matrix, the judge bit of the output subvector corresponding to valid should be as close to 1 as possible, and the judge bit of the output subvector corresponding to noise should be as close to 0 as possible;

[0117] 2. In the output matrix, the head bit of the output sub-vector corresponding to the first member of any derived sub-sequence obtained from the cw partition should be 1 as much as possible, and the head bit of the output sub-vector corresponding to the last member should be 0 as much as possible. In other cases, this bit should be between 0 and 1, not close to either of them. It is recommended to use a traditional DNN or a Transformer with a self-attention mechanism for the Rdtc function, because it is a non-linear function.

[0118] The Interpretable Sequence Spike Neural Network (ISNN) module is similar to the SNN but with some structural changes. The structure and function of the ISNN are as follows:

[0119] 1. The spiking neurons in ISNNs use a simplified LIF (leaky integrate-and-fire) model. To simplify computation and improve interpretability, the propagation delay and continuous firing behavior of neural impulses are eliminated. The neuron's action is simplified from firing within a short period after the potential reaches a threshold to firing only for a brief instant after the potential reaches the threshold. Simultaneously, the potential of subsequent neurons receiving the neural impulse no longer rises continuously, but immediately increases by a single number. 2. Some neurons in ISNNs can be blocked, replacing the refractory period of biological neurons with this characteristic. When blocked, the potential impulses input to the synapse will not affect the neuron's potential. The behavior of ISNN neurons is:

[0120]

[0121] u(t+dt)=u(t)+du(t)+CU j ·∑ j w j ·c j (t) Equation 2

[0122] s(t)=1, u(t)=u r2 if u(t)≥u th

[0123] s(t) = 0 if u(t) < u th

[0124] In the formula, t represents the time variable, u(t) is the membrane potential function, and s(t) is the output pulse potential function; dt represents the time step, and du(t) represents the spontaneous change of the membrane potential within one time step; τ is a constant, u r1 The resting potential, u r2 To reset the potential; u th It is the membrane potential threshold; w jLet c be the weight of the j-th input synapse. j (t) is the pulse potential function of the j-th input synapse; CU j It is the control condition for the j-th input synapse, a Boolean function that determines the CU based on the specific situation when a potential pulse is received. j Select 0 or 1. CU j The condition is whether the potentials of several self- or other neurons are within a certain range; where c j (t) is 1 only at discrete points, and 0 for all other values ​​of t. This means that c j There is no continuous part in the domain of (t) such that c j (t) takes a value of 1. The above parameters are different in each neuron. The formula shows that under normal circumstances, the neuronal membrane potential changes by du(t) every dt time interval, and Equation 1 gives the ratio between the two; however, when a potential pulse comes from the input synapse, the influence on the membrane potential depends on whether the synapse is blocked. If it is affected, the membrane potential immediately increases by a constant according to the weight of the input synapse; otherwise, there is no change. When the membrane potential is higher than u... th Immediately afterward, a potential pulse is released, after which the membrane potential drops to u. r2 .

[0125] The basic components of an ISNN include the inference gear submodule and the non-spiking neuron filter submodule. Each igear (inference gear) submodule consists of four neurons: trigger, upper limit, lower limit, and AND (triggered, upper limit, lower limit, AND operation). The output of the trigger is only connected to the upper and lower limits. The potentials of the upper and lower limits control the AND operation, while the potential of the upper limit controls the trigger (this relationship is represented by a dashed line). A schematic diagram of the igear submodule can be seen below. Figure 14 .

[0126] The inputs to igear include igear input1 and igear input2. igear input1 is the output from other igear components and is used only as input to the trigger. igear input2 is the output from the filter component (described later) and is used only as input to the AND operation. Connecting igear1 to igear2 means using the output of igear1 as input to the trigger of igear2, and connecting the filter to igear2 means using the output of the filter as input to the AND operation of igear2. igear can be abstracted as follows: Figure 15 The components shown are as follows. A filter is a non-spiking neuron component of the ISNN that receives input from the FBR. Each filter monitors a cluster; the filter monitoring cluster 1 is named filter1. The function ftr(filter1) is defined to calculate the range of cluster 1 monitored by filter1. This range is calculated from the centroid, average, and variance of cluster 1. The specific calculation method is given by the user. Feature values ​​falling within this range are considered to be close to the centroid and belong to this cluster. If a feature value from the FBR falls within ftr(filter1), filter1 immediately generates a momentary pulse.

[0127] The filter outputs are all connected to igear. The implementation of the filter is quite simple; it can be achieved using a small number of ANN layers or other simple function structures. A schematic diagram of the non-spiking neuron filter submodule can be seen below. Figure 16 .

[0128] An ISNN consists of an igear and a filter. A long-term memory consists of an ISNN network that simulates a class of ATArs. If the ISNN network s1 simulates B, then when a derivative subsequence of B is passed sequentially from the FBR along time, s1 will generate a momentary pulse after its transmission is complete, informing the LBRI that it has recognized this class of ATArs. The following will describe how an ISNN performs this simulation.

[0129] First, we will explain how the igear chain, composed of interconnected igear structures, simulates the derivative permutation a of class AATar. a is a permutation of cw, and the positional eigenvalues ​​of each member of a have been calculated.

[0130] Any TSQ can be expanded into a "TSQ triple". The structure of a TSQ triple is as follows:<s,t,p> Where s is the sequence itself, t is its time difference vector, and p is the position matrix. For any TSQ a, a s ,a t ,a p Let p represent the three attributes of the triplet 'a'. Let the length of 'a' be n, then p is pos. d A matrix of size ×n.

[0131] For any j ≤ n, the j-th vector of p is the positional eigenvalue of the j-th member of a. Using the function triplize (tripletization), triples of TSQ are generated, where the positional eigenvalues ​​of TSQ need to be given in advance. Then, the triples of a are triplize(a, a...). p ).

[0132] Now using triplize(a, a p Construct an igear chain belonging to 'a', where 'a' is a subsequence or permutation of 'cw'. Let 'a' be a... s Each member generates an igear, and for a s Any m-th member in the array is called the m-th igear. If the (m+1)-th and (m-1)-th igears exist, connect the m-th igear to the (m+1)-th igear and the (m-1)-th igear to the m-th igear. The and neuron of the m-th igear also needs to be connected to a. s The filter of the m-th member cluster. A filter can connect to multiple a. s The igears of the members are defined as long as these members belong to the same cluster. This forms a sequentially linked chain of igears. c And mark its last igear as the output bit. c It can be abstracted into a TSQ with igear as its element.

[0133] a c It will exhibit the following behavior. When a TSQ a' with a structure similar to a is passed in, a c The igear in the connection will emit pulses sequentially according to the connection order. Finally, after the member equivalent to the last member of a in a' arrives, a... c The last igear emits an a c The output pulse. Where, for any m (m>1, m≤n), it is equivalent to a(m). num The arrival time of member a' is time 0, and it is only equivalent to a(m+1) within a certain time range. num Member a' arrives, a cThe (m+1)th igear will emit a pulse. This time range is called the allowable range, and the method for calculating it is given later. Construct the igear structure with the m-th and (m+1)-th members, as follows: Figure 17 As shown.

[0134] Assume the allowable range of the m-th and m+1-th members is [start, end], and they receive inputs from filter m and filter m+1 respectively. After the equivalent member of m arrives, igear m emits a pulse, and the trigger of igear m+1 will receive the pulse instantaneously and also immediately emit a potential pulse. (The neurons inside igear mentioned below are all structures inside igear m+1.) Under normal circumstances, the synapse of the and neuron used to receive the input of filter m+1 is in a blocked state, that is, its control unit function value is 0. After receiving the potential pulse, its input synapse of filter m+1 will be in an open state for a period of time due to the potential change of the upper limit and lower limit, that is, its control unit function value is 1 (the specific implementation will be described later). This time range is the allowable range [start, open]. This means that igear m+1 can only receive a pulse from filter m+1 after receiving a pulse from igear m.

[0135] The specific implementation of the upper limit and lower limit, let the input weight of the trigger with respect to the upper limit be w. u The input weight for the lower limit is w. l Among them, w u ≥0,w l ≤0. After the trigger emits a pulse, the object is subjected to w. u and w l The value affects the potential of the upper limit; it will rise by a constant and then begin to fall, while the potential of the lower limit will fall by a constant and then begin to rise.

[0136] For the upper limit, w u The value of must satisfy this condition:

[0137] For the lower limit, w l The value of must satisfy this condition:

[0138] Where, τ u It is the τ parameter in the upper limit, τ lThis is the τ parameter in the lower limit. a≈b indicates that the values ​​of a and b are very close, but not necessarily equal. The allowable range of the difference between a and b is determined by the user. The control conditions for the upper limit and lower limit are always 1.

[0139] and control conditions CU a for:

[0140]

[0141] else, C U a = 0

[0142] Trigger control conditions CU t for:

[0143]

[0144]

[0145] Where u u u l These are the potentials for the upper and lower limits. The threshold should be two constants, and Parameter τ u u u u l , The value is determined by the allowable range [start, end], and any value that satisfies the above conditions is acceptable. Once the allowable range is obtained, it can be calculated using an equation.

[0146] Taking the instant the trigger pulse is emitted as time 0, according to the above formula, the internal potential will only be greater than the upper limit around the time interval [0, end]. The lower limit only has an internal potential greater than [start, -∞] around the time interval. This means that only pulses emitted by filter m+1 within the allowable range [start, end] will be received by the AND neuron, and the trigger will not receive pulses emitted by other Igears within the range [0, end]. Filter m+1 monitors a c (m+1) num Therefore, only those belonging to a c (m+1) num When the characteristic value is input from the FBR, filter m+1 will generate a pulse.

[0147] In summary, after the trigger emits a pulse, there is only a pulse belonging to a within a range of approximately [start, end]. c (m+1) num When the eigenvalue is received from the FBR, igear m+1 will generate a pulse. For the first member of 'a', there is no corresponding igear, so filter 1 is directly connected to the trigger of igear2. This is because any eigenvalue belonging to the first member cluster can be considered as the start of an approximate sequence.

[0148] The following describes how to calculate the allowable range [start, end]. An ATr dataset is defined as a set of TSQ triples representing one or more ATr derivative permutations. Class B and C ATr datasets (Data) refer to datasets where all TSQs belong to either class B or class C ATr. Other naming conventions follow the same pattern. Calculating the allowable range requires not only triplize(a) but also the ATr dataset. Let AData be a class A dataset. triplize(a, a... p Let the set md be: the sequence of all triples in Adata that are related to a. c (m+1) num A set of members that are equivalent (approximate in status eigenvalues). For any element c in md, calculate pred(c) and add it to the time difference set timeset. Statistically analyze the distribution of time differences in timeset and use statistical methods to find the region where the time difference distribution is most concentrated. This region is the obtained allowable range [start, end]. The function that maps a TSQ triple and its associated dataset to an igear chain according to the above steps is called chainize, i.e.:

[0149] chainize(triplize(a,a p ),Adata)=a c

[0150] The aforementioned igear chain can simulate a class of ATArs to some extent, but it can only recognize sequences very similar in structure. Its recognition range is narrow, its error tolerance is low, and it cannot effectively recognize a class of ATArs. The ISNN network is a reinforcement learning method based on the igear chain, which greatly improves the recognition ability and can recognize a wider range of ATArs. The following is the process of creating an ISNN.

[0151] Bilateral mapping to generate ISNN: Now we create an ISNN network s1 to simulate a class of ATArs named B, and we have a dataset D of class B ATArs.

[0152] 1. Construct a set Dtuples. For any TSQ triplet in D, tripletize(b, b) p Perform the following operations: Construct the igear chain b using the chainize function. c =chainize(triplize(b,b) p ), D). Using b c and b p Construct a tuple for b c ,b p (This type of tuple is called a singly linked tuple). Let Dtuples be the set of singly linked tuples generated from all elements in D. Define the function that maps an ATar dataset to a set of singly linked tuples as tuplize, i.e., tuplize(D) = Dtuples.

[0153] 2. Map the set of Dtuples to an ISNN network using the connect function. The connect function is implemented as follows: 1) connect accepts a set of single-chain tuples as input and outputs an ISNN network. 2) Let the connect function accept a set of tuples, let chains be the set of igear chains of all tuples in tuples, and let the igear chain and matrix in any tuple a in tuples be named a. c q p For a, define the set eqln. a,n =eql(chains,a c (n+1) num (and a) c (n+1) num All members of any igear chain belonging to the same equivalence class). Let a c The output of any nth member is connected to eqln a,n This applies to all inputs of igear. That is, for eqln... a,n For any element m, we have a c The output of the nth igear is used as the input of m. 3) After all igears in chains have undergone the operation in 2), they will be interconnected to form the ISNN network output. The output is used as the output, i.e., connect(tuples) = output. The b0(n) in the igear chain b0 is then used as the input of m. num Connect to all instances of b0(n+1). num ​Equivalent igears. Then connect(Dtuples) = s1. In s1, each sub-permutation's igear connects not only to igears belonging to its own sequence, but also to igears in other sub-permutations that are equivalent to its next member. This strategy can cause igear loops, which are normal and should not be avoided. In an ISNN network, a single igear's trigger will connect to multiple other igears. In this case, the trigger will not immediately emit a pulse after receiving a pulse from one igear. The weights from other igears to the trigger need to be calculated. For a trigger, its behavior should be to emit a pulse only after a certain proportion of inputs have been received within a short period. Following this principle, assign values ​​to the output weights of all igears according to the use case.

[0154] The structure and behavior of ISNN have been explained. The following section will describe the Biological Instinct function Lgen (Long-term memory generator) related to ISNN. When we confirm the existence of a previously unlearned ATAr in the cw sequence, we run the Rdtc function to extract features from the cw sequence, generating several ATAr classes and several derived sub-permutations for each ATAr class. Lgen's function is to transform the Rdtc extraction results into an ISNN network. Lgen only needs to provide the ISNN network structure of the cw extraction results and its parameters. The process is as follows: 1. Create the ATAr dataset singles, extract the members with a judge bit of 1 from the output sequence and form a permutation (called the regular sequence). Divide the regular sequence. For any m-th member in the regular sequence whose head bit is close to 1, search for the m'-th member immediately following it whose head bit is close to 0, and extract the regular subsequence regular'ss. m,m' If there are no members whose head position is close to 0 after the m-th member, then no processing is performed. The set of all subsequences extracted in the above manner is called arrangements. For any TSQ s in arrangements, its positional eigenvalues ​​are extracted from the output matrix and reassembled in order to form matrix s. p Use tripletize to find the triplet triplet(s, s) of s. p The set of all triples obtained by `arranges` is called `singles`, and `singles` is the required ATar dataset. The dataset creation process is as follows: Figure 18As shown. 2. Perform a double mapping on singles to obtain the ISNN network union. Perform a double mapping on singles, i.e., union = connect(tuplize(singles)). 3. Divide the union into a number set: Define separate as a function that maps an ISNN to a set of ISNNs. For any ISNN, abstract it as a graph G, where igear are the nodes of G. If there is a connection between igears, abstract this connection as an undirected edge. Now map any input in to output out, i.e., separate(in) = out. After abstracting the elements in out and in into a graph, the separate function will find all maximal connected subgraphs in in and form the corresponding ISNN subnetworks into a set out for output. The third step is to divide the union to obtain the ISNN set number. Each network in number represents a class ATa, number = separate(union). Let s be any network in number. 4. Let s be any network in the number. Find all the output bits igear and connect them to an output neuron belonging only to s. Only neurons in s that emit pulses within a certain proportion of output bits igear within a short period will emit pulses. Determine the relevant parameters based on this constraint. 5. Add each network in the number to long-term memory and integrate the equivalence classes in the dataset into the equivalence class pool eqlpool. The eqlpool contains multiple equivalence classes, each represented by parameters such as centroid position eigenvalues ​​and range. Use the position eigenvalue matrix in D to perform position eigenvalue clustering and add the newly obtained classes to the eqlpool. If the newly obtained class x largely overlaps with an existing class y in the eqlpool, merge x into y and adjust the centroid and range of y. Refer to the clustering algorithm for details. The process of creating long-term memory from the dataset is as follows: Figure 19 As shown.

[0155] The output of the output neuron in long-term memory is the output of LBRI. The ISNN network corresponding to the output of different long-term memories determines which part of the memory was triggered by the input feature value. The Rdtc detected signal is designed to discover new ATAs not added to long-term memory. Its strategy is to periodically scan the cw sequence using the Rdtc function, collecting the positional feature values ​​of members with a judge bit of 1. It then determines whether these feature values ​​are in the equivalence class of the eqlpool. If, after a certain number of Rdtc executions, a certain proportion of positional feature values ​​are found not in the eqlpool, then a new ATA with a significantly different structure has appeared, and a detected signal is issued.

[0156] A schematic diagram of the LBRI model can be found here. Figure 20 .

[0157] To more clearly illustrate the concept of this invention, human motion recognition will be used as an example for further explanation:

[0158] A receptor-brain mode is adopted, where a CNN-like structure is used as the receptor and an LBRI (LBunch-Based Receptor Intelligence Array) is used as the brain. The receptor continuously captures images from a fixed camera at fixed intervals h and extracts feature values, then transmits these feature values ​​to the LBRI via a front-end (FB). The LBRI receives one feature value every h. Different poses of each person should be extracted with different feature values, and similar poses correspond to similar feature values. For example, raising the right hand and raising the right foot correspond to different feature values, while two poses with the right hand raised to similar heights should correspond to similar feature values. The camera continuously captures images, and the receptor continuously processes them. When a person enters the field of view, the receptor should notice them and extract their pose feature values ​​in each image. When there are many people in the field of view, the receptor should be attentive, extracting the pose of only one person continuously. The receptor is not within the scope of this invention; it is only mentioned here as an example of the requirements for the receptor in action recognition.

[0159] Brain: LBRI model. Whenever a feature value is input from the FB (Front-Ended Layout) model, it is processed by the DRNN (Dual Layout Neural Network) within the RANN (Random Access Layout) and then passed to the LBRI's internal feature bus, FBR (Feature Bus Back). After processing by the DRNN, each feature value represents not only the person's current pose but also the ongoing action. For example, if a person is filmed raising their hand above their head at a speed of v, the final input pose for this action is "hand hanging vertically above the head," and the feature value input from the FB is a1. In the action of raising the hand above the head at a speed of 2v, the final input pose is still "hand hanging vertically above the head," and the feature value input from the FB is a2. Because the final poses are almost identical, a1 and a2 are extremely similar. However, the DRNN considers not only the pose at this moment but also previous poses due to its short memory. Therefore, because the previously input feature values ​​are different, the processed feature values ​​b1 and b2 are not similar. The feature values ​​processed by the DRNN represent a motion trend. Although b1 and b2 correspond to similar poses, their speeds and motion trends are different, hence they are not similar. While raising a hand overhead in a straight line and in a curved line may appear identical in posture, the DRNN, combining previous data, can distinguish the different movement trends and thus assign different feature values. Movement trend refers to the state of a person's limb movement, including but not limited to speed, linear or curved motion, and direction. If a person is in different postures, the feature values ​​extracted by the DRNN will definitely be dissimilar. Even in the same posture, the DRNN will derive different feature values ​​due to the movement trend. Only when the posture and movement trend are the same will the DRNN calculate similar feature values. In summary, the feature values ​​in FBR (Flexible Background Analyzer) contain information about a person's posture and movement trend, symbolizing "movement posture." Similar feature values ​​in FBR represent the same movement posture, while dissimilar feature values ​​represent different movement postures.

[0160] When FBR feature values ​​are input, they are clustered into various clusters by object memory. Each cluster represents a different motion posture, and then sequence memory, responsible for remembering the order in which the motion postures appeared, will remember them. Short-term memory is formed in this way; if a motion posture has not appeared for too long, it will be forgotten.

[0161] Initially, the eqlpool contains no positional feature classes. Rdtc periodically scans short-term memory, generating positional features. If an action repeatedly occurs in short-term memory, Rdtc will identify it as belonging to a class ATar during a particular scan. After collecting the positional features of this ATar, if a large proportion of positional features do not belong to the eqlpool, a detected signal is generated. Otherwise, no detected signal is generated. The issuance of a detected signal indicates that Rdtc has discovered a new recurring action.

[0162] If a detected signal is issued, Lgen immediately operates, using the short-term memory generated by Rdtc to create a dataset, and using the dataset to create an ISNN network according to the method given in the patent, forming a new long-term memory symbolizing a completely new action.

[0163] Initially, there are no ISNNs in the long-term memory module. Lgen continuously generates ISNNs and adds them to the long-term memory over time. Each long-term memory corresponds to one ISNN network and also to a remembered, recurring action. When feature values ​​are received from the FBR, if a remembered action is reproduced, the corresponding ISNN in the long-term memory will recognize it and output it after the input is complete.

[0164] In summary, in action recognition, the LBRI model will remember unseen actions if they appear multiple times, and will recognize remembered actions if they reappear. Therefore, the LBRI model is an unsupervised action recognition method.

[0165] Based on the same inventive concept, the present invention also provides an action recognition system based on unsupervised neural network LBRI to implement the action recognition method based on unsupervised neural network LBRI, including an LBRI model, wherein the LBRI model is used to solve the problem of action recognition.

[0166] It should be noted that, for the sake of simplicity, the foregoing embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to this application.

[0167] The above embodiments describe the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Modifications and variations made by those skilled in the art without departing from the spirit and scope of the invention should be within the protection scope of the appended claims.

Claims

1. An action recognition method based on unsupervised neural network LBRI, characterized in that, Includes the following steps: S1: Obtain action information and extract feature values; S2: The feature values ​​are transmitted to the LBRI model via the feature bus FB. The feature value is the feature value sequence TSQ input over time. The feature value sequence TSQ is classified into the permutation class ATar. S3: The LBRI model memorizes new permutation class ATar and recognizes old permutation class ATar, thus solving the problem of action recognition; The LBRI model includes a Long Short-Term Memory (LSTMF) network framework module, a biological instinct neural network training module, a recurrent association neural network (RANN) module, and an interpretable sequence spiking neural network (ISNN) module. The Long Short-Term Memory (LSTMF) network framework module includes a short-term memory submodule, a long-term memory submodule, and a biological instinct submodule. The short-term memory submodule is used to process and unconditionally record the input feature values ​​in chronological order. The long-term memory submodule is used to reason about the input feature values. The biological instinct submodule is used to identify recurring, regular feature value sequences and generate new long-term memories. The biological instinct neural network training module includes an Rdtc biological instinct submodule and an Lgen biological instinct submodule. The Rdtc biological instinct submodule is used to periodically retrieve short-term memories and determine whether the sequences there are recurring regular segments, classify them, and indicate them. The Lgen biological instinct submodule is used to generate long-term memories. If Rdtc detects recurring regular segments, it generates new long-term memories based on these segments. The Recurrent Associative Neural Network (RANN) module includes a Differential Recurrent Neural Network (DRNN) submodule, an object memory submodule, and a sequence memory submodule. The Differential Recurrent Neural Network (DRNN) submodule is used to find the trend of feature value changes; the object memory submodule is used to classify the memory of feature values; and the sequence memory submodule is used to simulate the characteristic of biological short-term memory to remember the newest and forget the oldest. The interpretable sequence spiking neural network (ISNN) module includes an inference igear submodule and a non-spiking neuron filter submodule. The inference igear submodule is used to simulate the permutation class ATAr. The non-spiking neuron filter submodule is used to receive the input of the feature value bus FBR and output it to the inference igear submodule.

2. The action recognition method based on the unsupervised neural network LBRI as described in claim 1, characterized in that, The LBRI model will remember several permutation classes ATar. When a memorized permutation class ATar is input into the LBRI model, the LBRI model will recognize it; when a new permutation class ATar is input into the LBRI model, the LBRI model will memorize it.

3. The action recognition method based on unsupervised neural network LBRI as described in claim 1, characterized in that, The receptor in the receptor-brain model RB receives stimuli from the outside world and extracts feature values ​​from them.

4. The action recognition method based on the unsupervised neural network LBRI as described in claim 3, characterized in that, The receptor includes a CNN with fully connected layers removed.

5. An action recognition system based on an unsupervised neural network LBRI, used to implement the action recognition method based on an unsupervised neural network LBRI as described in any one of claims 1 to 4, characterized in that, This includes the LBRI model, which is used to solve the problem of action recognition.

Images