Dendritic information fusion-based neuro-dynamic visual signal denoising method and information storage multiplexing method

Abstract

The application discloses a neural quasi-state visual signal denoising method based on tree-shaped information fusion and an information storage and multiplexing method thereof. The denoising method comprises the following steps: 1. filtering original neural quasi-state visual signals in a time-space window, and organizing time-space information of the neural quasi-state visual signals by using a tree-based data structure; 2. establishing a convolutional neural network model dedicated to tree-shaped information, and fusing the tree-shaped information; and 3. training the established convolutional neural network model on a data set, obtaining optimal model parameters, and removing noise signals in the neural quasi-state visual signals. The information storage and multiplexing method comprises the following steps: deploying the trained model, storing and multiplexing the features of the signal processing results, and realizing recursive neural quasi-state visual signal denoising. The application can realize high-precision and high-speed noise removal of the neural quasi-state visual signals, and the denoised signals can be applied to downstream tasks in related fields.

Description

Technical Field

[0001] The present invention relates to the field of neuromorphic visual signal processing, and more specifically to an event signal denoising method based on tree-like information fusion and a method for storing and reusing its information. Background Art

[0002] Neuromorphic cameras (bionic event cameras) are an emerging type of visual sensor. Unlike traditional cameras, they can independently capture pixel-level brightness changes in an image. Each pixel generates a signal upon detecting a brightness change. The key features of neuromorphic cameras include high dynamic range, high temporal resolution, and low latency. Due to their hardware properties and unique operating mode, neuromorphic cameras generate a large amount of random noise signals during operation. This noise signal exhibits different spatiotemporal relationships than normal signals. Neuromorphic visual signal denoising methods are mainly divided into two categories: traditional algorithms and deep learning-based algorithms.

[0003] Traditional algorithms for denoising neuromorphic visual signals typically design a spatiotemporal window. For a newly generated signal, the algorithm counts the number of neuromorphic visual signals within the window and sets a threshold for distinguishing signal noise from signal noise to achieve denoising. While traditional methods can remove noise signals with low latency, their simplistic logic makes denoising accuracy difficult to guarantee.

[0004] Deep learning-based algorithms extract features from the distribution of all signals within the spatiotemporal window of a newly generated signal, using the learned features to distinguish between signal and noise. Currently available feature extraction methods include those based on convolutional neural networks. This approach can achieve higher denoising accuracy than traditional methods, but due to its complex logic and the need for more computation, denoising speed is difficult to guarantee.

[0005] In general, as a preprocessing part for subsequent downstream tasks of neuromorphic cameras, neuromorphic visual signal denoising requires low requirements for signal processing latency. At the same time, the accuracy of denoising directly affects the quality of downstream tasks. Current traditional algorithms and deep learning algorithms find it difficult to meet both requirements simultaneously. Summary of the Invention

[0006] In order to overcome the shortcomings of the existing technology, the present invention proposes a neuromorphic visual signal denoising method based on tree-like information fusion and its information storage and multiplexing method, in order to achieve high-speed neuromorphic visual signal noise removal and ensure the accuracy of neuromorphic visual signal denoising under low latency conditions.

[0007] In order to achieve the above-mentioned object, the present invention adopts the following technical solutions:

[0008] The invention provides a method for denoising a neuromorphic visual signal based on tree-structured information fusion, which comprises the following steps:

[0009] Step 1: Obtain a neuromorphic visual signal sequence with noise labels and perform preprocessing, including: defining relevant parameters and spatiotemporal windows of the signal, establishing signal association, obtaining signal tree groups, obtaining tree tensors and processing relative positions, pairing tree tensor groups and labels, and outputting them;

[0010] Step 1.1: Define the relevant parameters and time and space windows of the signal:

[0011] Step 1.1.1. Define the relevant parameters of the signal:

[0012] Using a neuromorphic vision camera to obtain a time-ordered sequence of neuromorphic vision signals ES ={ es 1, es 2,..., es i ,..., es n} and its corresponding noise label ES label = { l 1, l 2,..., l i ,..., l n},in, es i Indicates the i neuromorphic visual signals, and es i ={ x i , y i , t i , p i}, x i for es i The pixel horizontal coordinate, y i for es i The pixel vertical coordinate, t i for es i timestamp, p i for es i The polarity ofe i express es i exist ES The index in , then the index sequence is recorded as E ={ e 1, e 2,..., e i ,..., e n}; l i express es i If the noise label l i =1, indicating es i is a noise signal, if l i = 0, indicating es i It is a normal signal; i =1,2,..., n , n is the total number of signals;

[0013] Step 1.1.2, define the spatiotemporal window:

[0014] Defining a spatiotemporal window STW= { wx, wy, wt},in, wx Represents the window distance on the horizontal axis, wy Indicates the window distance on the vertical axis, wt Indicates the window distance on the time axis;

[0015] Step 1.2: Establish single-hop association of signals and associated signal groups ESRelation :

[0016] Step 1.3: Establish multi-hop association of signals and obtain the associated signal tree group ES tree , define the relevant relationships on the tree and obtain the valid bit mask Availmask :

[0017] Step 1.4: Neuromorphic visual signal sequence ES Expand and get the associated signal tree tensor EStreetensor :

[0018] Step 1.5: Convert the total associated signal tree tensor EStreetensor , noisy labels ES label Total valid bit mask Availmask As training data;

[0019] Step 2: Construct a neural network, including: mask batch normalization layer MaskBatchNorm , convolution extraction unit Convlayer and multilayer perceptrons MLP concat :

[0020] Step 2.1, establish N The mask batch normalization layer of the layer is used to normalize the features of different layers and output the aggregated features of each layer:

[0021] Among them, Layer mask batch normalization layer MaskBatchNorm Using formula (1) and formula (2) Layer input feature Teninput Tenmask Perform the operation and get Layer mean Hedi Layer variance var :

[0022] (1)

[0023] (2)

[0024] In formula (1) and formula (2), Represents the element product of the matrix, sum dim=2 Indicates the sum of the second dimension of the matrix;

[0025] Using formula (3), we can get Layer Mask Batch Normalization Layer Output normalized feature MaskBatchNormOutput :

[0026] (3)

[0027] In formula (3), eps represents the parameter to prevent the denominator from being zero;

[0028] Step 2.2: Establish an N-layer convolution extraction unit to fuse the features of different layers and output the normalized features of each layer:

[0029] Among them, the convolution extraction unit Convlayer of the lth layer Using formula (4) and formula (5), the input feature Tenfcinput of the lth layer, which is composed of the features of the parent node and its child nodes, is Process it and get Feature Tensorf after layer aggregation :

[0030] (4)

[0031] (5)

[0032] In formula (4) and formula (5), Tenmask Indicates the Layer validity bit mask, MLPup Indicates the Layer-wise multi-layer perceptron layer, MaskBatchNorm Indicates the Layer mask code batch normalization layer, Maxpooling Indicates the The maximum pooling layer of the layer, MLPeq Indicates the Layers of equal-dimensional multilayer perceptrons, Indicates the Residual connections of layers;

[0033] Step 2.3, use the N-layer convolution extraction unit to extract the signal tree tensor TreeTensor i Perform aggregation to obtain es i The total root node tensor RootTly i ;

[0034] Step 2.4, splicing es i After obtaining the root node tensor of N layers, we get es i ConcatRootTensor i , and input into the multi-layer perceptron MLP concat , and then use formula (6) to get es i The noise probability value pred i :

[0035] (6)

[0036] In formula (6), Sigmoid is the activation function;

[0037] Step 3, based on es i The noise probability value and noisy labels i Construct the i-th binary cross entropy loss function L BCEi, and use the Adam optimizer to train the neural network, calculate the binary cross entropy loss function to backpropagate to the network for reference optimization, and stop training when the binary cross entropy loss function converges to obtain the optimal denoising model for neuromorphic visual signals, and output the probability value of noise, and judge the signal with a probability value greater than 0.5 as noise and remove it.

[0038] The neuromorphic visual signal denoising method based on tree-structured information fusion described in the present invention is also characterized in that step 1.2 includes the following steps:

[0039] Step 1.2.1, extract es i The original associated signal group:

[0040] If es i and the jth neuromorphic visual signal es j Satisfy |x j -x i ∣≤wx∩∣y j -y i ∣≤wy∩∣t j -t i ∣≤wt, then it means es j In es i In the space-time window, remember es j ∈STW(es i ), thus obtaining es i Original associated signal group ES0 i ={j | es j ∈ES, es j ∈STW(es i )};

[0041] Step 1.2.2: original associated signal group ES0 i To group:

[0042] If the original associated signal group ES0 i If the pixel horizontal coordinates and pixel vertical coordinates of the neuromorphic visual signals corresponding to the two indexes are the same, the indexes of the corresponding two neuromorphic visual signals are grouped together to obtain all index groups;

[0043] Step 1.2.3, original associated signal group ES0 i The number of neuromorphic visual signals corresponding to each index group in is constrained:

[0044] Define a hyperparameter store max , traversing ES0 i All index groups, if the number of indexes in the index group exceeds store max, then save the indexes corresponding to the two signals with the largest timestamps, thereby obtaining all the index groups after filtering, and concatenate all the index groups after filtering to form the associated signal group ES1 after the first round of filtering i ;

[0045] Step 1.2.4: After the first round of filtering, the associated signal group ES1 i The total number of constraints:

[0046] Define a hyperparameter relation max If ES1 i The number of indices in exceeds the relation max , then randomly select ES1 i Select relation max indexes, and the remaining indexes are discarded. If ES1 i The number of indexes in relation is not reached max , then use the remaining digits Fill in the blanks to form the associated signal group ES2 after the second round of filtering i , and recorded as the associated signal group ES of the i-th signal i ;

[0047] Step 1.2.5: Define the associated signal group ES with index -1 -1 :

[0048] ES -1 Defined as a relation of length max One-dimensional list, and ES -1 All values ​​in are -1;

[0049] Step 1.2.6: Follow the process of steps 1.2.1 to 1.2.4 to obtain the associated signal group of each neuromorphic visual signal in ES, so as to obtain the total associated signal group ESRelation={ES -1 ,ES1,...ES i ,...ES n}.

[0050] Furthermore, the step 1.3 includes the following steps:

[0051] Step 1.3.1: for the i-th neuromorphic visual signal es i Define a hyperparameter N for the number of layers and create N empty lists layer1,...,layer k ,...,layer N , where layer k represents the kth empty list;

[0052] Step 1.3.2, es i Associated signal group ES i Put it into layer1, and then traverse the layer from k=1 to k=layN-1 k Each number in the ESRelation is indexed and the associated signal group in the ESRelation corresponding to each index is placed in the layer k+1 In, thus generating es i The associated signal tree group Tree i =concat(i,layer1,...layer k ,...layer N ), where concat means concatenating arrays;

[0053] Step 1.3.3, define the parent node of the root node i as itself, and define Tree i The parent node and its child node are a parent-child node pair;

[0054] Step 1.3.4, Tree i All elements that are not -1 are set to , set all elements with a value of -1 to 0, and get es i The corresponding valid bit mask Availmask i ;

[0055] Step 1.3.5: Follow the process of steps 1.3.1 to 1.3.4 to obtain the associated signal tree group and its valid bit mask of each neuromorphic visual signal in ES, thereby obtaining the total associated signal tree group ES. tree ={Tree1,Tree2,...Tree i ,...Tree n} and its total effective bit mask Availmask={Availmask1,Availmask2,...Availmask i ,...Availmask n}.

[0056] Furthermore, the step 1.4 includes the following steps:

[0057] Step 1.4.1. Add the neuromorphic visual signal corresponding to the -1 index to ES, denoted as es -1 ={0,0,0,0}, thus obtaining the extended neuromorphic visual sequence ESAppend ={es -1 ,es1,es2,...,es n};

[0058] Step 1.4.2, initialize the empty list list tensor , from the associated signal tree group Tree i Start traversing from the root node, extract the number c on the currently traversed node and the number f on its parent node, and find the cth neuromorphic visual signal es corresponding to the currently traversed node in ESAppend c and the fth neuromorphic visual signal es f , the relative position of the currently traversed node and its parent node {x c -x f , y c -y f , t c -t f , p c}Add to list in sequence tensor Finally, list tensor After splicing the relative positions in , we get es i The associated signal tree tensor TreeTensor i ; where x c -x f Indicates es c with es f The horizontal coordinate difference, y c -y f represents y c -y f Indicates es c with es f The vertical coordinate difference, t c -t f Indicates es c with es f The time difference, p c Indicates es c Polarity;

[0059] Step 1.4.3, follow the process of step 1.4.2 to obtain the associated signal tree tensor of each neuromorphic visual signal in ES, and record it as the total associated signal tree tensor EStreetensor={TreeTensor1,TreeTensor2,...TreeTensor i ,...TreeTensor n}.

[0060] Furthermore, the step 2.3 includes the following steps:

[0061] Step 2.3.1, The signal tree tensor of the layer is recorded as TreeT l,i, where the signal tree tensor at layer 0 is recorded as TreeT 0,i , and initialize TreeT 0,i =TreeTensor i ;

[0062] Step 2.3.2, using The convolution extraction unit of the layer -1 layer signal tree tensor is aggregated to obtain the The signal tree tensor of the layer and save the root node tensor:

[0063] Traverse -1 layer of signal tree tensor on each parent-child node features and features corresponding to the effective bit mask, and use Convlayer Process them one by one to get the first Layer signal tree tensor TreeT l,i , and save the first Layer signal tree tensor TreeT l,i Tensor RootT of the root node l,i ;

[0064] Step 2.3.2: After recursively processing the signal tree tensor using the N-layer convolution extraction unit according to the process of step 2.3.1, the root node tensor corresponding to the N-layer signal tree tensor is obtained, which is recorded as es i The total root node tensor RootTly i ={RootT 1,i ,...RootT l,i ,...RootT N,i )}.

[0065] The information storage and reuse method of the present invention based on the neuromorphic visual signal denoising method of the tree-like information fusion comprises the following steps:

[0066] Step 4: Build an inference network to make convolution reusable:

[0067] Step 4.1: Construct registers of different layers on a pixel-by-pixel basis and initialize the registers of each layer:

[0068] Step 4.1.1. Create registers at different levels for pixels at different positions on the neuromorphic vision camera array to store the features of each level generated by signal calculations at different pixels:

[0069] Set store for each pixel position separately maxGroup register groups, each register group consists of a time register, a polarity register and an N-layer feature memory, where the feature memory of the k-th layer is used to store the features extracted by the corresponding k-th layer convolution extraction unit; k∈[1,N];

[0070] Step 4.1.2, register initialization:

[0071] At the beginning, all register groups are initialized, where the time register is set to -1, the polarity register is set to 0, and the feature memories of the N layers are all initialized to all 0s;

[0072] Whenever a new neuromorphic visual signal es new = {x new , y new , t new , p new} comes, check the store of all pixels max The time register of the group, and the storage value of the time register is less than t new -w t Initialize the register group; where x new Indicates es new The pixel horizontal coordinate, y new Indicates es new The pixel vertical coordinate, t new Indicates es new The timestamp of p new Indicates es new Polarity;

[0073] Step 4.2, initialization of new signal:

[0074] For a neuromorphic visual signal es new , through x new , y new Find the pixel corresponding to the position of the neuromorphic vision camera array, recorded as pixel new , for pixel new store max The register group corresponding to the minimum storage value of the time register in the group register group is initialized, and the initialized register group is used to store es new Compute the resulting features;

[0075] Step 4.3: Find the correlation between the new signal and the past signal:

[0076] Using the spatiotemporal window STW to es new Processing is performed to establish the relationship between the neuromorphic visual information stored on the neuromorphic visual camera array and es newThe association of the pixels at different positions on the neuromorphic vision camera array has a horizontal coordinate range greater than x new -wx and less than x new +wx, the vertical coordinate range is greater than y new -wy and less than y new +wy all pixels corresponding to the time register of the register group, and the storage value in the time register is greater than t new -wt's register group is marked as "hit", indicating that the signal stored in the corresponding memory group is in es new The space-time window STW is within the range of es, so that the signals stored in all hit register groups are taken as new associated signals;

[0077] If the number of hit signals in the spatiotemporal window STW is higher than the relation max Then randomly select relation max A signal as es new associated signals;

[0078] Step 4.4: Feature aggregation and storage of new signals:

[0079] Use the trained N-layer convolution extraction unit to extract es new and with es new The associated signals are aggregated, thus es new with es new The horizontal and vertical coordinates of the associated signals and the time constitute the signal tree tensor TreeT at the 0th layer. 0,new ;

[0080] Using the kth convolution extraction unit Convlayer of the optimal denoising model k For the signal tree tensor TreeT of the k-1 layer k-1,new Perform feature aggregation, obtain the result of the k-th layer feature aggregation and write it to store es new In the feature register of the kth layer;

[0081] Will be es new with es new The value of the feature register of the k-th layer of the associated signal constitutes the signal tree tensor TreeT of the k-th layer k,new , so that starting from k=1 and ending at N, the results of feature aggregation at each layer are written to store es new The feature register group completes the feature aggregation:

[0082] Step 4.5, will be used to store es new After concatenating the feature aggregation results stored in all feature registers, we get es newThe concatenated root node tensor ConcatRootTensor new , and use the MLP of the optimal denoising model concat ConcatRootTensor new After processing, the Sigmoid function outputs es new The noise probability pred new .

[0083] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the method, and the processor is configured to execute the program stored in the memory.

[0084] The present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program executes the steps of the method when executed by a processor.

[0085] Compared with the prior art, the beneficial effects of the present invention are embodied in:

[0086] 1. This paper proposes a technology for organizing spatiotemporal features and performing information fusion based on a tree-based data structure. Compared with traditional convolutional neural networks, the tree-based data structure enhances the network's spatiotemporal receptive field and its ability to extract spatiotemporal features. It also proposes a convolution head dedicated to tree-like information, ultimately improving the accuracy of noise removal.

[0087] 2. The present invention proposes an information storage and reuse technology. Compared with traditional convolutional neural networks, by storing the network's calculation results and reusing the previous network's processing results when new signals arrive, it can greatly reduce the network's computational overhead during the inference process and ultimately improve the speed of noise removal.

[0088] 3. The present invention proposes a special training strategy to ensure that the algorithm is strictly consistent with the inference process during the feedforward propagation process during training. At the same time, during the backpropagation process, by masking the invalid bits, the influence of the invalid bit mask on the algorithm operation results is blocked, thereby improving the training speed while ensuring the accuracy of the denoising method. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 Schematic diagram of the process of the present invention;

[0090] Figure 2 A schematic diagram of a signal tree corresponding to a signal obtained by the pre-order traversal method of the present invention;

[0091] This is a convolution structure diagram of the present invention;

[0092] Figure 3 is the overall network reasoning diagram of the present invention;

[0093] Figure 4 Schematic diagram of the inference process of the present invention. DETAILED DESCRIPTION

[0094] In this embodiment, a neuromorphic visual signal denoising algorithm based on tree-like information fusion mainly uses the spatiotemporal window mechanism to establish the connection between the signal and the past signal, and then uses the convolution extraction unit to aggregate the information layer by layer to generate the final spatiotemporal features, and uses the spatiotemporal features to achieve classification. The general process of the algorithm is as follows Figure 5 As shown, the steps are as follows:

[0095] Step 1: Obtain a neuromorphic visual signal sequence with noise labels and perform preprocessing, including: defining relevant parameters and spatiotemporal windows of the signal, establishing signal association, obtaining signal tree groups, obtaining tree tensors and processing relative positions, pairing tree tensor groups and labels, and outputting them;

[0096] Step 1.1: Define the relevant parameters and time and space windows of the signal:

[0097] Step 1.1.1. Define the relevant parameters of the signal:

[0098] Using a neuromorphic vision camera to obtain a time-ordered sequence of neuromorphic vision signals Figure 1 ={ ES 1, es 2,..., es i ,..., es n} and its corresponding noise label = { l 1, l 2,..., l i ,..., l n},in, es i Indicates the i neuromorphic visual signals, and ESlabel i ={ x i , y i , t i , p i}, x i for es iThe pixel horizontal coordinate, y i for es i The pixel vertical coordinate, t i for es i timestamp, p i for es i The polarity of e i express es i exist es The index in , then the index sequence is recorded as E ={ e 1, e 2,..., e i ,..., e n}; l i express es i If the noise label l i =1, indicating ES i is a noise signal, if l i = 0, indicating es i It is a normal signal; i =1,2,..., n , n is the total number of signals;

[0099] Step 1.1.2, define the spatiotemporal window:

[0100] Defining a spatiotemporal window es { es},in, STW= Represents the window distance on the horizontal axis, wx, wy, wt Indicates the window distance on the vertical axis, wx Indicates the window distance on the time axis. In this embodiment wy 6. wt 6. wx= 0.02.

[0101] Step 1.2: Establish single-hop association of signals and associated signal groups:

[0102] Step 1.2.1. Extraction wy= i The original associated signal group:

[0103] if wt= i Hedi j neuromorphic visual signals es j Satisfaction x j - x i ∣≤ es ∩∣ y j - y i ∣≤ es ∩∣ t j - t i ∣≤ wx , it means wy j exist wt i In the space-time window, record es j ∈ es ( es i ), thus obtaining STW i The original associated signal group es i ={ j | es j ∈ ES0 , es j ∈ ES ( es i )};

[0104] Step 1.2.2: original correlation signal group STW i To group:

[0105] If the original associated signal group es i If the pixel horizontal coordinates and pixel vertical coordinates of the neuromorphic visual signals corresponding to the two indexes are the same, the indexes of the corresponding two neuromorphic visual signals are grouped together to obtain all index groups;

[0106] Step 1.2.3: original correlation signal group ES0 i The number of neuromorphic visual signals corresponding to each index group in is constrained:

[0107] Define a hyperparameter ES0 max , traverse ES0i All index groups, if the number of indexes in the index group exceeds store max , then save the indexes corresponding to the two signals with the largest timestamps, thereby obtaining all the index groups after filtering, and concatenate all the index groups after filtering to form the associated signal group after the first round of filtering ES0 i In this embodiment, store max is 2.

[0108] Step 1.2.4: After the first round of filtering, the associated signal group ES1 i The total number of constraints:

[0109] Define a hyperparameter store max ,if ES1 i The number of indexes in exceeds relation max , then randomly from ES1 i Select relation max indexes, and the remaining indexes are discarded. If ES1 i The number of indexes in the relation max , then the remaining bits are filled with -1 to form the associated signal group after the second round of filtering ES1 i , and record it as i Signal group associated with a signal relation i In this embodiment, ES2 max is 16;

[0110] Step 1.2.5: Define the associated signal group ES with index -1 -1 :

[0111] ES -1 Defined as a length ES max One-dimensional list, and ES -1 All values ​​in are -1;

[0112] Step 1.2.6: Follow the process from step 1.2.1 to step 1.2.4 to obtain relation The associated signal group of each neuromorphic visual signal in , thus obtaining the total associated signal group relation ={ ES -1 , ESRelation 1,... ESi ,... ES n}.

[0113] Step 1.3: Establish multi-hop association of signals and obtain associated signal tree group, define the correlation relationship on the tree and obtain the valid bit mask, such as ES As shown, for the convenience of drawing ES The root node is recorded as a, and the other nodes are represented simply as follows:

[0114] Step 1.3.1, for i neuromorphic visual signals Figure 2 i Define a layer's hyperparameters N ,Establish N An empty list Figure 2 1,..., es k ,..., layer N ,in, layer k Indicates the k An empty list. In this embodiment, N is 3;

[0115] Step 1.3.2, layer i Associated signal group layer i Put in es 1, then from k =1Start to k = ES 1 ends, traverse layer k Each number in the , and use each number as an index, and each index corresponds to layN- The associated signal group in layer k+1 , thus generating ESRelation i Associated signal tree group layer i = es ( i , Tree 1,... concat k ,... layer layN ),in, layer Indicates concatenation of arrays;

[0116] Step 1.3.3, define the root node i The parent node is itself, define layer iThe parent node and its child node are a parent-child node pair;

[0117] Step 1.3.4, concat i All elements that are not -1 are set to 1, and all elements that are -1 are set to 0, so that we get Tree i The corresponding valid bit mask Tree i ;

[0118] Step 1.3.5: Follow the process from step 1.3.1 to step 1.3.4 to obtain es The associated signal tree group and its valid bit mask of each neuromorphic visual signal in , so as to obtain the total associated signal tree group Availmask tree ={ ES 1, ES 2,... Tree i ,... Tree n} and its total valid bit mask Tree ={ Tree 1, Availmask 2,... Availmask i ,... Availmask n}.

[0119] Step 1.4: Neuromorphic visual signal sequence Availmask Expand and get the associated signal tree tensor:

[0120] Step 1.4.1, Availmask The neuromorphic visual signal corresponding to the -1 index is added to ES -1 ={0,0,0,0}, thus obtaining the extended neuromorphic visual sequence ES ={ es -1 , ESAppend 1, 2,..., es n};

[0121] Step 1.4.2, initialize an empty list es tensor , from the associated signal tree group i Start traversing from the root node and extract the data on the currently traversed node. c and the number on its parent node f , and find the node currently traversed ines The corresponding c neuromorphic visual signals es c Hedi f neuromorphic visual signals list f , the relative position of the currently traversed node and its parent node { x c - x f , y c - y f , t c - t f , p c} are added to Tree tensor In the end, ESAppend tensor After splicing the relative positions in es i The associated signal tree tensor of es i ;in, x c - x f express list c and list f The horizontal axis difference, y c - y f express y c - y f express es c and TreeTensor f The vertical coordinate difference, t c - t f express es c and es f The time difference, p c express es c Polarity;

[0122] Step 1.4.3: Follow the process of step 1.4.2 to obtain esThe associated signal tree tensor of each neuromorphic visual signal in is recorded as the total associated signal tree tensor es ={ es 1, es 2,... ES i ,... EStreetensor n}.

[0123] Step 1.5: Convert the total associated signal tree tensor TreeTensor , noisy labels TreeTensor label Total valid bit mask TreeTensor As training data, used for network training.

[0124] Step 2: Construct a neural network, including: mask batch normalization layer TreeTensor , convolution extraction unit EStreetensor and multilayer perceptrons ES concat :

[0125] Step 2.1, establish N The mask batch normalization layer of the layer is used to normalize the features of different layers:

[0126] Among them, Layer mask batch normalization layer MaskBatchNorm Using formula (1) and formula (2) Layer input feature Teninput Tenmask Perform the operation and get Layer mean Hedi Layer variance var :

[0127] (1)

[0128] (2)

[0129] In formula (1) and formula (2), Represents the element product of the matrix, sum dim=2 Indicates the sum of the second dimension of the matrix;

[0130] Using formula (3), we can get Layer Mask Batch Normalization Layer Output normalized feature MaskBatchNormOutput :

[0131] (3)

[0132] In formula (3), eps=1e -16 Represents a parameter that prevents the denominator from being zero;

[0133] Step 2.2: Establish an N-layer convolution extraction unit to fuse the features of different layers and output the normalized features of each layer, such as Availmask As shown:

[0134] Among them, the convolution extraction unit Convlayer of the lth layer Using formula (4) and formula (5), the input feature Tenfcinput of the lth layer, which is composed of the features of the parent node and its child nodes, is Process it and get Feature Tensorf after layer aggregation :

[0135] (4)

[0136] (5)

[0137] In formula (4) and formula (5), Tenmask Indicates the Layer validity bit mask, MLPup Indicates the Layer-wise multi-layer perceptron layer, MaskBatchNorm Indicates the Layer mask code batch normalization layer, Maxpooling Indicates the The maximum pooling layer of the layer, MLPeq Indicates the Layers of equal-dimensional multilayer perceptrons, Indicates the Residual connections of layers;

[0138] Step 2.3, use the N-layer convolution extraction unit to extract the signal tree tensor TreeTensor i Perform aggregation to obtain es i The total root node tensor RootTly i ,like MaskBatchNorm As shown;

[0139] Step 2.3.1, The signal tree tensor of the layer is recorded as TreeT l,i , where the signal tree tensor at layer 0 is recorded as TreeT 0,i , and initialize TreeT 0,i=TreeTensor i ;

[0140] Step 2.3.2, using The convolution extraction unit of the layer -1 layer signal tree tensor is aggregated to obtain the The signal tree tensor of the layer and save the root node tensor, such as Convlayer As shown in the figure, after the feature aggregation of the signal tree tensor at layer 0, the signal tree tensor at layer 1 is obtained. The dotted line on the signal tree tensor in the figure represents the path of information aggregation. The final signal tree tensor at layer 1 is the parent node of the signal tree tensor at layer 0, but the content stored in layer 1 is different from that in layer 0. Layer 1 stores the information after aggregation:

[0141] Traverse -1 layer of signal tree tensor on each parent-child node features and features corresponding to the effective bit mask, and use Convlayer Process them one by one to get the first Layer signal tree tensor TreeT l,i , and save the first Layer signal tree tensor TreeT l,i Tensor RootT of the root node l,i ;

[0142] Step 2.3.2: After recursively processing the signal tree tensor using the N-layer convolution extraction unit according to the process of step 2.3.1, the root node tensor corresponding to the N-layer signal tree tensor is obtained, which is recorded as es i The total root node tensor RootTly i ={RootT 1,i ,...RootT l,i ,...RootT N,i )},like MLP As shown in the middle, the root node on the signal tree is saved separately. For the root node, this architecture can gradually enlarge the receptive field, as shown in the dotted frame at the bottom of the figure;

[0143] Step 2.4, splicing es i After obtaining the root node tensor of N layers, we get es i ConcatRootTensor i , and input into the multi-layer perceptron MLP concat , and then use formula (6) to get es i The noise probability value pred i ,like Figure 3As shown in the upper part, the features after feature aggregation are sent to the multi-layer perceptron through feature splicing. Figure 4 The function obtains the probability value of the final noise:

[0144] (6)

[0145] In formula (6), Sigmoid is the activation function;

[0146] Step 3, based on es i The noise probability value and noisy labels i Construct the i-th binary cross entropy loss function L BCEi , and use the Adam optimizer to train the neural network, calculate the binary cross entropy loss function to backpropagate to the network for reference optimization, and stop training when the binary cross entropy loss function converges to obtain the optimal denoising model for neuromorphic visual signals, and output the probability value of noise, and judge the signal with a probability value greater than 0.5 as noise and remove it.

[0147] In this embodiment, an information storage and reuse method for a neuromorphic visual signal denoising method based on tree-structured information fusion includes the following steps:

[0148] Step 4: Build an inference network to make convolution reusable:

[0149] Step 4.1: Construct different layers of registers on a pixel-by-pixel basis and initialize the registers:

[0150] Step 4.1.1, construct different layers of registers on a pixel-by-pixel basis, and establish storage units for pixels at different positions on the neuromorphic vision camera array to store the features of each layer generated by signal calculations on different pixels, such as Figure 4 As shown, for the sake of simplicity, the polarity register is not drawn, and only one set of feature registers for each pixel is drawn:

[0151] The positions of different pixels in the neural neuromorphic vision camera are encoded by the combination of the pixel horizontal and vertical coordinates. Figure 4 max A register group consists of a time register, a polarity register and N feature memories. The N-layer feature memories are used to store the features extracted by the N-layer convolution extraction unit. The first layer's feature memory layer1 register group, the second layer's feature memory layer2 register group, and so on. NLayer feature memory layerN register group;

[0152] Step 4.1.2, register initialization:

[0153] At the beginning, all register groups are initialized, the time register is set to -1, and the polarity register is set to 0. N The feature memory of the layer is initialized to all 0s;

[0154] Every time a new signal Figure 4 new Come and check all the pixels Sigmoid max The time register of the group, the storage value of the time register is less than t new - w t Initialize the register group;

[0155] Step 4.2, initialization of new signal:

[0156] For a neuromorphic visual signal Figure 5 new = { x new , y new , t new , p new},pass x new , y new Find the pixel corresponding to the position of the neuromorphic vision camera array, recorded as store new , es new Pixel store max The register group with the smallest time register storage value in the group register group is initialized, and the initialized register group is used to store es new Compute the resulting features;

[0157] Step 4.3: Find the correlation between the new signal and the past signal:

[0158] Using spatiotemporal windows pixel right pixel new Processing is performed to establish the association between past neuromorphic visual information and current signals, and the horizontal coordinate range is greater than x new -store , less than x new + es The vertical axis range is greater than y new - STW , less than y new + es The time register of the feature register group corresponding to all pixels is set to t new - wx The characteristic register group is marked as hit, indicating that the signal stored in the hit characteristic memory group is in the new signal wx, new In the spatiotemporal window, the signals stored in all hit feature register groups are used as wy new Associated signals; if the number of hit signals within the spatiotemporal window is higher than wy max Then randomly select wt max A signal as es new Associated signals, such as es Shown on the left.

[0159] Step 4.4: Feature aggregation and storage of new signals:

[0160] and relation new Signal and relation new The associated signal is aggregated using the trained N-layer convolution extraction unit, and the es new Signal and new The horizontal and vertical coordinates of the associated signals and the time component of the signal tree tensor at level 0 Figure 5 0,new , using the k-th convolutional extraction unit of the trained optimal model es k Perform feature aggregation on the signal tree tensor of the k-1th layer and write the result of feature aggregation to storage es new In the k-th layer feature register group of the signal, [[ID=e10]]es new Signal and es new The value of the k-th layer feature register group of the associated signal constitutes the signal tree tensor of the k-th layer TreeT k,new . Starting from k=1 and ending at N, write in sequence for storage Convlayernew The feature register group completes the feature aggregation, such as es As shown in the middle, the arrows in the figure represent the aggregation of features;

[0161] Step 4.5, will be used for storage es new The feature aggregation results stored in the feature register group are spliced ​​together to obtain es new Signal TreeT new like es As shown in the middle dotted box, and the optimal model is trained using Figure 5 concat Processing, through es Function Get es new The noise probability ConcatRootTensor Figure 5 MLP Sigmoid es pred new .

[0162] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.

[0163] In this embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are executed.

[0164] During the specific implementation process, in order to verify the performance of the method proposed in the present invention, some classic traditional methods and deep learning methods were compared, and the accuracy was used as the evaluation indicator.

[0165] The higher the accuracy, the better. We tested the model using DVSNOISE20 street scenes (Alley), bench scenes (Beach), bicycle scenes (Bike), fast-moving correction board scenes (CheckerFast), slow-moving correction board scenes (CheckerSlow), conference scenes (Conference), fast-moving laboratory scenes (Labfast), slow-moving laboratory scenes (Labslow), asphalt ground scenes (Pavers), football field scenes (Soccer), chair scenes (Stairs), and wall scenes (Wall). DVSNOISE20 is a real dataset that captures neuromorphic visual signals generated in different scenes and labels them using auxiliary information such as optical flow and object contour edges. The final experimental results are shown in Table 1:

[0166] Table 1. Experimental results of some scenes in the DVSNOISE20 dataset

[0167]

[0168] As shown in Table 1, the algorithm of the present invention generally outperforms the existing NN traditional denoising algorithm and the EdnCNN deep learning denoising algorithm in terms of indicators, indicating that the method of the present invention has better denoising performance.

[0169] The present invention can be applied to offline event signal denoising tasks, and the algorithm of the present invention can also be modified into hardware and deployed in scenarios requiring high-speed processing.

Claims

1. A neuromorphic visual signal denoising method based on tree-structured information fusion, characterized in that: The steps include: Step 1: Obtain a neuromorphic visual signal sequence with noise labels and perform preprocessing, including: defining relevant parameters and spatiotemporal windows of the signal, establishing signal association, obtaining signal tree groups, obtaining tree tensors and processing relative positions, pairing tree tensor groups and labels, and outputting them; Step 1.1: Define the relevant parameters and time and space windows of the signal: Step 1.1.

1. Define the relevant parameters of the signal: The neuromorphic visual camera is used to obtain the neuromorphic visual signal sequence ES={es1,es2,...,es i ,...,es n } and its corresponding noise label ES label = {l1,l2,...,l i ,...,l n }, where es i represents the i-th neuromorphic visual signal, and es i ={x i , y i , t i , p i }, x i for es i The pixel horizontal coordinate, y i for es i The pixel vertical coordinate, t i for es i The timestamp of p i for es i Polarity, let e i Indicates es i The index in ES is recorded as E={e1,e2,...,e i ,...,e n }; l i Indicates es i Noise label, if l i =1, means es i is a noise signal, if l i = 0, indicating es i is a normal signal; i=1,2,...,n, n is the total number of signals; Step 1.1.2, define the spatiotemporal window: Define the spatiotemporal window STW={wx, wy, wt}, where wx represents the window distance on the horizontal axis, wy represents the window distance on the vertical axis, and wt represents the window distance on the time axis; Step 1.2: Establish the single-hop association of the signal and the associated signal group ESRelation: Step 1.3: Establish multi-hop association of signals and obtain the associated signal tree group ES tree , define the relevant relationships on the tree and obtain the valid bit mask Availmask: Step 1.4: Expand the neuromorphic visual signal sequence ES and obtain the associated signal tree tensor EStreetensor: Step 1.5, the total associated signal tree tensor EStreetensor, the noise label ES label and the total valid bit mask Availmask as training data; Step 2: Construct a neural network, including: MaskBatchNorm, Convlayer, and Multilayer Perceptron (MLP). concat : Step 2.1: Create an N-layer masked batch normalization layer to normalize the features of different layers and output the aggregated features of each layer: Among them, Layer mask batch normalization layer MaskBatchNorm Using formula (1) and formula (2) Layer input feature Teninput Tenmask Perform the operation and get Layer mean Hedi Layer variance var : (1) (2) In formula (1) and formula (2), Represents the element product of the matrix, sum dim=2 Indicates the sum of the second dimension of the matrix; Using formula (3), we can get Layer Mask Batch Normalization Layer Output normalized feature MaskBatchNormOutput : (3) In formula (3), eps represents the parameter to prevent the denominator from being zero; Step 2.2: Establish an N-layer convolution extraction unit to fuse the features of different layers and output the normalized features of each layer: Among them, the convolution extraction unit Convlayer of the lth layer Using formula (4) and formula (5), the input feature Tenfcinput of the lth layer, which is composed of the features of the parent node and its child nodes, is Process it and get Feature Tensorf after layer aggregation : (4) (5) In formula (4) and formula (5), Tenmask Indicates the Layer validity bit mask, MLPup Indicates the Layer-wise multi-layer perceptron layer, MaskBatchNorm Indicates the Layer mask code batch normalization layer, Maxpooling Indicates the The maximum pooling layer of the layer, MLPeq Indicates the Layers of equal-dimensional multilayer perceptrons, Indicates the Residual connections of layers; Step 2.3, use the N-layer convolution extraction unit to extract the signal tree tensor TreeTensor i Perform aggregation to obtain es i The total root node tensor RootTly i ; Step 2.4, splicing es i After obtaining the root node tensor of N layers, we get es i ConcatRootTensor i , and input into the multi-layer perceptron MLP concat , and then use formula (6) to get es i The noise probability value pred i : (6) In formula (6), Sigmoid is the activation function; Step 3, based on es i The noise probability value and noisy labels i Construct the i-th binary cross entropy loss function L BCEi , and use the Adam optimizer to train the neural network, calculate the binary cross entropy loss function to backpropagate to the network for reference optimization, and stop training when the binary cross entropy loss function converges to obtain the optimal denoising model for neuromorphic visual signals, and output the probability value of noise, and judge the signal with a probability value greater than the threshold as noise and remove it.

2. The neuromorphic visual signal denoising method based on tree-structured information fusion according to claim 1 is characterized in that: The step 1.2 includes the following steps: Step 1.2.1, extract es i The original associated signal group: If es i and the jth neuromorphic visual signal es j Satisfy |x j -x i ∣≤wx∩∣y j -y i ∣≤wy∩∣t j -t i ∣≤wt, then it means es j In es i In the space-time window, remember es j ∈STW(es i ), thus obtaining es i Original associated signal group ES0 i ={j | es j ∈ES, es j ∈STW(es i )}; Step 1.2.2: original associated signal group ES0 i To group: If the original associated signal group ES0 i If the pixel horizontal coordinates and pixel vertical coordinates of the neuromorphic visual signals corresponding to the two indexes are the same, the indexes of the corresponding two neuromorphic visual signals are grouped together to obtain all index groups; Step 1.2.3, original associated signal group ES0 i The number of neuromorphic visual signals corresponding to each index group in is constrained: Define a hyperparameter store max , traversing ES0 i All index groups, if the number of indexes in the index group exceeds store max , then save the indexes corresponding to the two signals with the largest timestamps, thereby obtaining all the index groups after filtering, and concatenate all the index groups after filtering to form the associated signal group ES1 after the first round of filtering i ; Step 1.2.4: After the first round of filtering, the associated signal group ES1 i The total number of constraints: Define a hyperparameter relation max If ES1 i The number of indices in exceeds the relation max , then randomly select ES1 i Select relation max indexes, and the remaining indexes are discarded. If ES1 i The number of indexes in relation is not reached max , then use the remaining digits Fill in the blanks to form the associated signal group ES2 after the second round of filtering i , and recorded as the associated signal group ES of the i-th signal i ; Step 1.2.5: Define the associated signal group ES with index -1 -1 : ES -1 Defined as a relation of length max One-dimensional list, and ES -1 All values ​​in are -1; Step 1.2.6: Follow the process of steps 1.2.1 to 1.2.4 to obtain the associated signal group of each neuromorphic visual signal in ES, so as to obtain the total associated signal group ESRelation={ES -1 ,ES1,...ES i ,...ES n }.

3. The neuromorphic visual signal denoising method based on tree-structured information fusion according to claim 2 is characterized in that: The step 1.3 includes the following steps: Step 1.3.1: for the i-th neuromorphic visual signal es i Define a hyperparameter N for the number of layers and create N empty lists layer1,...,layer k ,...,layer N , where layer k represents the kth empty list; Step 1.3.2, es i Associated signal group ES i Put it into layer1, and then traverse the layer from k=1 to k=layN-1 k Each number in the ESRelation is indexed and the associated signal group in the ESRelation corresponding to each index is placed in the layer k+1 In, thus generating es i The associated signal tree group Tree i =concat(i,layer1,...layer k ,...layer N ), where concat means concatenating arrays; Step 1.3.3, define the parent node of the root node i as itself, and define Tree i The parent node and its child node are a parent-child node pair; Step 1.3.4, Tree i All elements that are not -1 are set to , set all elements with a value of -1 to 0, and get es i The corresponding valid bit mask Availmask i ; Step 1.3.5: Follow the process of steps 1.3.1 to 1.3.4 to obtain the associated signal tree group and its valid bit mask of each neuromorphic visual signal in ES, thereby obtaining the total associated signal tree group ES. tree ={Tree1,Tree2,...Tree i ,...Tree n } and its total effective bit mask Availmask={Availmask1,Availmask2,...Availmask i ,...Availmask n }.

4. The neuromorphic visual signal denoising method based on tree-structured information fusion according to claim 3 is characterized in that: The step 1.4 includes the following steps: Step 1.4.

1. Add the neuromorphic visual signal corresponding to the -1 index to ES, denoted as es -1 ={0,0,0,0}, thus obtaining the extended neuromorphic visual sequence ESAppend={es -1 ,es1,es2,...,es n }; Step 1.4.2, initialize the empty list list tensor , from the associated signal tree group Tree i Start traversing from the root node, extract the number c on the currently traversed node and the number f on its parent node, and find the cth neuromorphic visual signal es corresponding to the currently traversed node in ESAppend c and the fth neuromorphic visual signal es f , the relative position of the currently traversed node and its parent node {x c -x f , y c -y f , t c -t f , p c }Add to list in sequence tensor Finally, list tensor After splicing the relative positions in , we get es i The associated signal tree tensor TreeTensor i ; where x c -x f Indicates es c with es f The horizontal coordinate difference, y c -y f represents y c -y f Indicates es c with es f The vertical coordinate difference, t c -t f Indicates es c with es f The time difference, p c Indicates es c Polarity; Step 1.4.3, follow the process of step 1.4.2 to obtain the associated signal tree tensor of each neuromorphic visual signal in ES, and record it as the total associated signal tree tensor EStreetensor={TreeTensor1,TreeTensor2,...TreeTensor i ,...TreeTensor n }.

5. The neuromorphic visual signal denoising method based on tree-structured information fusion according to claim 4 is characterized in that: The step 2.3 includes the following steps: Step 2.3.1, The signal tree tensor of the layer is recorded as TreeT l,i , where the signal tree tensor at layer 0 is recorded as TreeT 0,i , and initialize TreeT 0,i =TreeTensor i ; Step 2.3.2, using The convolution extraction unit of the layer -1 layer signal tree tensor is aggregated to obtain the The signal tree tensor of the layer and save the root node tensor: Traverse -1 layer of signal tree tensor on each parent-child node features and features corresponding to the effective bit mask, and use Convlayer Process them one by one to get the first Layer signal tree tensor TreeT l,i , and save the first Layer signal tree tensor TreeT l,i Tensor RootT of the root node l,i ; Step 2.3.2: After recursively processing the signal tree tensor using the N-layer convolution extraction unit according to the process of step 2.3.1, the root node tensor corresponding to the N-layer signal tree tensor is obtained, which is recorded as es i The total root node tensor RootTly i ={RootT 1,i ,...RootT l,i ,...RootT N,i )}.

6. An information storage and reuse method based on the neuromorphic visual signal denoising method of tree-structured information fusion according to claim 1, characterized in that: The steps include: Step 4. Build an inference network to achieve reusability of convolution: Step 4.

1. Build registers for different layers pixel by pixel and initialize the registers for each layer: Step 4.1.

1. Create registers at different levels for pixels at different positions on the neuromorphic vision camera array to store the features of each level generated by signal calculations at different pixels: Set store for each pixel position separately max Group register groups, each register group consists of a time register, a polarity register and an N-layer feature memory, where the feature memory of the k-th layer is used to store the features extracted by the corresponding k-th layer convolution extraction unit; k∈[1,N]; Step 4.1.2, register initialization: At the beginning, all register groups are initialized, where the time register is set to -1, the polarity register is set to 0, and the feature memories of the N layers are all initialized to all 0s; Whenever a new neuromorphic visual signal es new = {x new , y new , t new , p new } comes, check the store of all pixels max The time register of the group, and the storage value of the time register is less than t new -w t Initialize the register group; where x new Indicates es new The pixel horizontal coordinate, y new Indicates es new The pixel vertical coordinate, t new Indicates es new The timestamp of p new Indicates es new Polarity; Step 4.2, initialization of new signal: For a neuromorphic visual signal es new , through x new , y new Find the pixel corresponding to the position of the neuromorphic vision camera array, recorded as pixel new , for pixel new store max The register group corresponding to the minimum storage value of the time register in the group register group is initialized, and the initialized register group is used to store es new Compute the resulting features; Step 4.3: Find the correlation between the new signal and the past signal: Using the spatiotemporal window STW to es new Processing is performed to establish the relationship between the neuromorphic visual information stored on the neuromorphic visual camera array and es new The association of the pixels at different positions on the neuromorphic vision camera array has a horizontal coordinate range greater than x new -wx and less than x new +wx, the vertical coordinate range is greater than y new -wy and less than y new +wy all pixels corresponding to the time register of the register group, and the storage value in the time register is greater than t new -wt's register group is marked as "hit", indicating that the signal stored in the corresponding memory group is in es new The space-time window STW is within the range of es, so that the signals stored in all hit register groups are taken as new associated signals; If the number of hit signals in the spatiotemporal window STW is higher than the relation max Then randomly select relation max A signal as es new associated signals; Step 4.4: Feature aggregation and storage of new signals: Use the trained N-layer convolution extraction unit to extract es new and with es new The associated signals are aggregated, thus es new with es new The horizontal and vertical coordinates of the associated signals and the time constitute the signal tree tensor TreeT at the 0th layer. 0,new ; Using the kth convolution extraction unit Convlayer of the optimal denoising model k For the signal tree tensor TreeT of the k-1 layer k-1,new Perform feature aggregation, obtain the result of the k-th layer feature aggregation and write it to store es new In the feature register of the kth layer; Will be es new with es new The value of the feature register of the k-th layer of the associated signal constitutes the signal tree tensor TreeT of the k-th layer k,new , so that starting from k=1 and ending at N, the results of feature aggregation at each layer are written to store es new The feature register group completes the feature aggregation: Step 4.5, will be used to store es new After concatenating the feature aggregation results stored in all feature registers, we get es new The concatenated root node tensor ConcatRootTensor new , and use the MLP of the optimal denoising model concat ConcatRootTensor new After processing, the Sigmoid function outputs es new The noise probability pred new .

7. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports a processor to execute the method according to any one of claims 1 to 6, and the processor is configured to execute the program stored in the memory.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are performed.

Images