Neural mimicry visual signal denoising method based on tree information fusion and information storage multiplexing method thereof
Through the neuromimicry visual signal denoising method based on tree information fusion, combined with neural network and information storage multiplexing technology, the problems of noise removal accuracy and speed in the existing technology are solved, and efficient neuromimicry visual signal denoising is achieved.
Patent Information
- Application Number
- CN202411294414.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-14
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2044-09-14
AI Technical Summary
The existing neuromimicry visual signal denoising methods are difficult to achieve high accuracy and low latency at the same time. The traditional algorithms lack the denoising accuracy, while the complex calculation of deep learning algorithms leads to slow denoising speed.
Using a neuromimicry visual signal denoising method based on tree information fusion, the signal association is established by defining the signal's relevant parameters and space-time windows, the signal tree tensor is obtained and the relative position is processed, the neural network is constructed for training, and the mask batch is used. The normalization layer and the convolution extraction unit perform feature fusion to achieve prediction and removal of noise probability, and reduce computational overhead through information storage multiplexing technology.
It improves the accuracy and speed of noise removal, enhances the spatial and temporal receptive field of the network, reduces the computational overhead in the inference process, and realizes efficient neuromimicry visual signal denoising.
Smart Images

Figure CN119027340A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of neuromorphic visual signal processing, and specifically to an event signal denoising method based on tree-like information fusion and a method for storing and reusing its information. Background Art
[0002] Neuromorphic camera (bionic event camera) is an emerging visual sensor. Unlike traditional cameras, it can independently capture pixel-level brightness changes in images. Each pixel will independently generate a signal after detecting brightness changes. The main features of neuromorphic cameras include: high dynamic range, high temporal resolution, and low latency. Due to the hardware properties and unique working mode of neuromorphic cameras, neuromorphic cameras will generate a large amount of random noise signals during operation. The noise signals of neuromorphic cameras show different spatiotemporal relationships than normal signals. Neuromorphic visual signal denoising methods are mainly divided into two categories: traditional algorithms and deep learning-based algorithms.
[0003] In traditional algorithms for denoising neuromorphic visual signals, a spatiotemporal window is usually designed. For a newly generated signal, the number of neuromorphic visual signals in the spatiotemporal window is counted, and a threshold for distinguishing the number of signal noise is set to achieve the purpose of denoising. Traditional methods can remove noise signals with low latency, but due to the relatively simple judgment logic, the accuracy of denoising is difficult to guarantee.
[0004] Based on the deep learning algorithm, the distribution of all signals in the spatiotemporal window of the newly generated signal is extracted, and the signal and noise are distinguished through the learned features. The existing feature extraction methods include methods based on convolutional neural networks. This method can achieve higher denoising accuracy than traditional methods, but because its judgment logic is more complex and requires more calculations, the denoising speed is difficult to guarantee.
[0005] In general, as a preprocessing part of the subsequent downstream tasks of the neuromorphic camera, the denoising of neuromorphic visual signals requires low requirements on the delay of signal processing. At the same time, the accuracy of denoising directly affects the quality of downstream tasks. Current traditional algorithms and deep learning algorithms are difficult to meet these two requirements at the same time. Summary of the invention
[0006] In order to overcome the shortcomings of the prior art, the present invention proposes a neuromorphic visual signal denoising method based on tree-like information fusion and an information storage and multiplexing method thereof, in order to achieve high-speed neuromorphic visual signal noise removal and ensure the accuracy of neuromorphic visual signal denoising under low-latency conditions.
[0007] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme: The invention is characterized in that a neuromorphic visual signal denoising method based on tree-like information fusion comprises the following steps: Step 1, obtaining a neuromorphic visual signal sequence with a noise label and preprocessing it, including: defining relevant parameters and spatiotemporal windows of the signal, establishing signal association, obtaining a signal tree group, obtaining a tree tensor and processing relative positions, pairing the tree tensor group and the label and outputting them; Step 1.1, define the relevant parameters and time and space windows of the signal: Step 1.1.1, define the relevant parameters of the signal: Using a neuromorphic vision camera to obtain a time-ordered sequence of neuromorphic vision signals ES ={ es 1, es 2,..., es i ,..., es n} and its corresponding noise label ES label = { l 1, l 2,..., l i ,..., l n},in, es i Indicates i neuromorphic visual signals, and es i ={ x i , y i , t i , p i}, x i for es i The pixel horizontal coordinate, y i for es i The pixel ordinate, t i for es i timestamp, p i for es i The polarity of e i express es i exist ESThe index in , then the index sequence is recorded as E ={ e 1, e 2,..., e i ,..., e n}; l i express es i If the noise label l i =1, indicating es i is a noise signal, if l i = 0, indicating es i It is a normal signal; i =1,2,..., n , n is the total number of signals; Step 1.1.2, define the space-time window: Defining a spatiotemporal window STW= { wx, wy, wt},in, wx Represents the window distance on the horizontal axis, wy Represents the window distance on the ordinate axis, wt Indicates the window distance on the time axis; Step 1.2: Establish single-hop association of signals and associated signal groups ESRelation : Step 1.3: Establish multi-hop association of signals and obtain associated signal tree group ES tree , define the relevant relationships on the tree and obtain the valid bit mask Availmask : Step 1.4: Neuromorphic visual signal sequence ES Expand and get the associated signal tree tensor EStreetensor : Step 1.5: Convert the total associated signal tree tensor EStreetensor , Noise Label ES label Total valid bit mask Availmask As training data; Step 2: Construct a neural network, including: mask batch normalization layer MaskBatchNorm , convolution extraction unit Convlayer and multilayer perceptron MLP concat : Step 2.1, Establish NThe masked batch normalization layer of the layer is used to normalize the features of different layers and output the aggregated features of each layer: Among them, Layer MaskBatchNorm Using formula (1) and formula (2) Layer input feature Teninput With mask Tenmask Perform the operation and get Layer mean and Layer variance var : (1) (2) In formula (1) and formula (2), Represents the element product of the matrix, sum dim=2 It means to sum the second dimension of the matrix; Using formula (3), we can get Layer Mask Batch Normalization Layer Output normalized features MaskBatchNormOutput : (3) In formula (3), eps represents the parameter to prevent the denominator from being zero; Step 2.2: Establish an N-layer convolution extraction unit to fuse the features of different layers and output the normalized features of each layer: Among them, the convolution extraction unit Convlayer of the lth layer Using equations (4) and (5), we can get the input feature Tenfcinput of the lth layer, which is composed of the features of the parent node and its child nodes. Process it and get Feature Tensorf after layer aggregation : (4) (5) In formula (4) and formula (5), Tenmask Indicates Layer validity bit mask, MLPup Indicates Layer-wise multi-layer perceptron layer, MaskBatchNorm Indicates Layer mask code batch normalization layer, Maxpooling Indicates The maximum pooling layer of the layer, MLPeq Indicates Layers of equal-dimensional multilayer perceptrons, Indicates Residual connections of layers; Step 2.3: Use the N-layer convolutional extraction unit to extract the signal tree tensor TreeTensor i Perform aggregation to obtain es i The total root node tensor RootTly i ; Step 2.4, splicing es i After obtaining the root node tensor of N layers, we get es i ConcatRootTensor i , and input into the multi-layer perceptron MLP concat , and then use formula (6) to get es i The noise probability value pred i : (6) In formula (6), Sigmoid is the activation function; Step 3, based on es i The noise probability value and noise labels i Construct the i-th binary cross entropy loss function L BCEi , and use the Adam optimizer to train the neural network, calculate the binary cross entropy loss function to back-propagate to the network for reference optimization, and stop training when the binary cross entropy loss function converges to obtain the optimal denoising model for neuromorphic visual signals, and output the probability value of noise, and judge the signal with a probability value greater than 0.5 as noise and remove it.
[0008] The neuromorphic visual signal denoising method based on tree-like information fusion described in the present invention is also characterized in that step 1.2 includes the following steps: Step 1.2.1, extract es i The original associated signal group: If es i and the jth neuromorphic visual signal es j Satisfy ∣x j -x i ∣≤wx∩∣y j -y i ∣≤wy∩∣t j -t i ∣≤wt, then it means esj In es i In the space-time window, remember es j ∈STW(es i ), thus obtaining es i The original associated signal group ES0 i ={j | es j ∈ES, es j ∈STW(es i )}; Step 1.2.2: original associated signal group ES0 i To group: If the original associated signal group ES0 i If the pixel abscissa and pixel ordinate of the neuromorphic visual signals corresponding to the two indexes are the same, the indexes of the corresponding two neuromorphic visual signals are grouped together to obtain all index groups; Step 1.2.3: original associated signal group ES0 i The number of neuromorphic visual signals corresponding to each index group in is constrained: Define a hyperparameter store max , traverse ES0 i All index groups, if the number of indexes in the index group exceeds store max , then save the indexes corresponding to the two signals with the largest timestamps, thereby obtaining all the index groups after filtering, and concatenate all the index groups after filtering to form the associated signal group ES1 after the first round of filtering i ; Step 1.2.4: After the first round of filtering, the associated signal group ES1 i The total number of constraints: Define a hyperparameter relation max If ES1 i The number of indexes in exceeds the relation max , then randomly select ES1 i Select relation max indexes, and the remaining indexes are discarded. If ES1 i The number of indexes in relation is not reached max , then use the remaining bits Fill in the blanks to form the associated signal group ES2 after the second round of filtering i , and recorded as the associated signal group ES of the i-th signal i ; Step 1.2.5: Define the associated signal group ES with index -1 -1 : ES -1Defined as a relation of length max A one-dimensional list of ES -1 All values in are -1; Step 1.2.6: Follow the process of steps 1.2.1 to 1.2.4 to obtain the associated signal group of each neuromorphic visual signal in ES, so as to obtain the total associated signal group ESRelation = {ES -1 ,ES1,...ES i ,...ES n}.
[0009] Furthermore, the step 1.3 includes the following steps: Step 1.3.1: for the i-th neuromorphic visual signal es i Define a hyperparameter N for the number of layers and create N empty lists layer1,...,layer k ,...,layer N , where layer k represents the kth empty list; Step 1.3.2, es i The associated signal group ES i Put it into layer1, and then traverse the layer from k=1 to k=layN-1 k Each number in the ESRelation is indexed and the associated signal group in the ESRelation corresponding to each index is put into the layer k+1 In order to generate es i Tree group of associated signals i =concat(i,layer1,...layer k ,...layer N ), where concat means concatenating arrays; Step 1.3.3, define the parent node of the root node i as itself, and define Tree i The parent node and its child node are a parent-child node pair; Step 1.3.4: Tree i All elements that are not -1 are set to , set all elements with a value of -1 to 0, and get es i The corresponding valid bit mask Availmask i ; Step 1.3.5: Obtain the associated signal tree group and its valid bit mask of each neuromorphic visual signal in ES according to the process of steps 1.3.1 to 1.3.4, so as to obtain the total associated signal tree group ES. tree={Tree1,Tree2,...Tree i ,...Tree n} and its total effective bit mask Availmask={Availmask1,Availmask2,...Availmask i ,...Availmask n}.
[0010] Furthermore, the step 1.4 includes the following steps: Step 1.4.1. Add the neuromorphic visual signal corresponding to the -1 index to ES, denoted as es -1 ={0,0,0,0}, thus obtaining the extended neuromorphic visual sequence ESAppend ={es -1 ,es1,es2,...,es n}; Step 1.4.2, initialize the empty list list tensor , from the associated signal tree group Tree i Start traversing from the root node of , extract the number c on the currently traversed node and the number f on its parent node, and find the cth neuromorphic visual signal es corresponding to the currently traversed node in ESAppend c and the fth neuromorphic visual signal es f , the relative position of the currently traversed node and its parent node {x c -x f , y c -y f , t c -t f , p c}Add to list in sequence tensor Finally, list tensor After splicing the relative positions in es i The associated signal tree tensor TreeTensor i ; where x c -x f Indicates c With es f The horizontal coordinate difference, y c -y f Represents y c -y f Indicates c With es f The vertical coordinate difference, t c -t f Indicates c With es fThe time difference, p c Indicates c Polarity; Step 1.4.3, follow the process of step 1.4.2 to obtain the associated signal tree tensor of each neuromorphic visual signal in ES, and record it as the total associated signal tree tensor EStreetensor={TreeTensor1,TreeTensor2,...TreeTensor i ,...TreeTensor n}.
[0011] Further, the step 2.3 includes the following steps: Step 2.3.1. The signal tree tensor of the layer is denoted as TreeT l,i , where the signal tree tensor at layer 0 is denoted as TreeT 0,i , and initialize TreeT 0,i =TreeTensor i ; Step 2.3.2: Use The convolutional extraction unit of the layer -1 layer signal tree tensor is aggregated to get the The signal tree tensor of the layer and save the root node tensor: Traverse -1 layer of signal tree tensor on each parent-child node pair and the valid bit mask corresponding to the feature, and use Convlayer After processing one by one, the first Layer signal tree tensor TreeT l,i , and save the first Layer signal tree tensor TreeT l,i Tensor RootT of the root node l,i ; Step 2.3.2: After recursively processing the signal tree tensor using the N-layer convolution extraction units in accordance with the process of step 2.3.1, the root node tensor corresponding to the N-layer signal tree tensor is obtained, which is recorded as es i The total root node tensor RootTly i ={RootT 1,i ,...RootT l,i ,...RootT N,i )}.
[0012] The information storage and reuse method of the neuromorphic visual signal denoising method based on the tree-like information fusion of the present invention comprises the following steps: Step 4: Build an inference network to make convolution reusable: Step 4.1: Construct registers of different layers pixel by pixel and initialize registers of each layer: Step 4.1.1. Create registers of different layers for pixels at different positions on the neuromorphic vision camera array to store the features of each layer generated by signal calculations on different pixels: Set store for each pixel position separately max A register group, each register group is composed of a time register, a polarity register and an N-layer feature memory, wherein the feature memory of the k-th layer is used to store the features extracted by the corresponding k-th layer convolution extraction unit; k∈[1,N]; Step 4.1.2, register initialization: At the beginning, all register groups are initialized, where the time register is set to -1, the polarity register is set to 0, and the feature memories of the N layers are all initialized to all 0; Whenever a new neuromorphic visual signal es new = {x new , y new , t new , p new} comes, check the store of all pixels max The time register of the group, and the storage value of the time register is less than t new -w t Initialize the register group; where x new Indicates new The pixel horizontal coordinate, y new Indicates new The pixel ordinate, t new Indicates new The timestamp, p new Indicates new Polarity; Step 4.2, initialization of new signal: For a neuromorphic visual signal es new , through x new , y new Find the pixel corresponding to the position of the neuromorphic vision camera array, denoted as pixel new , for pixel new store max The register group corresponding to the smallest storage value of the time register in the group register group is initialized, and the initialized register group is used to store es new Calculate the resulting features; Step 4.3: Find the correlation between the new signal and the past signal: Using the spatiotemporal window STW to es new Processing is performed to establish the neuromorphic visual information stored on the neuromorphic visual camera array and es new The association of traversing the horizontal coordinate range of pixels at different positions on the neuromorphic vision camera array is greater than x new -wx and less than x new +wx, the vertical coordinate range is greater than y new -wy and less than y new +wy all pixels corresponding to the time register of the register group, and the storage value in the time register is greater than t new -wt's register group is marked as "hit", indicating that the signal stored in the corresponding memory group is in es new The space-time window STW is within the space-time window STW, so that the signals stored in all the register sets hit are taken as the same as es new associated signals; If the number of hit signals in the spatiotemporal window STW is higher than the relation max If there are , then randomly select relation max The signal is as new associated signals; Step 4.4: Feature aggregation and storage of new signals: Use the trained N-layer convolution extraction unit to extract es new As well as with es new The associated signals are feature aggregated, thus es new With es new The horizontal and vertical coordinates of the associated signals and the time form the signal tree tensor TreeT at the 0th level. 0,new ; Using the kth convolution extraction unit Convlayer of the optimal denoising model k For the signal tree tensor TreeT of the k-1th layer k-1,new Perform feature aggregation, obtain the result of the k-th layer feature aggregation and write it to store es new In the feature register of the kth layer; Will be by es new With es new The value of the feature register of the k-th layer of the associated signal constitutes the signal tree tensor TreeT of the k-th layer k,new , so that starting from k=1 and ending at N, the results of feature aggregation at each layer are written to store es new The feature register group completes the feature aggregation: Step 4.5, will be used to store es new After concatenating the feature aggregation results stored in all feature registers, es is obtained.new The concatenated root node tensor ConcatRootTensor new , and use the MLP of the optimal denoising model concat ConcatRootTensor new After processing, the Sigmoid function outputs es new The noise probability pred new .
[0013] An electronic device of the present invention includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the method, and the processor is configured to execute the program stored in the memory.
[0014] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program executes the steps of the method when executed by a processor.
[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The present invention proposes a technology for organizing spatiotemporal features and performing information fusion based on a tree-based data structure. Compared with traditional convolutional neural networks, the tree-based data structure enhances the network's spatiotemporal receptive field and the network's ability to extract spatiotemporal features. A convolution head dedicated to tree-like information is proposed, which ultimately improves the accuracy of noise removal.
[0016] 2. The present invention proposes an information storage and reuse technology. Compared with the traditional convolutional neural network, by storing the calculation results of the network and reusing the previous network processing results when a new signal arrives, it can greatly reduce the computational overhead of the network during the reasoning process and ultimately improve the speed of noise removal.
[0017] 3. The present invention proposes a special training strategy to ensure that the algorithm is strictly consistent with the inference process during the feedforward propagation process during the training process. At the same time, during the back propagation process, the invalid bits are masked to block the influence of the invalid bit mask on the algorithm operation results, thereby improving the training speed and ensuring the accuracy of the denoising method. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 It is a schematic diagram of the process of the present invention; Figure 2 A schematic diagram of a signal tree corresponding to a signal obtained by the pre-order traversal method of the present invention; Figure 3 It is a convolution structure diagram of the present invention; Figure 4 is the overall network reasoning diagram of the present invention; Figure 5 Schematic diagram of the inference process of the present invention. DETAILED DESCRIPTION
[0019] In this embodiment, a neuromorphic visual signal denoising algorithm based on tree-like information fusion mainly uses the spatiotemporal window mechanism to establish the connection between the signal and the past signal, and then uses the convolution extraction unit to aggregate the information layer by layer to generate the final spatiotemporal features, and uses the spatiotemporal features to achieve classification. The general process of the algorithm is as follows Figure 1 As shown, the steps are as follows: Step 1, obtaining a neuromorphic visual signal sequence with a noise label and preprocessing it, including: defining relevant parameters and spatiotemporal windows of the signal, establishing signal association, obtaining a signal tree group, obtaining a tree tensor and processing relative positions, pairing the tree tensor group and the label and outputting them; Step 1.1, define the relevant parameters and time and space windows of the signal: Step 1.1.1, define the relevant parameters of the signal: Using a neuromorphic vision camera to obtain a time-ordered sequence of neuromorphic vision signals ES ={ es 1, es 2,..., es i ,..., es n} and its corresponding noise label ESlabel = { l 1, l 2,..., l i ,..., l n},in, es i Indicates i neuromorphic visual signals, and es i ={ x i , y i , t i , p i}, x i for es i The pixel horizontal coordinate, y i for es i The pixel ordinate, t i for es itimestamp, p i for es i The polarity of e i express es i exist ES The index in , then the index sequence is recorded as E ={ e 1, e 2,..., e i ,..., e n}; l i express es i If the noise label l i =1, indicating es i is a noise signal, if l i = 0, indicating es i It is a normal signal; i =1,2,..., n , n is the total number of signals; Step 1.1.2, define the space-time window: Defining a spatiotemporal window STW= { wx, wy, wt},in, wx Represents the window distance on the horizontal axis, wy Represents the window distance on the ordinate axis, wt Indicates the window distance on the time axis. In this embodiment wx= 6. wy= 6. wt= 0.02.
[0020] Step 1.2: Establish single-hop association of signals and associated signal groups: Step 1.2.1. Extraction es i The original associated signal group: if es i and j Neuromorphic visual signals es j Satisfaction | x j - x i ∣≤ wx ∩∣ yj - y i ∣≤ wy ∩∣ t j - t i ∣≤ wt , then it means es j exist es i In the space-time window, record es j ∈ STW ( es i ), thus obtaining es i The original associated signal group ES0 i ={ j | es j ∈ ES , es j ∈ STW ( es i )}; Step 1.2.2: original correlation signal group ES0 i To group: If the original associated signal group ES0 i If the pixel abscissa and pixel ordinate of the neuromorphic visual signals corresponding to the two indexes are the same, the indexes of the corresponding two neuromorphic visual signals are grouped together to obtain all index groups; Step 1.2.3: original associated signal group ES0 i The number of neuromorphic visual signals corresponding to each index group in is constrained: Define a hyperparameter store max , traverse ES0 i All index groups, if the number of indexes in the index group exceeds store max , then save the indexes corresponding to the two signals with the largest timestamps, thereby obtaining all the index groups after filtering, and concatenate all the index groups after filtering to form the associated signal group after the first round of filtering ES1 i In this embodiment, store max is 2.
[0021] Step 1.2.4: After the first round of filtering, the associated signal group ES1 i The total number of constraints: Define a hyperparameter relation max ,if ES1 i The number of indexes in exceeds relation max , then randomly from ES1 i Select relation max indexes, and the remaining indexes are discarded. ES1 i The number of indexes in the relation max , the remaining bits are filled with -1 to form the associated signal group after the second round of filtering ES2 i , and recorded as i Signal group associated with a signal ES i In this embodiment, relation max is 16; Step 1.2.5: Define the associated signal group ES with index -1 -1 : ES -1 Defined as the length relation max A one-dimensional list of ES -1 All values in are -1; Step 1.2.6: Follow the process from step 1.2.1 to step 1.2.4 to obtain ES The associated signal group of each neuromorphic visual signal in , so as to obtain the total associated signal group ESRelation ={ ES -1 , ES 1,... ES i ,... ES n}.
[0022] Step 1.3, establish the multi-hop association of the signal and obtain the associated signal tree group, define the correlation relationship on the tree and obtain the valid bit mask, such as Figure 2 For the convenience of drawing Figure 2 The root node is recorded as a, and other nodes are represented in simplified form: Step 1.3.1, for i Neuromorphic visual signals es i Define a layer's hyperparameters N,Establish N An empty list layer 1,..., layer k ,..., layer N ,in, layer k Indicates k An empty list. In this embodiment, N is 3; Step 1.3.2: es i The associated signal group ES i Put in layer 1, then from k =1Start to k = layN- 1 End, traverse layer k Each number in the , and use each number as an index, and each index corresponds to ESRelation The associated signal group in is placed in layer k+1 , thus generating es i The associated signal tree group Tree i = concat ( i , layer 1,... layer k ,... layer layN ),in, concat Indicates concatenation of arrays; Step 1.3.3, define the root node i The parent node of is itself, define Tree i The parent node and its child node are a parent-child node pair; Step 1.3.4: Tree i All elements that are not -1 are set to 1, and all elements that are -1 are set to 0, so that es i The corresponding valid bit mask Availmask i ; Step 1.3.5: Follow the process from step 1.3.1 to step 1.3.4 to obtain ES The associated signal tree group and its valid bit mask of each neuromorphic visual signal in , so as to obtain the total associated signal tree group ES tree ={ Tree 1, Tree 2,...Tree i ,... Tree n} and its total valid bit mask Availmask ={ Availmask 1, Availmask 2,... Availmask i ,... Availmask n}.
[0023] Step 1.4: Neuromorphic visual signal sequence ES Expand and get the associated signal tree tensor: Step 1.4.1. ES The neuromorphic visual signal corresponding to the -1 index is added to es -1 ={0,0,0,0}, thus obtaining the extended neuromorphic visual sequence ESAppend ={ es -1 , es 1, es 2,..., es n}; Step 1.4.2, initialize an empty list list tensor , from the associated signal tree group Tree i Start traversing from the root node and extract the data on the currently traversed node. c and the number on its parent node f , and find the node currently traversed in ESAppend The corresponding c Neuromorphic visual signals es c and f Neuromorphic visual signals es f , the relative position of the currently traversed node and its parent node { x c - x f , y c - y f , t c - t f , p c}Add to list tensor In the end, listtensor After splicing the relative positions in es i The associated signal tree tensor of TreeTensor i ;in, x c - x f express es c and es f The horizontal axis difference, y c - y f express y c - y f express es c and es f The vertical coordinate difference, t c - t f express es c and es f The time difference, p c express es c Polarity; Step 1.4.3: Follow the process of step 1.4.2 to obtain ES The associated signal tree tensor of each neuromorphic visual signal in is recorded as the total associated signal tree tensor EStreetensor ={ TreeTensor 1, TreeTensor 2,... TreeTensor i ,... TreeTensor n}.
[0024] Step 1.5: Convert the total associated signal tree tensor EStreetensor , Noise Label ES label Total valid bit mask Availmask As training data, used for network training.
[0025] Step 2: Construct a neural network, including: mask batch normalization layer MaskBatchNorm , convolution extraction unit Convlayer and multilayer perceptron MLP concat : Step 2.1, EstablishN The mask batch normalization layer of the layer is used to normalize the features of different layers: Among them, Layer MaskBatchNorm Using formula (1) and formula (2) Layer input feature Teninput With mask Tenmask Perform the operation and get Layer mean and Layer variance var : (1) (2) In formula (1) and formula (2), Represents the element product of the matrix, sum dim=2 It means to sum the second dimension of the matrix; Using formula (3), we can get Layer Mask Batch Normalization Layer Output normalized features MaskBatchNormOutput : (3) In formula (3), eps=1e -16 represents the parameter to prevent the denominator from being zero; Step 2.2: Establish an N-layer convolution extraction unit to fuse the features of different layers and output the normalized features of each layer, such as Figure 3 As shown: Among them, the convolution extraction unit Convlayer of the lth layer Using equations (4) and (5), we can get the input feature Tenfcinput of the lth layer, which is composed of the features of the parent node and its child nodes. Process it and get Feature Tensorf after layer aggregation : (4) (5) In formula (4) and formula (5), Tenmask Indicates Layer validity bit mask, MLPup Indicates Layer-wise multi-layer perceptron layer, MaskBatchNorm Indicates Layer mask code batch normalization layer, Maxpooling Indicates The maximum pooling layer of the layer, MLPeq Indicates Layers of equal-dimensional multilayer perceptrons, Indicates Residual connections of layers; Step 2.3: Use the N-layer convolutional extraction unit to extract the signal tree tensor TreeTensor i Perform aggregation to obtain es i The total root node tensor RootTly i ,like Figure 4 As shown; Step 2.3.1. The signal tree tensor of the layer is denoted as TreeT l,i , where the signal tree tensor at layer 0 is denoted as TreeT 0,i , and initialize TreeT 0,i =TreeTensor i ; Step 2.3.2: Use The convolutional extraction unit of the layer -1 layer signal tree tensor is aggregated to get the The signal tree tensor of the layer and save the root node tensor, such as Figure 4 As shown in the figure, after the feature aggregation of the signal tree tensor at the 0th layer, the signal tree tensor at the 1st layer is obtained. The dotted line on the signal tree tensor in the figure represents the path of information aggregation. The final signal tree tensor at the 1st layer is the parent node of the signal tree tensor at the 0th layer, but the content stored in the 1st layer is different from that in the 0th layer. The 1st layer stores the aggregated information: Traverse -1 layer of signal tree tensor on each parent-child node pair and the valid bit mask corresponding to the feature, and use Convlayer After processing one by one, the first Layer signal tree tensor TreeT l,i , and save the first Layer signal tree tensor TreeT l,i Tensor RootT of the root node l,i ; Step 2.3.2: After recursively processing the signal tree tensor using the N-layer convolution extraction units in accordance with the process of step 2.3.1, the root node tensor corresponding to the N-layer signal tree tensor is obtained, which is recorded as es i The total root node tensor RootTly i={RootT 1,i ,...RootT l,i ,...RootT N,i )},like Figure 4 As shown in the middle, the root node on the signal tree is saved separately. For the root node, the architecture can gradually enlarge the receptive field, as shown in the dotted frame at the bottom of the figure; Step 2.4, splicing es i After obtaining the root node tensor of N layers, we get es i ConcatRootTensor i , and input into the multi-layer perceptron MLP concat , and then use formula (6) to get es i The noise probability value pred i ,like Figure 4 As shown in the upper part, the features after feature aggregation are sent to the multi-layer perceptron through feature splicing. Sigmoid The function obtains the probability value of the final noise: (6) In formula (6), Sigmoid is the activation function; Step 3, based on es i The noise probability value and noise labels i Construct the i-th binary cross entropy loss function L BCEi , and use the Adam optimizer to train the neural network, calculate the binary cross entropy loss function to back-propagate to the network for reference optimization, and stop training when the binary cross entropy loss function converges to obtain the optimal denoising model for neuromorphic visual signals, and output the probability value of noise, and judge the signal with a probability value greater than 0.5 as noise and remove it.
[0026] In this embodiment, an information storage and reuse method of a neuromorphic visual signal denoising method based on tree-like information fusion includes the following steps: Step 4: Build an inference network to make convolution reusable: Step 4.1: Construct different layers of registers on a pixel-by-pixel basis and initialize registers: Step 4.1.1, construct different layers of registers on a pixel-by-pixel basis, and establish storage units for pixels at different positions on the neuromorphic vision camera array to store the features of each layer generated by signal calculations on different pixels, such as Figure 5 As shown, for the sake of simplicity, the polarity register is not drawn, and only one set of characteristic registers for each pixel is drawn: The positions of different pixels in the neural neuromorphic vision camera are encoded by the combination of the pixel horizontal and vertical coordinates. store max A register group consists of a time register, a polarity register and N feature memories. The N-layer feature memories are used to store the features extracted by the N-layer convolution extraction units. The first-layer feature memory layer1 register group, the second-layer feature memory layer2 register group, and so on. N The feature memory of the layer layerN register group; Step 4.1.2, register initialization: At the beginning, all register groups are initialized, the time register is set to -1, and the polarity register is set to 0. N The feature memory of the layer is initialized to all 0s; Every time a new signal es new Come and check all the pixels store max The time register of the group, the storage value of the time register is less than t new - w t Initialize the register group; Step 4.2, initialization of new signal: For a neuromorphic visual signal es new = { x new , y new , t new , p new},pass x new , y new Find the pixel at the corresponding position of the neuromorphic vision camera array, denoted as pixel new , pixel new Pixel store max The register group with the smallest time register storage value in the group register group is initialized, and the initialized register group is used to store es new Calculate the resulting features; Step 4.3: Find the correlation between the new signal and the past signal: Using time and space windows STW right es new Processing is performed to establish the association between past neuromorphic visual information and current signals, traversing the horizontal coordinate range greater than x new - wx , less than x new + wx, The vertical axis range is greater than y new - wy , less than y new + wy The time register of the feature register group corresponding to all pixels is set to the value stored in the time register greater than t new - wt The feature register group is marked as hit, indicating that the signal stored in the hit feature memory group is in the new signal es new In the spatiotemporal window, the signals stored in all hit feature register groups are used as es new Associated signals; if the number of hit signals within the spatiotemporal window is higher than relation max If relation max The signal is es new Associated signals, such as Figure 5 Shown on the left.
[0027] Step 4.4: Feature aggregation and storage of new signals: and es new Signal and es new The associated signal is feature aggregated using the trained N-layer convolutional extraction unit, using es new Signal and es new The horizontal and vertical coordinates of the associated signals and the signal tree tensor of the 0th layer of time composition TreeT 0,new , using the k-th convolutional extraction unit trained to obtain the optimal model Convlayer k Perform feature aggregation on the signal tree tensor of the k-1th layer and write the result of feature aggregation to storage es new In the k-th layer feature register group of the signal, esnew Signal and es new The values of the k-th layer feature register group of the associated signal constitute the signal tree tensor of the k-th layer TreeT k,new . Starting from k=1 and ending at N, write in sequence for storage es new The feature register group completes the feature aggregation, such as Figure 5 As shown in the middle, the arrows in the figure indicate the aggregation of features; Step 4.5, will be used for storage es new The feature aggregation results stored in the feature register group are concatenated to obtain es new Signal ConcatRootTensor new like Figure 5 As shown in the middle dashed box, the optimal model is trained using MLP concat Processing, through Sigmoid Function Get es new The noise probability pred new .
[0028] In this embodiment, an electronic device includes a memory and a processor, wherein the memory is used to store a program that supports the processor to execute the above method, and the processor is configured to execute the program stored in the memory.
[0029] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium, and the computer program executes the steps of the above method when executed by a processor.
[0030] During the specific implementation process, in order to verify the performance of the method proposed in the present invention, some classic traditional methods and deep learning methods were compared, and the accuracy was used as the evaluation indicator.
[0031] The higher the accuracy, the better. We tested the Alley, Benches, Bike, CheckerFast, CheckerSlow, Conference, Labfast, Labslow, Pavers, Soccer, Stairs, and Wall scenes under DVSNOISE20. DVSNOISE20 is a real dataset that obtains the neuromorphic visual signals generated in different scenes and labels the neuromorphic visual signals through auxiliary information such as optical flow and object contour edges. The final experimental results are shown in Table 1: Table 1. Experimental results of some scenes in the DVSNOISE20 dataset
[0032] In Table 1, the algorithm of the present invention generally exceeds the existing NN traditional denoising algorithm and the EdnCNN deep learning denoising algorithm in terms of indicators, indicating that the method of the present invention has good denoising performance.
[0033] The present invention can be applied to offline event signal denoising tasks, and the algorithm of the present invention can also be modified into hardware and deployed in scenarios requiring high-speed processing.
Claims
1. A neuromorphic visual signal denoising method based on tree information fusion, characterized in that: The steps include: Step 1, obtaining a neuromorphic visual signal sequence with a noise label and preprocessing it, including: defining relevant parameters and spatiotemporal windows of the signal, establishing signal association, obtaining a signal tree group, obtaining a tree tensor and processing relative positions, pairing the tree tensor group and the label and outputting them; Step 1.1, define the relevant parameters and time and space windows of the signal: Step 1.1.1, define the relevant parameters of the signal: The neuromorphic visual camera is used to obtain the time-arranged neuromorphic visual signal sequence ES={es1,es2,...,es i ,...,es n } and its corresponding noise label ES label = {l1,l2,...,l i ,...,l n }, where es i represents the i-th neuromorphic visual signal, and es i ={x i , y i , t i , p i }, x i fores i The pixel horizontal coordinate, y i fores i The pixel ordinate, t i fores i The timestamp, p i fores i The polarity of e i Indicates i The index in ES is recorded as E={e1,e2,...,e i ,...,e n }; l i Indicates i Noise label, if l i =1, means es i is a noise signal, if l i = 0, indicating es i is a normal signal; i=1,2,...,n, n is the total number of signals; Step 1.1.2, define the space-time window: Define the spatiotemporal window STW={wx, wy, wt}, where wx represents the window distance on the horizontal axis, wy represents the window distance on the vertical axis, and wt represents the window distance on the time axis; Step 1.2: Establish the single-hop association of the signal and the associated signal group ESRelation of the signal: Step 1.3: Establish multi-hop association of signals and obtain the associated signal tree group ES tree , define the relevant relationships on the tree and obtain the valid bit mask Availmask: Step 1.4, expand the neuromorphic visual signal sequence ES and obtain the associated signal tree tensor EStreetensor: Step 1.5: The total associated signal tree tensor EStreetensor and the noise label ES label and the total valid bit mask Availmask as training data; Step 2: Construct a neural network, including: MaskBatchNorm, Convlayer, and MLP concat : Step 2.1: Establish an N-layer masked batch normalization layer to normalize the features of different layers and output the aggregated features of each layer: Among them, Layer MaskBatchNorm Using formula (1) and formula (2) Layer input feature Teninput With mask Tenmask Perform the operation and get Layer mean and Layer variance var : (1) (2) In formula (1) and formula (2), Represents the element product of the matrix, sum dim=2 It means to sum the second dimension of the matrix; Using formula (3), we can get Layer Mask Batch Normalization Layer Output normalized features MaskBatchNormOutput : (3) In formula (3), eps represents the parameter to prevent the denominator from being zero; Step 2.2: Establish an N-layer convolution extraction unit to fuse the features of different layers and output the normalized features of each layer: Among them, the convolution extraction unit Convlayer of the lth layer Using equations (4) and (5), we can get the input feature Tenfcinput of the lth layer, which is composed of the features of the parent node and its child nodes. Process it and get Feature Tensorf after layer aggregation : (4) (5) In formula (4) and formula (5), Tenmask Indicates Layer validity bit mask, MLPup Indicates Layer-wise multi-layer perceptron layer, MaskBatchNorm Indicates Layer mask code batch normalization layer, Maxpooling Indicates The maximum pooling layer of the layer, MLPeq Indicates Layers of equal-dimensional multilayer perceptrons, Indicates Residual connections of layers; Step 2.3: Use the N-layer convolutional extraction unit to extract the signal tree tensor TreeTensor i Perform aggregation to obtain es i The total root node tensor RootTly i ; Step 2.4, splicing es i After obtaining the root node tensor of N layers, we get es i ConcatRootTensor i , and input into the multi-layer perceptron MLP concat , and then use formula (6) to get es i The noise probability value pred i : (6) In formula (6), Sigmoid is the activation function; Step 3, based on es i The noise probability value and noise labels i Construct the i-th binary cross entropy loss function L BCEi , and use the Adam optimizer to train the neural network, calculate the binary cross entropy loss function to back-propagate to the network for reference optimization, and stop training when the binary cross entropy loss function converges to obtain the optimal denoising model for neuromorphic visual signals, and output the probability value of noise, and judge the signal with a probability value greater than the threshold as noise and remove it.
2. According to the neuromorphic visual signal denoising method based on tree information fusion according to claim 1, it is characterized in that: The step 1.2 comprises the following steps: Step 1.2.1, extract es i The original associated signal group: If es i and the jth neuromorphic visual signal es j Satisfy ∣x j -x i ∣≤wx∩∣y j -y i ∣≤wy∩∣t j -t i ∣≤wt, then it means es j In es i In the space-time window, remember es j ∈STW(es i ), thus obtaining es i The original associated signal group ES0 i ={j | es j ∈ES, es j ∈STW(es i )}; Step 1.2.2: original associated signal group ES0 i To group: If the original associated signal group ES0 i If the pixel abscissa and pixel ordinate of the neuromorphic visual signals corresponding to the two indexes are the same, the indexes of the corresponding two neuromorphic visual signals are grouped together to obtain all index groups; Step 1.2.3: original associated signal group ES0 i The number of neuromorphic visual signals corresponding to each index group in is constrained: Define a hyperparameter store max , traverse ES0 i All index groups, if the number of indexes in the index group exceeds store max , then save the indexes corresponding to the two signals with the largest timestamps, thereby obtaining all the index groups after filtering, and concatenate all the index groups after filtering to form the associated signal group ES1 after the first round of filtering i ; Step 1.2.4: After the first round of filtering, the associated signal group ES1 i The total number of constraints: Define a hyperparameter relation max If ES1 i The number of indexes in exceeds the relation max , then randomly select ES1 i Select relation max indexes, and the remaining indexes are discarded. If ES1 i The number of indexes in relation is not reached max , then use the remaining bits Fill in the blanks to form the associated signal group ES2 after the second round of filtering i , and recorded as the associated signal group ES of the i-th signal i ; Step 1.2.5: Define the associated signal group ES with index -1 -1 : ES -1 Defined as a relation of length max A one-dimensional list of ES -1 All values in are -1; Step 1.2.6: Follow the process of steps 1.2.1 to 1.2.4 to obtain the associated signal group of each neuromorphic visual signal in ES, so as to obtain the total associated signal group ESRelation = {ES -1 ,ES1,...ES i ,...ES n }.
3. According to claim 2, a neuromorphic visual signal denoising method based on tree information fusion is characterized in that: The step 1.3 comprises the following steps: Step 1.3.1: for the i-th neuromorphic visual signal es i Define a hyperparameter N for the number of layers and create N empty lists layer1,...,layer k ,...,layer N , where layer k represents the kth empty list; Step 1.3.2, es i The associated signal group ES i Put it into layer1, and then traverse the layer from k=1 to k=layN-1 k Each number in the ESRelation is indexed and the associated signal group in the ESRelation corresponding to each index is put into the layer k+1 In order to generate es i Tree group of associated signals i =concat(i,layer1,...layer k ,...layer N ), where concat means concatenating arrays; Step 1.3.3, define the parent node of the root node i as itself, and define Tree i The parent node and its child node are a parent-child node pair; Step 1.3.4: Tree i All elements that are not -1 are set to , set all elements with a value of -1 to 0, and get es i The corresponding valid bit mask Availmask i ; Step 1.3.5: Obtain the associated signal tree group and its valid bit mask of each neuromorphic visual signal in ES according to the process of steps 1.3.1 to 1.3.4, so as to obtain the total associated signal tree group ES. tree ={Tree1,Tree2,...Tree i ,...Tree n } and its total effective bit mask Availmask={Availmask1,Availmask2,...Availmask i ,...Availmask n }.
4. According to claim 3, a neuromorphic visual signal denoising method based on tree information fusion is characterized in that: The step 1.4 comprises the following steps: Step 1.4.
1. Add the neuromorphic visual signal corresponding to the -1 index to ES, denoted as es -1 ={0,0,0,0}, thus obtaining the extended neuromorphic visual sequence ESAppend={es -1 ,es1,es2,...,es n }; Step 1.4.2, initialize the empty list list tensor , from the associated signal tree group Tree i Start traversing from the root node of , extract the number c on the currently traversed node and the number f on its parent node, and find the cth neuromorphic visual signal es corresponding to the currently traversed node in ESAppend c and the fth neuromorphic visual signal es f , the relative position of the currently traversed node and its parent node {x c -x f , y c -y f , t c -t f , p c }Add to list in sequence tensor Finally, list tensor After splicing the relative positions in es i The associated signal tree tensor TreeTensor i ; where x c -x f Indicates c With es f The horizontal coordinate difference, y c -y f Represents y c -y f Indicates c With es f The vertical coordinate difference, t c -t f Indicates c With es f The time difference, p c Indicates c Polarity; Step 1.4.3, follow the process of step 1.4.2 to obtain the associated signal tree tensor of each neuromorphic visual signal in ES, and record it as the total associated signal tree tensor EStreetensor={TreeTensor1,TreeTensor2,...TreeTensor i ,...TreeTensor n }.
5. According to claim 4, a neuromorphic visual signal denoising method based on tree information fusion is characterized in that: The step 2.3 comprises the following steps: Step 2.3.
1. The signal tree tensor of the layer is denoted as TreeT l,i , where the signal tree tensor at layer 0 is denoted as TreeT 0,i , and initialize TreeT 0,i =TreeTensor i ; Step 2.3.2: Use The convolutional extraction unit of the layer -1 layer signal tree tensor is aggregated to get the The signal tree tensor of the layer and save the root node tensor: Traverse -1 layer of signal tree tensor on each parent-child node pair and the valid bit mask corresponding to the feature, and use Convlayer After processing one by one, the first Layer signal tree tensor TreeT l,i , and save the first Layer signal tree tensor TreeT l,i Tensor RootT of the root node l,i ; Step 2.3.2: After recursively processing the signal tree tensor using the N-layer convolution extraction units in accordance with the process of step 2.3.1, the root node tensor corresponding to the N-layer signal tree tensor is obtained, which is recorded as es i The total root node tensor RootTly i ={RootT 1,i ,...RootT l,i ,...RootT N,i )}.
6. An information storage and reuse method based on the neuromorphic visual signal denoising method of tree-like information fusion according to claim 1, characterized in that: The steps include: Step 4: Build an inference network to achieve reusability of convolution: Step 4.1: Build registers of different layers pixel by pixel and initialize registers of each layer: Step 4.1.
1. Create registers of different layers for pixels at different positions on the neuromorphic vision camera array to store the features of each layer generated by signal calculations on different pixels: Set store for each pixel position separately max A register group, each register group is composed of a time register, a polarity register and an N-layer feature memory, wherein the feature memory of the k-th layer is used to store the features extracted by the corresponding k-th layer convolution extraction unit; k∈[1,N]; Step 4.1.2, register initialization: At the beginning, all register groups are initialized, where the time register is set to -1, the polarity register is set to 0, and the feature memories of the N layers are all initialized to all 0; Whenever a new neuromorphic visual signal es new = {x new , y new , t new , p new } comes, check the store of all pixels max The time register of the group, and the storage value of the time register is less than t new -w t Initialize the register group; where x new Indicates new The pixel horizontal coordinate, y new Indicates new The pixel ordinate, t new Indicates new The timestamp, p new Indicates new Polarity; Step 4.2, initialization of new signal: For a neuromorphic visual signal es new , through x new , y new Find the pixel corresponding to the position of the neuromorphic vision camera array, denoted as pixel new , for pixel new store max The register group corresponding to the smallest storage value of the time register in the group register group is initialized, and the initialized register group is used to store es new Calculate the resulting features; Step 4.3: Find the correlation between the new signal and the past signal: Using the spatiotemporal window STW to es new Processing is performed to establish the neuromorphic visual information stored on the neuromorphic visual camera array and es new The association of traversing the horizontal coordinate range of pixels at different positions on the neuromorphic vision camera array is greater than x new -wx and less than x new +wx, the vertical coordinate range is greater than y new -wy and less than y new +wy all pixels corresponding to the time register of the register group, and the storage value in the time register is greater than t new -wt's register group is marked as "hit", indicating that the signal stored in the corresponding memory group is in es new The space-time window STW is within the space-time window STW, so that the signals stored in all the register sets hit are taken as the same as es new associated signals; If the number of hit signals in the spatiotemporal window STW is higher than the relation max If there are , then randomly select relation max The signal is as new associated signals; Step 4.4: Feature aggregation and storage of new signals: Use the trained N-layer convolution extraction unit to extract es new As well as with es new The associated signals are feature aggregated, thus es new With es new The horizontal and vertical coordinates of the associated signals and the time form the signal tree tensor TreeT at the 0th level. 0,new ; Using the kth convolution extraction unit Convlayer of the optimal denoising model k For the signal tree tensor TreeT of the k-1th layer k-1,new Perform feature aggregation, obtain the result of the k-th layer feature aggregation and write it to store es new In the feature register of the kth layer; Will be by es new With es new The value of the feature register of the k-th layer of the associated signal constitutes the signal tree tensor TreeT of the k-th layer k,new , so that starting from k=1 and ending at N, the results of feature aggregation at each layer are written to store es new The feature register group completes the feature aggregation: Step 4.5, will be used to store es new After concatenating the feature aggregation results stored in all feature registers, es is obtained. new The concatenated root node tensor ConcatRootTensor new , and use the MLP of the optimal denoising model concat ConcatRootTensor new After processing, the Sigmoid function outputs es new The noise probability pred new .
7. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store a program that supports a processor to execute the method according to any one of claims 1 to 6, and the processor is configured to execute the program stored in the memory.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are performed.
Citation Information
Patent Citations
Seismic denoising method based on convolutional neural network and visual transformation neural network
CN115330643A
Transform-based raw video denoising method
CN116596779A
Neuromorphic camera denoising method for general illumination conditions
CN117132490A
Video denoising using neural networks with spatial and temporal features
US11151695B1
Model management apparatus and hierarchical system for neural network operation
WO2024046459A1
