Method for processing fusion information based on inertia-dynamic vision-single photon sensor

Abstract

The invention provides a method for processing fusion information based on an inertia-dynamic vision-single photon sensor, relates to the technical field of visual information processing, and aims to solve the technical problem that an existing on-chip training algorithm is difficult to meet actual requirements due to overlarge calculated amount and serious resource consumption. The method comprises the following steps: acquiring moving posture information of a target, dynamic information of the target in a specified area scene and static detail information of the target in the specified area scene; inputting the acquired information into a pulse neural network for identification processing to obtain a classification result; wherein the training process of the spiking neural network comprises a first stage and a second stage; the first stage is used for performing ternary weight processing on the spiking neural network and quantizing and reducing weight data of the spiking neural network; and the second stage is used for carrying out gradient simplification processing on the spiking neural network obtained through training in the first stage through a lightweight single-layer space-time back propagation algorithm, and optimizing the calculation efficiency of the spiking neural network.

Description

Technical Field

[0001] The present invention relates to the field of visual information processing technology, and more specifically, to a processing method based on inertial-dynamic vision-single-photon sensor fusion information. Background Art

[0002] Faced with various visual tasks in complex scenarios, the device needs to accurately and comprehensively perceive the surrounding scene, while also obtaining the real-time status of the device itself in the scene. This requires a variety of different sensors suitable for the scene to acquire different information, and subsequently realize the computational processing of multiple visual or non-visual information.

[0003] Dynamic Vision Sensors (DVS) and Single Photon Avalanche Diodes (SPAD) perform well in high-speed dynamic scenes. The former can output dynamic information in the scene in the form of event pulses, while the latter can output pulse frames of the scene at an extremely high frame rate. Both perform well in the data acquisition stage.

[0004] However, in the subsequent visual or non-visual information computing and processing stage, the existing information processing system is limited by hardware resources. The on-chip training algorithms designed by them are difficult to meet actual needs due to excessive computational complexity and serious resource consumption. Summary of the Invention

[0005] In view of this, the present invention provides a processing method based on inertial-dynamic vision-single-photon sensor fusion information, aiming to solve the technical problem that the existing on-chip training algorithm is difficult to meet actual needs due to excessive computational complexity and serious resource consumption.

[0006] One aspect of the present invention provides a processing method based on inertial-dynamic vision-single-photon sensor fusion information, including: obtaining the motion posture information of the target, the dynamic information of the target in a specified area scene, and the static detail information of the target in the specified area scene; inputting the motion posture information, dynamic information, and static detail information into a pulse neural network for recognition processing to obtain a classification result; wherein, the training process of the pulse neural network includes a first stage and a second stage; the first stage is used to perform ternary weight processing on the pulse neural network and quantize and reduce the weight data of the pulse neural network; the second stage is used to perform gradient simplification processing on the pulse neural network obtained by training in the first stage through a lightweight single-layer spatiotemporal backpropagation algorithm to optimize the computational efficiency of the pulse neural network.

[0007] According to an embodiment of the present invention, obtaining the target's motion posture information, the target's dynamic information in a specified area scene, and the target's static detail information in a specified area scene includes: using an inertial sensor to obtain the target's motion posture information based on speed information; using a dynamic vision sensor to obtain the target's dynamic information in a specified area scene; and using a single-photon sensor to obtain the target's static detail information in a specified area scene.

[0008] According to an embodiment of the present invention, the ternary pulse neural network includes: a multi-layer ternary weighted convolution layer for extracting spatiotemporal features and local features from motion posture information, dynamic information and static detail information; a multi-layer fully connected layer for fusing spatiotemporal features with local features into a global representation; and a single-layer weighted fully connected layer for making decisions on the global representation and outputting the classification results after the decision.

[0009] According to an embodiment of the present invention, the output layer corresponding to each layer of the ternary pulse neural network is configured to be activated by a LIF pulse neuron; wherein, each LIF pulse neuron is configured with a corresponding membrane potential, the membrane potential is accumulated by the input current, and decays at a preset ratio over time; in response to the membrane potential reaching the activation threshold, the LIF pulse neuron emits a pulse.

[0010] According to an embodiment of the present invention, the weighted fully connected layer is configured as an 8-bit weighted fully connected layer.

[0011] According to an embodiment of the present invention, gradient simplification processing of the pulse neural network obtained after the first stage of training is performed by a lightweight single-layer spatiotemporal back propagation algorithm, including: inputting pulse coding to the pulse neural network after ternary weighting; using multiple layers of ternary weighted convolutional layers and multiple layers of fully connected layers to extract features of the pulse coding, and recording the changes in the membrane potential of each LIF pulse neuron; in response to the membrane potential reaching the activation threshold, the LIF pulse neuron emits a pulse, and the membrane potential of the LIF pulse neuron is updated to the membrane potential minus the activation threshold; the output of the single-layer weighted fully connected layer is compared with the expected output pulse, and the loss is calculated; the lightweight single-layer spatiotemporal back propagation algorithm is used to back propagate the error and update the real weight, wherein, in the process of back propagating the error, the LIF pulse neuron performs error calculation by approximate gradient; the real weight is re-projected back to the pulse neural network after ternary weighting to continue forward propagation, and iterative training is repeated until convergence.

[0012] According to an embodiment of the present invention, the propagation path of the backpropagation error using the lightweight single-layer spatiotemporal backpropagation algorithm includes: backpropagating the error through the spatial path, wherein the spatial path characterizes the structural direction of the ternary pulse neural network; backpropagating the error through the time path, wherein the time path characterizes the historical state of the LIF pulse neuron.

[0013] Another aspect of the present invention provides a processing device based on inertial-dynamic vision-single-photon sensor fusion information, including: an acquisition module for acquiring the motion posture information of the target, the dynamic information of the target in a specified area scene, and the static detail information of the target in the specified area scene; an identification module for inputting the motion posture information, dynamic information, and static detail information into a pulse neural network for identification processing to obtain a classification result; wherein, the training process of the pulse neural network includes a first stage and a second stage; the first stage is used to perform ternary weight processing on the pulse neural network and quantize and reduce the weight data of the pulse neural network; the second stage is used to perform gradient simplification processing on the pulse neural network obtained through the first stage training through a lightweight single-layer spatiotemporal backpropagation algorithm to optimize the computational efficiency of the pulse neural network.

[0014] Another aspect of the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above method.

[0015] Another aspect of the present invention provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the above method when executed.

[0016] Another aspect of the present invention provides a computer program product, which includes computer executable instructions. When the instructions are executed, they are used to implement the above method.

[0017] Compared with the prior art, the processing method based on inertial-dynamic vision-single-photon sensor fusion information provided by the present invention has at least the following beneficial effects:

[0018] (1) The processing method based on inertial-dynamic vision-single photon sensor fusion information provided by the embodiment of the present invention uses the improved spatio-temporal backpropagation (STBP) algorithm, that is, the lightweight single-layer spatio-temporal backpropagation algorithm to perform on-chip training on the ternary spiking neural network, and then deploys it in the recognition module to perform computational processing on the visual information. Since the ternary spiking neural network itself can greatly reduce the computational overhead and storage through the ternary weights, combined with the efficient training of the lightweight single-layer spatio-temporal backpropagation algorithm, high-efficiency and low-power information processing is achieved, thereby solving the technical problems of excessive computation and serious resource consumption.

[0019] (2) The processing method based on inertial-dynamic vision-single photon sensor fusion information provided by the embodiment of the present invention is more suitable for logic-level mapping operations because the outputs of DVS and SPAD are pulse data, which are more consistent with the convolution of three-value weights. Therefore, it can reduce power consumption and improve computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which:

[0021] Figure 1 A flowchart of a method for processing information based on inertial-dynamic vision-single-photon sensor fusion according to an embodiment of the present invention is schematically shown;

[0022] Figure 2 A block diagram schematically illustrates a structure of a device for processing information based on inertial-dynamic vision-single-photon sensor fusion according to an embodiment of the present invention;

[0023] Figure 3 The structure block diagram of an electronic device suitable for implementing a method for processing inertial-dynamic vision-single photon sensor fusion information according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0024] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present invention. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of embodiments of the present invention. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concept of the present invention.

[0025] The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. The terms "comprise", "include", etc. used herein indicate the presence of the features, steps, operations and / or components, but do not exclude the presence or addition of one or more other features, steps, operations or components.

[0026] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0027] When expressions such as "at least one of A, B, and C, etc." are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0028] In the embodiments of the present invention, the collection, updating, analysis, processing, use, transmission, provision, disclosure, and storage of all data involved (including, but not limited to, user personal information) comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. In particular, necessary measures are taken to prevent unauthorized access to user personal information data and maintain the security of user personal information and network security.

[0029] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0030] Figure 1 The flowchart of the method for processing inertial-dynamic vision-single photon sensor fusion information according to an embodiment of the present invention is schematically shown.

[0031] like Figure 1 As shown, the dynamic target tracking and recognition method of dynamic vision-single photon sensor fusion in this embodiment may include operations S1 to S2.

[0032] In operation S1 , motion posture information of a target, dynamic information of the target in a designated area scene, and static detail information of the target in a designated area scene are acquired.

[0033] In operation S2, the motion posture information, dynamic information and static detail information are input into the spiking neural network for recognition processing to obtain a classification result.

[0034] The training process of the pulse neural network includes a first stage and a second stage.

[0035] The first stage is used to perform ternary weight processing on the spiking neural network and quantize and reduce the weight data of the spiking neural network.

[0036] The second stage is used to perform gradient simplification processing on the spiking neural network trained in the first stage through a lightweight single-layer spatiotemporal backpropagation algorithm to optimize the computational efficiency of the spiking neural network.

[0037] In this embodiment, taking the visual recognition task as an example, after obtaining relevant information of the target, the ternary spiking neural network (S-TWN) obtained through the first stage training is selected as the overall architecture to perform recognition processing on it to obtain a classification result.

[0038] Ternaryization is a quantization method that quantizes the weight data in a neural network into 1, -1, and 0. It converts the 32-bit floating-point weight data in the network into data that only requires 2 bits of storage, greatly reducing the storage space required for the weight data and related computing overhead.

[0039] Furthermore, due to hardware resource constraints, the designed on-chip training algorithm needs to minimize computational complexity. Compared to traditional artificial neural network (ANN) backpropagation training, the spatio-temporal backpropagation (STBP) algorithm focuses on both spatial and temporal propagation information, making it a more effective training algorithm for spiking neural networks (SNNs), which incorporate temporal information.

[0040] Based on the STBP algorithm, this embodiment proposes an improved lightweight single-layer STBP algorithm to optimize the ternary pulse neural network obtained by the first stage training. Through reasonable improvement, the complex derivative calculation of the original algorithm is optimized into a simple pulse counting operation. The ternary pulse neural network is trained using the improved lightweight single-layer STBP algorithm, and then deployed in the recognition module to perform computational processing on visual information. Since the ternary pulse neural network itself can greatly reduce computational overhead and storage through ternary weights, combined with the efficient training of the lightweight single-layer spatiotemporal backpropagation algorithm, high-efficiency and low-power information processing is achieved, thereby solving the technical problems of excessive computational complexity and serious resource consumption.

[0041] According to an embodiment of the present invention, operation S1 acquires the motion posture information of the target, the dynamic information of the target in the specified area scene, and the static detail information of the target in the specified area scene, which may specifically include:

[0042] The inertial sensor is used to obtain the target's motion posture information based on the speed information.

[0043] Use dynamic vision sensors to obtain dynamic information of the target in a specified area scene.

[0044] Single-photon sensors are used to obtain static detail information of the target in a specified area scene.

[0045] In this embodiment, for example, an inertial sensor, a dynamic vision sensor, and a single photon sensor may be combined to collect the required visual information, and then the information may be fused.

[0046] The processing method based on inertial-dynamic vision-single-photon sensor fusion information provided by the embodiment of the present invention is naturally suitable for logic-level mapping operations because the outputs of DVS and SPAD are both pulse data, which are more consistent with the convolution of three-value weights. Therefore, it can reduce power consumption and improve computing efficiency.

[0047] According to an embodiment of the present invention, the structure of the pulse neural network may specifically include: multi-layer ternary weight convolution layers, multi-layer fully connected layers and a single-layer weighted fully connected layer.

[0048] Among them, multi-layer ternary weighted convolutional layers are used to extract spatiotemporal features and local features in motion posture information, dynamic information and static detail information.

[0049] Multi-layer fully connected layers are used to fuse spatiotemporal features with local features into a global representation.

[0050] The single-layer weighted fully connected layer is used to make decisions on the global representation and output the classification results after the decision. The single-layer weighted fully connected layer is configured as an 8-bit weighted fully connected layer.

[0051] The output layer corresponding to each layer of the ternary spiking neural network is configured to be activated by LIF spiking neurons.

[0052] The Leaky Integrate-and-Fire (LIF) model is a spiking neural network model that simulates the working process of biological neurons.

[0053] Among them, each LIF pulse neuron is configured with a corresponding membrane potential. The membrane potential is accumulated through the input current and decays at a preset ratio over time. When the membrane potential reaches the activation threshold, the LIF pulse neuron is activated, emits a pulse and resets the membrane potential.

[0054] The weight quantization method of the ternary pulse neural network is as follows:

[0055] In order to make the performance of the ternary spiking neural network as good as possible, the first training phase can be further optimized. This embodiment minimizes the error between the floating-point full-precision weights and the ternary weights by scaling factor α, that is:

[0056]

[0057] Among them, I represents the network layer input, W represents the floating point full precision weight, Represents the quantized ternary weight.

[0058] Then the optimization problem can be expressed as:

[0059]

[0060]

[0061] Before further derivation, this implementation introduces the parameter Δ as the quantization standard of the weight W, namely:

[0062]

[0063]

[0064] Combining formulas (2), (4), and (5), formula (3) can be transformed into:

[0065]

[0066] in, , Indicates the number of elements in it.

[0067] In formula (6), is a constant that is independent of α. Then, for a given Δ value, the above optimization problem has an optimal value of α:

[0068]

[0069] Further substituting (7) into (6), we can obtain an optimization problem only about Δ:

[0070]

[0071] For the optimization problem above, there is no analytical solution at this time, so we can assume the distribution of weight W to give an approximate solution. If we assume that W is uniformly distributed in [-α, α], then the approximate optimal value of Δ is about α / 3, that is When W conforms to the normal distribution , then the approximate optimal value of Δ is about 0.6σ, that is:

[0072]

[0073] The training method of the ternary pulse neural network is as follows:

[0074] In this embodiment, there are two issues that need to be considered in network training:

[0075] First, a quantization-aware training strategy for the network. Although the quantization methods discussed above are as optimal as possible, directly ternary quantizing a trained network can still result in significant errors. Therefore, quantization-aware training is necessary. This involves modeling the effects of quantization during training to achieve higher accuracy.

[0076] Second, in spiking neural networks, the mathematical model of the activation behavior of spike neurons is a step function, which is not differentiable. In gradient descent-based training, appropriate methods are needed to circumvent this problem.

[0077] In this embodiment, for the first point, Quantization Aware Training (QAT) is adopted.

[0078] The training strategy used by the algorithm is based on binary neural networks and related work on XNOR-Net. In this quantization strategy, ternary quantization is only performed during the forward and backward propagation of the network convolutional layer. However, this embodiment still retains a copy of the floating-point full-precision weight data for weight updates. The specific process is as follows:

[0079] (1) Quantize the floating-point 32-bit weight W into a ternary weight using a quantization method , calculate the corresponding Δ and α values.

[0080] (2) As the weights are forward propagated, the error function is calculated.

[0081] (3) Use Perform standard backpropagation gradient descent calculations.

[0082] (4) Update the floating-point full-precision weight W using the gradient descent calculation results in (3).

[0083] (5) Repeat steps (1)(2)(3)(4).

[0084] In this embodiment, for the second point, surrogate gradient learning is adopted.

[0085] The essence of the gradient substitution method is to replace the ungradient step function of the spiking neuron with another gradient function. During the forward propagation process, the behavior of the neuron remains consistent, as follows:

[0086]

[0087] Obviously, this function cannot be used to find the gradient, so this embodiment uses other functions to fit this process, for example, the Fastsigmoid function is used as follows:

[0088]

[0089] The gradient of formula (11) is used for calculation during the back propagation process:

[0090]

[0091] Although the ternary pulse neural network trained by the above method performs well enough, the amount of computation can still be reduced through further optimization.

[0092] For example, the improved lightweight single-layer STBP algorithm is used to further optimize the training process of the ternary spiking neural network.

[0093] According to an embodiment of the present invention, performing gradient simplification processing on the spiking neural network obtained through the first stage training by using a lightweight single-layer spatiotemporal back propagation algorithm may specifically include:

[0094] (1) Input pulse encoding to the spiking neural network after ternary weighting.

[0095] (2) Multi-layer ternary weighted convolutional layers and multi-layer fully connected layers are used to extract features of pulse coding and record the changes in the neuronal membrane potential of each LIF pulse.

[0096] (3) In response to the membrane potential reaching the activation threshold, the LIF-spiking neuron fires a pulse, and the membrane potential of the LIF-spiking neuron is updated to the membrane potential minus the activation threshold.

[0097] The activation process of the LIF spiking neuron is shown in formula (10). In the classic model of existing spiking neurons, the membrane potential of the neuron is reset to 0 after activation, which is expressed as follows:

[0098]

[0099] However, such a model has significant limitations. First, setting the membrane potential to zero results in the loss of some input information during the forward propagation process. For example, when a neuron's membrane potential approaches its activation threshold, the neuron will activate regardless of whether a small or large current is input, but the difference in current magnitude cannot be distinguished. Second, the calculation of the gradient in formula (13) involves complex operations.

[0100] To this end, this embodiment proposes to improve the reset behavior of neurons from zeroing to subtracting the activation threshold voltage, as follows:

[0101]

[0102]

[0103] Formula (15) is more concise than the gradient of Formula (13) and is equally effective.

[0104] (4) Compare the output of the single-layer weighted fully connected layer with the expected output pulse and calculate the loss.

[0105] (5) The error is back-propagated using a lightweight single-layer spatiotemporal back-propagation algorithm to update the real weights. During the back-propagation process, the LIF spike neurons perform error calculations by approximating the gradient.

[0106] Although the alternative gradient algorithm has been mentioned above, the formula (12) used can be used for training on GPU, but for on-chip training, the computational hardware overhead involved is too large.

[0107] The lightweight on-chip training algorithm is based on the following assumptions:

[0108]

[0109] Although this assumption is not However, experience shows that formula (16) is still effective for BP (backpropagation) training of SNN.

[0110] In this embodiment, the last layer of the algorithm can be set to 10 LIF neuron outputs, corresponding to the 10 classification items of the classification task. For a single output channel (neuron), its error function is:

[0111]

[0112] Where S represents the actual output pulse, and y represents the correct number of pulse outputs. For the correct classification category, the number of pulses should be equal to the total time steps, and for the incorrect classification category, the number of pulses should be 0.

[0113] For formula (17), find the output pulse S T The gradient of , we get:

[0114]

[0115] Here, we need to get the gradient of the error function L with respect to the weight W and bias b:

[0116]

[0117]

[0118] Among them, Ut represents the membrane potential at time t, W represents the weight vector, and It represents the output pulse at time t. From formulas (19) and (20), we can see that we only need to find The value of , the gradient descent result can be calculated.

[0119]

[0120] Substituting formula (15) into formula (21), we get:

[0121]

[0122] Substituting the alternative gradient formula (16) into this formula, we get a concise form:

[0123]

[0124] in, The actual number of pulse outputs and the ideal number of pulses are given by formula (18). In this way, we only need to know when the neuron was activated to easily calculate the gradient used for learning without too much complicated calculation.

[0125] (6) The spiking neural network, after reprojecting the real weights back to the ternary weights, continues forward propagation and repeats iterative training until convergence.

[0126] Finally, the real weights are re-projected back to the ternary pulse neural network to continue forward propagation, and the iterative training is repeated until the algorithm converges and then the training is stopped.

[0127] In this embodiment, lightweight integer on-chip training can be implemented through hardware, specifically as follows:

[0128] (1) According to the actual output pulse number and the ideal pulse number, the 10 neurons can be obtained from formula (18): The result is a 1x10 vector, where each value in the vector is the actual number of pulses fired by each neuron minus the ideal number of pulses. Based on simulations, this value typically does not exceed ±3. To be on the safe side, this data can be stored using a 4-bit two's complement, resulting in a total of 10x4 bits of data.

[0129] (2) Further find out at which moments the neuron was activated. Multiply the input at the activation moment (512*1 pulse sequence) by the result in (1) and accumulate the results of all time steps to obtain 512*1 vectors, a total of 10. Concatenate them to obtain a 512*10 matrix, which contains the Δw information. The data is stored using 8-bit two's complement, a total of 512*10*8 bits of data.

[0130] (3) Then multiply the results in (1) by the number of pulse emissions to obtain a 1*10 vector, which contains the Δb information. For uniformity, the data is still represented using 8-bit two's complement.

[0131] (4) Subtract Δw*lr from the original weight matrix, where lr is the learning rate, to obtain the updated weights. The specific value of the learning rate depends on the size of the input batch for a single training run. In theory, a larger batch size and a correspondingly smaller lr result in better training results.

[0132] lr can be a power of 0.5. This is implemented in hardware using a bit shift. This shifts the result of Δw to the right by the corresponding number of bits (tentatively 8 or higher). Δw*lr is represented as 16-bit data, retaining decimal places. The number of decimal places depends on the number of bits shifted in this step. The weight change is obtained by summing the different Δw*lr values ​​obtained for different inputs in a batch and discarding the decimal places.

[0133] (5) The update of bias b is similar and will not be repeated here.

[0134] According to an embodiment of the present invention, the propagation path of the back propagation error using the lightweight single-layer spatiotemporal back propagation algorithm may specifically include:

[0135] The error is back-propagated through the spatial path, where the spatial path represents the structural direction of the ternary spiking neural network.

[0136] The error is back-propagated through the temporal path, where the temporal path represents the historical state of the LIF spiking neuron.

[0137] Figure 2 The structure block diagram of the device for processing information based on inertial-dynamic vision-single photon sensor fusion according to an embodiment of the present invention is schematically shown.

[0138] like Figure 2 As shown, the structure of the processing device based on inertial-dynamic vision-single photon sensor fusion information of this embodiment may include, for example: an acquisition module 210 and a recognition module 220.

[0139] The acquisition module 210 is used to acquire the target's motion posture information, the target's dynamic information in a designated area scene, and the target's static detail information in a designated area scene.

[0140] The recognition module 220 is used to input the motion posture information, dynamic information and static detail information into the pulse neural network for recognition processing to obtain a classification result.

[0141] Among them, the training process of the pulse neural network includes the first stage and the second stage; the first stage is used to perform ternary weight processing on the pulse neural network and quantize and reduce the weight data of the pulse neural network; the second stage is used to perform gradient simplification processing on the pulse neural network obtained through the first stage training through a lightweight single-layer spatiotemporal backpropagation algorithm to optimize the computational efficiency of the pulse neural network.

[0142] Figure 3 The structure block diagram of an electronic device suitable for implementing a method for processing inertial-dynamic vision-single photon sensor fusion information according to an embodiment of the present invention is schematically shown. Figure 3 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0143] like Figure 3 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 301, which can perform various appropriate actions and processes based on a program stored in a read-only memory (ROM) 302 or a program loaded from a storage unit 308 into a random access memory (RAM) 303. The processor 301 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 301 may also include onboard memory for caching purposes. The processor 301 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0144] The storage unit 308 stores various programs and data required for the operation of the electronic device 300. The processor 301, ROM 302, and storage unit 308 are interconnected via a bus 304. The processor 301 executes the programs stored in the ROM 302 and / or storage unit 308 to perform the various operations of the method flow according to the embodiment of the present invention. It should be noted that the programs may also be stored in one or more memories other than the ROM 302 and storage unit 308. The processor 301 may also execute the programs stored in the one or more memories to perform the various operations of the method flow according to the embodiment of the present invention.

[0145] According to an embodiment of the present invention, electronic device 300 may further include an input / output (I / O) interface 305, which is also connected to bus 304. Electronic device 300 may also include one or more of the following components connected to I / O interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including devices such as a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section 308 including a hard disk; and a communication section 309 including a network interface card such as a LAN card or modem. Communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to I / O interface 305 as needed. Removable media 311, such as a magnetic disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed in drive 310 as needed, so that computer programs read from the removable media can be installed into storage section 308 as needed.

[0146] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 309, and / or installed from the removable medium 311. When the computer program is executed by the processor 301, the above-mentioned functions defined in the system of the embodiment of the present invention are executed. According to an embodiment of the present invention, the system, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0147] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0148] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium. Examples include, but are not limited to, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0149] For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 302 and / or the storage portion 308 described above and / or one or more memories other than the ROM 302 and the storage portion 308 .

[0150] An embodiment of the present invention also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present invention. When the computer program product runs on an electronic device, the program code is used to enable the electronic device to implement the method provided by the embodiment of the present invention.

[0151] When the computer program is executed by the processor 301, the above functions defined in the system / device of the embodiment of the present invention are performed. According to the embodiment of the present invention, the above-described systems, devices, modules, units, etc. can be implemented by computer program modules.

[0152] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 309, and / or installed from a removable medium 311. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0153] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0154] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of the systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram may represent a module, program segment, or portion of code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the boxes may occur in an order different from that marked in the accompanying drawings. For example, two boxes shown in succession may actually be executed substantially in parallel, or they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, as well as the combination of boxes in the block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or may be implemented using a combination of dedicated hardware and computer instructions. It will be understood by those skilled in the art that the features described in the various embodiments of the present invention may be combined and / or coupled in various ways, even if such combinations or couplings are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention may be combined and / or coupled in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or combinations fall within the scope of the present invention.

[0155] The above describes embodiments of the present invention. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present invention. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be advantageously used in combination. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for processing information based on inertial-dynamic vision-single photon sensor fusion, characterized in that: The method comprises: Obtain the target's motion posture information, the target's dynamic information in the specified area scene, and the target's static detail information in the specified area scene; Inputting the motion posture information, the dynamic information and the static detail information into a spiking neural network for recognition processing to obtain a classification result; The training process of the pulse neural network includes a first stage and a second stage; The first stage is used to perform ternary weight processing on the pulse neural network, and quantize and reduce the weight data of the pulse neural network; The second stage is used to perform gradient simplification processing on the pulse neural network trained in the first stage through a lightweight single-layer spatiotemporal back propagation algorithm to optimize the computational efficiency of the pulse neural network.

2. The method according to claim 1, characterized in that The acquisition of the target's motion posture information, the target's dynamic information in the specified area scene, and the target's static detail information in the specified area scene includes: Use inertial sensors to obtain the target's motion posture information based on speed information; Use dynamic visual sensors to obtain dynamic information of the target in a specified area scene; Single-photon sensors are used to obtain static detail information of the target in a specified area scene.

3. The method according to claim 1, characterized in that The spiking neural network comprises: Multi-layer ternary weighted convolutional layers, used to extract spatiotemporal features and local features from the motion posture information, the dynamic information, and the static detail information; Multi-layer fully connected layers, for fusing the spatiotemporal features with the local features into a global representation; The single-layer weighted fully connected layer is used to make a decision on the global representation and output the classification result after the decision.

4. The method according to claim 3, characterized in that The output layer corresponding to each layer of the spiking neural network is configured to be activated by LIF spiking neurons; Each LIF spiking neuron is configured with a corresponding membrane potential, which is accumulated by the input current and decays at a preset ratio over time; In response to the membrane potential reaching an activation threshold, the LIF-spiking neuron fires a pulse.

5. The method according to claim 3, characterized in that The weighted fully connected layer is configured as an 8-bit weighted fully connected layer.

6. The method according to claim 3, characterized in that The gradient simplification processing of the spiking neural network obtained through the first stage training by using the lightweight single-layer spatiotemporal back propagation algorithm includes: Input pulse encoding to the spiking neural network after ternary weighting; Using the multi-layer ternary weighted convolutional layer and the multi-layer fully connected layer to extract features of the pulse code, and recording the change of the neuron membrane potential of each LIF pulse; In response to the membrane potential reaching an activation threshold, the LIF-pulsing neuron emits a pulse, and the membrane potential of the LIF-pulsing neuron is updated to a membrane potential minus the activation threshold; Compare the output of the single-layer weighted fully connected layer with the expected output pulse and calculate the loss; Backpropagating the error using a lightweight single-layer spatiotemporal backpropagation algorithm to update the real weights, wherein during the backpropagation of the error, the LIF spiking neuron performs error calculation by approximating the gradient; The spiking neural network after reprojecting the real weights back to the ternary weights continues forward propagation and repeats iterative training until convergence.

7. The method according to claim 6, characterized in that The propagation path of the back propagation error using the lightweight single-layer spatiotemporal back propagation algorithm includes: Backpropagating an error through a spatial path, wherein the spatial path represents a structural direction of the spiking neural network; The error is back-propagated through a temporal path, wherein the temporal path represents the historical state of the LIF spiking neuron.

8. A processing device based on inertial-dynamic vision-single photon sensor fusion information, characterized in that: The device comprises: An acquisition module is used to acquire the target's motion posture information, the target's dynamic information in a specified area scene, and the target's static detail information in a specified area scene; A recognition module, configured to input the motion posture information, the dynamic information, and the static detail information into a spiking neural network for recognition processing to obtain a classification result; The training process of the pulse neural network includes a first stage and a second stage; The first stage is used to perform ternary weight processing on the pulse neural network, and quantize and reduce the weight data of the pulse neural network; The second stage is used to perform gradient simplification processing on the pulse neural network trained in the first stage through a lightweight single-layer spatiotemporal back propagation algorithm to optimize the computational efficiency of the pulse neural network.

9. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors are enabled to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, characterized in that: When the instructions are executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7.

Images