Visual identification method and device based on pulse neural network

Abstract

The invention relates to the technical field of target recognition, and discloses a pulse neural network-based visual recognition method and device, and the method comprises the steps: carrying out the pulse coding of video sample data of a preset dynamic target, obtaining a pulse sequence, inputting the pulse sequence into a preset pulse neural network, carrying out the residual calculation, carrying out the long and short time sequence feature extraction of pulse network output features, and obtaining a pulse sequence; fusing the extracted long-short-term time sequence features with pulse network output features to obtain target fusion features, calculating loss values of the target fusion features, and performing reverse iteration updating on the pulse neural network to obtain a target long-short-term time sequence feature fusion model; and inputting video stream data of a to-be-identified dynamic target into the long and short term time sequence feature fusion model to obtain an identification result. According to the invention, through combination of long-term and short-term time sequence feature extraction, the defect that an existing pulse neural network cannot effectively utilize time sequence information to carry out dynamic data identification when capturing and identifying the dynamic data is overcome, and the visual identification accuracy of the dynamic data is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target recognition, and in particular to a visual recognition method and device based on a pulse neural network. Background Art

[0002] In existing spiking neural networks (SNNs), the network transmits pulse voltages between different time points, resulting in very limited transmission of timing information.

[0003] Since the spiking neural network mainly focuses on whether the pulse signal appears, and does not directly process continuous time information, the spiking neural network cannot effectively capture the temporal dynamic characteristics contained in the data. As a result, when processing time series information, the SNN may not be able to fully capture short-term burst characteristics and long-term change trends. Furthermore, due to the lack of sufficient time series information extraction mechanism, it leads to a slow response to dynamic changes and cannot effectively distinguish key dynamic patterns, which in turn affects the recognition effect. Summary of the invention

[0004] In view of this, the present invention provides a visual recognition method and device based on a pulse neural network to solve the problem of poor dynamic data recognition effect, thereby improving the accuracy of visual recognition of dynamic data.

[0005] In a first aspect, the present invention provides a visual recognition method based on a spiking neural network, the method comprising:

[0006] Pulse coding is performed on video sample data of a preset dynamic target to obtain a pulse sequence;

[0007] The pulse sequence is input into a preset pulse neural network for residual calculation to obtain a pulse network output feature, and the long-term and short-term time series features are extracted from the pulse network output feature, and at the same time, the extracted long-term and short-term time series features are fused with the pulse network output feature to obtain a target fusion feature;

[0008] Calculating the loss value of the target fusion feature using a preset loss function;

[0009] Perform reverse iterative update on the spiking neural network according to the loss value, and when the loss value is less than a preset loss threshold, stop the reverse iterative update process to obtain a target long-term and short-term time series feature fusion model;

[0010] The video stream data of the dynamic target to be identified is input into the target long-term and short-term temporal feature fusion model to obtain the identification result.

[0011] The embodiment of the present invention converts video sample data into a pulse sequence. Compared with the direct processing of pulse signals in the prior art, the pulse sequence obtained by pulse coding can improve the analysis effect of dynamic information. The pulse network output features are obtained by residual calculation, and long-term and short-term temporal features are extracted to obtain target fusion features. The target fusion features increase the feature expression in the time dimension, and solve the problem that the pulse neural network cannot effectively capture the temporal dynamic features contained in the data. The target fusion features in this embodiment combine the characteristics of long-term and short-term temporal series and the pulse signal itself. Therefore, the target fusion features are used to continuously update the loss value in the pulse network signal, thereby updating the pulse network and obtaining the target long-term and short-term temporal feature fusion model. The target long-term and short-term temporal feature fusion model finally generated has the ability to efficiently identify dynamic targets. When the video stream data of the identified dynamic target is input into the target long-term and short-term temporal feature fusion model, a more accurate recognition result can be obtained, which solves the problem of poor dynamic data recognition effect, thereby improving the accuracy of visual recognition of dynamic data.

[0012] In an optional embodiment, the preset pulse neural network includes at least two residual blocks and at least two long-term and short-term temporal feature extraction modules, and the number of the residual blocks is the same as that of the long-term and short-term temporal feature extraction modules and they are in a one-to-one correspondence.

[0013] The embodiment of the present invention uses multiple residual blocks and long- and short-term temporal feature extraction modules to form a preset pulse neural network in a one-to-one correspondence. Different residual blocks and temporal feature modules can realize the fusion of residual information and long- and short-term temporal features, so that the final target fusion features can more comprehensively represent the information of dynamic targets.

[0014] In an optional implementation, the pulse sequence is input into a preset pulse neural network for residual calculation to obtain a pulse network output feature, and the pulse network output feature is subjected to long-term and short-term time series feature extraction, and at the same time, the extracted long-term and short-term time series features are fused with the pulse network output feature to obtain a target fusion feature, specifically including:

[0015] Inputting the pulse sequence into a first residual block of the pulse neural network to obtain a first pulse output;

[0016] Inputting the first pulse output into the first long-short term time series feature extraction module to obtain the first long-short term time series feature;

[0017] The first long-term and short-term time series feature and the first pulse output are fused by an addition operation to obtain a first fused feature;

[0018] The first fusion feature and the pulse sequence are fused by addition operation to obtain the first residual output;

[0019] Starting from the first residual output, the target fusion feature is obtained by using the subsequent N-1 residual outputs according to the preset fusion process;

[0020] The preset fusion process includes:

[0021] Input the N-2th residual output to the N-1th residual block of the pulse neural network to obtain the N-1th pulse output; input the N-1th pulse output to the N-1th long-term and short-term time series feature extraction module to obtain the N-1th long-term and short-term time series feature; fuse the N-1th long-term and short-term time series feature with the N-1th pulse output through an addition operation to obtain the N-1th fused feature; fuse the N-1th fused feature and the N-2th residual output through an addition operation to obtain the N-1th residual output, wherein N is a preset value, representing an integer greater than 3;

[0022] The N-1th residual output is input to the Nth residual block of the pulse neural network to obtain the Nth pulse output; the Nth pulse output is input to the Nth long-term and short-term time series feature extraction module to obtain the Nth long-term and short-term time series feature; the Nth long-term and short-term time series feature is fused with the N-1th pulse output through an addition operation to obtain the Nth fused feature, wherein the Nth fused feature is the target fused feature.

[0023] The embodiment of the present invention fuses the output features of the pulse network with the long-term and short-term temporal features, and is equivalent to adding temporal feature extraction to each residual block, ensuring that features at different levels are strengthened by the long-term and short-term temporal features, thereby improving recognition accuracy. By combining the pulse neural network with the temporal feature extraction module, the model realizes learning of the temporal features of dynamic targets.

[0024] In an optional implementation, the calculating the loss value of the target fusion feature by using a preset loss function specifically includes:

[0025] The target fusion features are calculated by an exponential function to obtain the prediction probabilities of each category corresponding to the video sample data of the preset dynamic target;

[0026] The cross entropy loss function is used to calculate the difference information between the predicted probability and the true label corresponding to the video sample data of the preset dynamic target to determine the loss value.

[0027] The embodiment of the present invention uses an exponential function to calculate the predicted probability and uses a cross-entropy loss function to calculate the difference between the predicted probability and the true label, which is equivalent to continuously updating the model through target fusion features, thereby improving the dynamic target recognition capability and further improving the accuracy of visual recognition.

[0028] In an optional implementation, the long-term and short-term time series feature extraction module includes at least one convolution kernel, and the long-term and short-term time series feature extraction module uses a Sigmoid function to perform data nonlinear operations.

[0029] In the embodiment of the present invention, the long-term and short-term time series feature extraction module includes at least one convolution kernel, and the Sigmoid function is used to perform data nonlinear operations in the long-term and short-term time series feature extraction module. By combining the convolution kernel and the Sigmoid function, the long-term and short-term time series feature extraction module can more accurately fuse the time series information in the dynamic target, structurally enhance the feature expression effect in the time dimension, so that the generated target fusion feature can better reflect the data characteristics of the dynamic target, thereby improving the accuracy of dynamic target recognition.

[0030] In an optional implementation, the long-term and short-term time series feature extraction module further includes a fully connected layer, and the number of output neurons of the fully connected layer is the same as the number of each category of the preset sample data.

[0031] The long-term and short-term time series feature extraction module of the embodiment of the present invention includes a fully connected layer, the number of output neurons of this layer is the same as the number of each category of preset sample data. The fully connected layer can map the extracted long-term and short-term time series features to a higher dimensional space, making the features between different categories more obvious. Since the number of output neurons is the same as the number of categories, each neuron only needs to learn to recognize the features of a specific category, thereby improving the accuracy of identifying dynamic targets of different types.

[0032] In a second aspect, the present invention provides a visual recognition device based on a pulse neural network, the device comprising:

[0033] A sample encoding module, used for pulse encoding the video sample data of a preset dynamic target to obtain a pulse sequence;

[0034] A feature fusion module is used to input the pulse sequence into a preset pulse neural network for residual calculation to obtain the pulse network output feature, and to extract the long-term and short-term time series features of the pulse network output feature, and at the same time, to fuse the extracted long-term and short-term time series features with the pulse network output feature to obtain the target fusion feature;

[0035] A loss calculation module, used to calculate the loss value of the target fusion feature using a preset loss function;

[0036] An iterative training module, used for reverse iteratively updating the spiking neural network according to the loss value, and stopping the reverse iterative updating process when the loss value is less than a preset loss threshold, to obtain a target long-term and short-term time series feature fusion model;

[0037] The model recognition module is used to input the video stream data of the dynamic target to be recognized into the target long-term and short-term temporal feature fusion model to obtain the recognition result.

[0038] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the visual recognition method based on pulse neural network of the above-mentioned first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the visual recognition method based on a pulse neural network according to the first aspect or any corresponding embodiment thereof.

[0040] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the visual recognition method based on a pulse neural network according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 is a flow chart of a visual recognition method based on a pulse neural network according to an embodiment of the present invention;

[0043] Figure 2 is a schematic diagram of short-term temporal feature extraction of a visual recognition method based on a pulse neural network according to an embodiment of the present invention;

[0044] Figure 3 is a schematic diagram of long-term temporal feature extraction of a visual recognition method based on a pulse neural network according to an embodiment of the present invention;

[0045] Figure 4 is a schematic diagram of the structure of a target long-term and short-term temporal feature fusion model of a visual recognition method based on a pulse neural network according to an embodiment of the present invention;

[0046] Figure 5 is a flow chart of another visual recognition method based on a pulse neural network according to an embodiment of the present invention;

[0047] Figure 6is a flow chart of another visual recognition method based on a pulse neural network according to an embodiment of the present invention;

[0048] Figure 7 is a structural block diagram of a visual recognition device based on a pulse neural network according to an embodiment of the present invention;

[0049] Figure 8 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0051] In existing spiking neural networks (SNNs), the network transmits pulse voltages between different time points, resulting in very limited transmission of timing information.

[0052] Spiking neural networks (SNNs) cannot effectively identify dynamic data, mainly because spiking neural networks rely on the "presence" or "absence" of pulse signals, rather than identifying the time information of the appearance and disappearance of the signal.

[0053] Therefore, when information changes suddenly in a short period of time, the spiking neural network will ignore short-term dynamic changes. Since it cannot accurately grasp the changes in time, the SNN will inevitably find it difficult to identify key dynamic patterns, thus affecting the final recognition effect.

[0054] The embodiments of the present invention provide a visual recognition method based on a pulse neural network, which is applied to visual recognition scenarios based on a pulse neural network, such as video stream data that needs to identify short-term dynamic change characteristics. By combining long-term and short-term temporal feature extraction, more accurate dynamic pattern recognition is achieved, thereby improving the model's recognition ability and accuracy for dynamic video streams, and overcoming the defect that existing pulse neural networks cannot effectively use temporal information for accurate dynamic data recognition when capturing and identifying dynamic data.

[0055] According to an embodiment of the present invention, an embodiment of a visual recognition method based on a pulse neural network is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0056] In this embodiment, a visual recognition method based on a pulse neural network is provided, which can be used for the above-mentioned computer. Figure 1 is a flow chart of a visual recognition method based on a pulse neural network according to an embodiment of the present invention, such as Figure 1 As shown, the process includes the following steps:

[0057] Step S101 , pulse encoding is performed on video sample data of a preset dynamic target to obtain a pulse sequence.

[0058] It should be noted that pulse coding refers to converting continuous video data into discrete pulse sequences. The purpose of this embodiment is to use a coding mechanism to convert dynamic video sample data into a series of pulse sequences, so that the pulse neural network can imitate the processing method of the biological nervous system to process the input dynamic video stream data information. Specifically, the pulse coding can be Poisson coding or time difference coding.

[0059] The video sample data of the preset dynamic target can be an RGB image or a dynamic data stream in DVS format. Specifically, one is a static RGB image, such as Cifar10 and ImageNet datasets; the other is a dynamic event stream dataset, such as Cifar10-DVS and DVS-Gesture datasets, which are converted from corresponding static datasets. In this embodiment, these two types of video sample data are different from traditional video data and can be obtained from real-time dynamic videos or from preset datasets and databases.

[0060] Step S102, input the pulse sequence into a preset pulse neural network for residual calculation to obtain the pulse network output feature, and extract the long-term and short-term time series features of the pulse network output feature. At the same time, the extracted long-term and short-term time series features are fused with the pulse network output feature to obtain the target fusion feature.

[0061] It should be noted that the spiking neural network is a highly biologically simulated neural network model that can simulate the human brain's ability to process visual information. Residual calculation refers to the method of using residual calculation to reduce the gradient vanishing problem, which can improve the model training effect.

[0062] Specifically, the process of extracting long-term and short-term time series features is implemented by a long-term and short-term time series feature extraction module. The pulse sequence is input into the pulse neural network and processed through multiple residual blocks to obtain the pulse network output features that already contain time series information. However, these time series information cannot be processed in the original pulse neural network. In this embodiment, the long-term and short-term time series feature extraction module is used to further extract the long-term and short-term information in the time series information, and the extracted long-term and short-term time series features are fused with the pulse network output features. This fusion represents the combination of the short-term and long-term dynamic change characteristics of the input data. The long-term and short-term time series feature extraction module includes a short-term time series feature extraction module and a long-term time series feature extraction module.

[0063] For example, Figure 2 As shown in the figure, in the short-term temporal feature extraction, the module first uses the shift operation to fuse the pulse features of adjacent time points, which can capture the dynamic changes between adjacent time points, and uses the convolution operation to extract short-term temporal features from these fused features. The extracted short-term temporal features are fused with the original pulse features by multiplication to finally generate new pulse features. Among them, Feature Fusion refers to the process of combining features at different time points, C / 2channels shift left refers to shifting the channels of the feature map and moving the first half of the channels of the feature map to the left, Temporal conV is a convolution operation on the fused features to extract short-term temporal features, truncate refers to removing unnecessary parts during feature processing, and Multiplication is to fuse the extracted short-term temporal features with the original pulse features by multiplication.

[0064] For example, Figure 3As shown in the figure, in the long-term time series feature extraction, the module performs a mean operation on the pulse feature in the channel dimension and fuses the features of all time points together. The information of all time points can be comprehensively considered to extract the long-term time series features. The extracted long-term time series features are fused with the new pulse features through broadcast multiplication to obtain the final pulse features. The use of broadcast multiplication makes the fusion process more flexible and can effectively combine the long-term time series information with the current features. Among them, Feature Fusion represents the process of combining features at different time points, Mean represents the mean operation, which is used to reduce the dimension of time series data, Temporal conv represents temporal convolution, which captures the local features of data in the time dimension, and Broadcast multiplication represents that broadcast multiplication fuses the extracted long-term time series features with the current pulse features. Broadcast multiplication can automatically expand the features of smaller dimensions to align them with the features of larger dimensions, thereby achieving effective feature combination and better integrating long-term time series information with current features.

[0065] Step S103, using a preset loss function to calculate the loss value of the target fusion feature.

[0066] Specifically, a preset loss function is used to calculate the loss value of the target fusion feature. The choice of loss function can be mean square error, cross entropy calculation, etc., depending on the specific model training task. The target fusion feature in this embodiment combines long-term and short-term temporal features, which can improve the model's ability and accuracy in identifying dynamic video stream data.

[0067] Step S104, reverse iteratively updating the pulse neural network according to the loss value, when the loss value is less than a preset loss threshold, stopping the reverse iterative updating process to obtain a target long-term and short-term time series feature fusion model.

[0068] Specifically, according to the calculated loss value, the back propagation algorithm is used to update the weights of the pulse neural network. When the loss value is lower than the preset loss threshold, the training stops, indicating that the model has converged and can perform the task of video stream recognition.

[0069] Step S105, inputting the video stream data of the dynamic target to be identified into the target long-term and short-term temporal feature fusion model to obtain the identification result.

[0070] Specifically, the trained long-term and short-term temporal feature fusion model is used to identify the input video stream data. Since the target long-term and short-term temporal feature fusion model trained in the present embodiment has learned the temporal features of the video stream, it can identify dynamic targets more accurately.

[0071] For example, refer to Figure 4, when a dog-related dynamic video stream is input, the video stream is first converted into multiple frames as input. In the specific target video stream data recognition task, the input multiple frames are first subjected to preliminary feature extraction through the Conv1 layer, and then enter different stages in the MultiStep Blocks (i.e., stage1 to stage4), and each stage is further subjected to feature fusion processing. Among them, the Conv layer and the SN layer are used in combination to extract long and short time series features in the video stream image, and enhance the information by simulating the pulse behavior of neurons. The enhanced feature map data stream will pass through the next layer of the network, and the LM and GM layers will be applied to reduce the dimension of the feature map and retain the key recognition information. At the same time, through the pooling operation (i.e., FC and Mean), the amount of feature map data can be further simplified and more critical features can be highlighted. The final output probability distribution is generated. The output result is to determine which category the target object in the dynamic video data belongs to. For example, "Class is DandieDinmont" means that the model predicts that the dog in the input video stream data is "DandieDinmont".

[0072] The embodiment of the present invention converts video sample data into a pulse sequence. Compared with the direct processing of pulse signals in the prior art, the pulse sequence obtained by pulse coding can improve the analysis effect of dynamic information. The pulse network output features are obtained by residual calculation, and long-term and short-term temporal features are extracted to obtain target fusion features. The target fusion features increase the feature expression in the time dimension, and solve the problem that the pulse neural network cannot effectively capture the temporal dynamic features contained in the data. The target fusion features in this embodiment combine the characteristics of long-term and short-term temporal series and the pulse signal itself. Therefore, the target fusion features are used to continuously update the loss value in the pulse network signal, thereby updating the pulse network and obtaining the target long-term and short-term temporal feature fusion model. The target long-term and short-term temporal feature fusion model finally generated has the ability to efficiently identify dynamic targets. When the video stream data of the identified dynamic target is input into the target long-term and short-term temporal feature fusion model, a more accurate recognition result can be obtained, which solves the problem of poor dynamic data recognition effect, thereby improving the accuracy of visual recognition of dynamic data.

[0073] In this embodiment, a visual recognition method based on a pulse neural network is provided, which can be used for the above-mentioned computer. Figure 5 is a flow chart of a visual recognition method based on a pulse neural network according to an embodiment of the present invention, such as Figure 5 As shown, the process includes the following steps:

[0074] Step S501: Pulse coding is performed on the video sample data of the preset dynamic target to obtain a pulse sequence. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0075] Step S502, input the pulse sequence into a preset pulse neural network for residual calculation to obtain the pulse network output feature, and extract the long-term and short-term time series features of the pulse network output feature. At the same time, the extracted long-term and short-term time series features are fused with the pulse network output feature to obtain the target fusion feature.

[0076] In an optional embodiment, the preset pulse neural network includes at least two residual blocks and at least two long-term and short-term temporal feature extraction modules, and the number of the residual blocks is the same as that of the long-term and short-term temporal feature extraction modules and they are in a one-to-one correspondence.

[0077] The embodiment of the present invention uses multiple residual blocks and long- and short-term temporal feature extraction modules to form a preset pulse neural network in a one-to-one correspondence. Different residual blocks and temporal feature modules can realize the fusion of residual information and long- and short-term temporal features, so that the final target fusion features can more comprehensively represent the information of dynamic targets.

[0078] Specifically, the above step S502 includes:

[0079] Step S5021, input the pulse sequence into the first residual block of the pulse neural network to obtain the first pulse output.

[0080] Step S5022: input the first pulse output to the first long-short term time series feature extraction module to obtain the first long-short term time series feature.

[0081] Step S5023, the first long-term and short-term time series feature and the first pulse output are fused by an addition operation to obtain a first fused feature.

[0082] Step S5024, the first fusion feature and the pulse sequence are fused by an addition operation to obtain a first residual output.

[0083] Step S5025, starting from the first residual output, using the subsequent N-1 residual outputs according to the preset fusion process to obtain the target fusion feature;

[0084] The preset fusion process includes:

[0085] Input the N-2th residual output to the N-1th residual block of the pulse neural network to obtain the N-1th pulse output; input the N-1th pulse output to the N-1th long-term and short-term time series feature extraction module to obtain the N-1th long-term and short-term time series feature; fuse the N-1th long-term and short-term time series feature with the N-1th pulse output through an addition operation to obtain the N-1th fused feature; fuse the N-1th fused feature and the N-2th residual output through an addition operation to obtain the N-1th residual output, wherein N is a preset value, representing an integer greater than 3;

[0086] The N-1th residual output is input to the Nth residual block of the pulse neural network to obtain the Nth pulse output; the Nth pulse output is input to the Nth long-term and short-term time series feature extraction module to obtain the Nth long-term and short-term time series feature; the Nth long-term and short-term time series feature is fused with the N-1th pulse output through an addition operation to obtain the Nth fused feature, wherein the Nth fused feature is the target fused feature.

[0087] Exemplarily, each residual block corresponds to a long-term and short-term temporal feature extraction module. If there are three residual blocks and three long-term and short-term temporal feature extraction modules, residual block A1 corresponds to long-term and short-term temporal feature extraction module B1, residual block A2 corresponds to long-term and short-term temporal feature extraction module B2, and residual block A3 corresponds to long-term and short-term temporal feature extraction module B3. The following steps are required to obtain the target fusion feature:

[0088] (1) The pulse sequence is input into the residual block A1 to obtain the first pulse output; the first pulse output is input into the long-term and short-term time series feature extraction module B1 to obtain the first long-term and short-term time series feature; these two features are fused into the first fused feature, and then fused with the pulse sequence to obtain the first residual output.

[0089] (2) The first residual output is used as input and passes through the residual block A2 to obtain a second pulse output; the second pulse output is input into the long-term and short-term time series feature extraction module B2 to obtain a second long-term and short-term time series feature; these two features are fused into a second fused feature, and the second fused feature is fused with the first residual output to obtain a second residual output.

[0090] (3) The second residual output is used as input and passes through residual block A3 to obtain a third pulse output; the third pulse output is input into the long-term and short-term time series feature extraction module B3 to obtain the third long-term and short-term time series feature; these two features are fused into the final target fusion feature.

[0091] In addition, during the training of the long-term and short-term time series feature fusion model, the training parameters include: learning rate, sample batch, and maximum number of training times. In addition, the stochastic gradient descent method is used as the optimizer of the long-term and short-term time series feature fusion model.

[0092] Furthermore, the long-term and short-term time series feature extraction module includes at least one convolution kernel, and the long-term and short-term time series feature extraction module uses a Sigmoid function to perform data nonlinear operations.

[0093] It should be noted that for data on the timeline, the convolution kernel operation can efficiently extract the correlation features between consecutive frames, helping the model understand the dynamic changes of training data in the short term. After the convolution operation, the features are nonlinearly transformed through the Sigmoid function, which can amplify the significant features in the dynamically changing time signal, thereby accurately extracting the change information in the long and short term, and improving the model's recognition accuracy of dynamic targets.

[0094] In the embodiment of the present invention, the long-term and short-term time series feature extraction module includes at least one convolution kernel, and the Sigmoid function is used to perform data nonlinear operations in the long-term and short-term time series feature extraction module. By combining the convolution kernel and the Sigmoid function, the long-term and short-term time series feature extraction module can more accurately fuse the time series information in the dynamic target, structurally enhance the feature expression effect in the time dimension, so that the generated target fusion feature can better reflect the data characteristics of the dynamic target, thereby improving the accuracy of dynamic target recognition.

[0095] Furthermore, the long-term and short-term time series feature extraction module also includes a fully connected layer, and the number of output neurons of the fully connected layer is the same as the number of each category of the preset sample data.

[0096] The long-term and short-term time series feature extraction module of the embodiment of the present invention includes a fully connected layer, the number of output neurons of this layer is the same as the number of each category of preset sample data. The fully connected layer can map the extracted long-term and short-term time series features to a higher dimensional space, making the features between different categories more obvious. Since the number of output neurons is the same as the number of categories, each neuron only needs to learn to recognize the features of a specific category, thereby improving the accuracy of identifying dynamic targets of different types.

[0097] Step S503, using a preset loss function to calculate the loss value of the target fusion feature. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0098] Step S504, reverse iteratively update the spiking neural network according to the loss value, and when the loss value is less than a preset loss threshold, stop the reverse iterative update process to obtain the target long-term and short-term time series feature fusion model. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0099] Step S505: Input the video stream data of the dynamic target to be identified into the target long-term and short-term temporal feature fusion model to obtain the identification result. Figure 1 Step S105 of the illustrated embodiment will not be described in detail here.

[0100] The embodiment of the present invention fuses the output features of the pulse network with the long-term and short-term temporal features, and is equivalent to adding temporal feature extraction to each residual block, ensuring that features at different levels are strengthened by the long-term and short-term temporal features, thereby improving recognition accuracy. By combining the pulse neural network with the temporal feature extraction module, the model realizes learning of the temporal features of dynamic targets.

[0101] In this embodiment, a visual recognition method based on a pulse neural network is provided, which can be used for the above-mentioned computer. Figure 6 is a flow chart of a visual recognition method based on a pulse neural network according to an embodiment of the present invention, such as Figure 6 As shown, the process includes the following steps:

[0102] Step S601: Pulse coding is performed on the video sample data of the preset dynamic target to obtain a pulse sequence. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0103] Step S602: Input the pulse sequence into a preset pulse neural network for residual calculation to obtain the pulse network output feature, and extract the long-term and short-term time series features from the pulse network output feature. At the same time, the extracted long-term and short-term time series features are fused with the pulse network output feature to obtain the target fusion feature. For details, please refer to Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0104] Step S603, using a preset loss function to calculate the loss value of the target fusion feature. Specifically, step S603 includes:

[0105] Step S6031, calculating the target fusion feature through an exponential function to obtain the prediction probability of each category corresponding to the video sample data of the preset dynamic target.

[0106] Step S6032, using the cross entropy loss function to calculate the difference information between the predicted probability and the real label corresponding to the video sample data of the preset dynamic target, and determine the loss value.

[0107] It should be noted that the exponential function can be a Softmax function, which is used as the activation function of the output layer of the multi-classification model to convert the original score of the network output into a normalized probability distribution. The loss function in this embodiment is a cross entropy loss function. The principle of cross entropy calculation loss function is that if the probability predicted by the model is close to the true category, the loss value will be smaller, and if the gap is larger, the loss value will be larger. Therefore, by minimizing the cross entropy loss, the model can continuously adjust parameters and gradually improve the accuracy of the prediction.

[0108] Step S604, reverse iteratively update the spiking neural network according to the loss value, and when the loss value is less than a preset loss threshold, stop the reverse iterative update process to obtain the target long-term and short-term time series feature fusion model. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0109] Step S605: Input the video stream data of the dynamic target to be identified into the target long-term and short-term temporal feature fusion model to obtain the identification result. Figure 1 Step S105 of the illustrated embodiment will not be described in detail here.

[0110] The long-term and short-term time series feature extraction module of the embodiment of the present invention includes a fully connected layer, the number of output neurons of this layer is the same as the number of each category of preset sample data. The fully connected layer can map the extracted long-term and short-term time series features to a higher dimensional space, making the features between different categories more obvious. Since the number of output neurons is the same as the number of categories, each neuron only needs to learn to recognize the features of a specific category, thereby improving the accuracy of identifying dynamic targets of different types.

[0111] In the present embodiment, a visual recognition device based on a pulse neural network is also provided, and the system is used to implement the above-mentioned embodiments and preferred embodiments, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0112] This embodiment provides a visual recognition device based on a pulse neural network, such as Figure 7 As shown, including:

[0113] The sample encoding module 701 is used to perform pulse encoding on the video sample data of the preset dynamic target to obtain a pulse sequence;

[0114] The feature fusion module 702 is used to input the pulse sequence into a preset pulse neural network for residual calculation to obtain the pulse network output feature, and to extract the long-term and short-term time series features of the pulse network output feature, and at the same time, to fuse the extracted long-term and short-term time series features with the pulse network output feature to obtain the target fusion feature;

[0115] The loss calculation module 703 is used to calculate the loss value of the target fusion feature using a preset loss function;

[0116] An iterative training module 704 is used to perform reverse iterative update on the spiking neural network according to the loss value, and when the loss value is less than a preset loss threshold, stop the reverse iterative update process to obtain a target long-term and short-term time series feature fusion model;

[0117] The model recognition module 705 is used to input the video stream data of the dynamic target to be recognized into the target long-term and short-term temporal feature fusion model to obtain the recognition result.

[0118] In an optional embodiment, the preset pulse neural network includes at least two residual blocks and at least two long-term and short-term temporal feature extraction modules, and the number of the residual blocks is the same as that of the long-term and short-term temporal feature extraction modules and they are in a one-to-one correspondence.

[0119] In some optional implementations, the feature fusion module 702 includes:

[0120] A pulse input unit, used for inputting the pulse sequence into a first residual block of the pulse neural network to obtain a first pulse output;

[0121] A time series feature extraction unit, used for inputting the first pulse output into the first long-short term time series feature extraction module to obtain a first long-short term time series feature;

[0122] A fusion feature unit, used for fusing the first long-term and short-term time series feature and the first pulse output through an addition operation to obtain a first fusion feature;

[0123] The residual fusion unit is used to obtain the target fusion feature by using the subsequent N-1 residual outputs according to the preset fusion process starting from the first residual output;

[0124] The preset fusion process includes:

[0125] Input the N-2th residual output to the N-1th residual block of the pulse neural network to obtain the N-1th pulse output; input the N-1th pulse output to the N-1th long-term and short-term time series feature extraction module to obtain the N-1th long-term and short-term time series feature; fuse the N-1th long-term and short-term time series feature with the N-1th pulse output through an addition operation to obtain the N-1th fused feature; fuse the N-1th fused feature and the N-2th residual output through an addition operation to obtain the N-1th residual output, wherein N is a preset value, representing an integer greater than 3;

[0126] The N-1th residual output is input to the Nth residual block of the pulse neural network to obtain the Nth pulse output; the Nth pulse output is input to the Nth long-term and short-term time series feature extraction module to obtain the Nth long-term and short-term time series feature; the Nth long-term and short-term time series feature is fused with the N-1th pulse output through an addition operation to obtain the Nth fused feature, wherein the Nth fused feature is the target fused feature.

[0127] In an optional implementation, the loss calculation module 703 specifically includes:

[0128] A prediction probability calculation unit, used to calculate the target fusion feature through an exponential function to obtain the prediction probability of each category corresponding to the video sample data of the preset dynamic target;

[0129] The loss value calculation unit is used to calculate the difference information between the predicted probability and the real label corresponding to the video sample data of the preset dynamic target by using the cross entropy loss function to determine the loss value.

[0130] In an optional implementation, the long-term and short-term time series feature extraction module includes at least one convolution kernel, and the long-term and short-term time series feature extraction module uses a Sigmoid function to perform data nonlinear operations.

[0131] In an optional implementation, the long-term and short-term time series feature extraction module further includes a fully connected layer, and the number of output neurons of the fully connected layer is the same as the number of each category of the preset sample data.

[0132] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0133] The visual recognition device based on pulse neural network in this embodiment is presented in the form of functional units, where the units refer to ASIC (Application Specific Integrated Circuit) circuits, processors and memories that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0134] The embodiment of the present invention also provides a computer device having the above Figure 7 The visual recognition device based on the pulse neural network is shown.

[0135] See also Figure 8 , Figure 8 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 8 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0136] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0137] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0138] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0139] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0140] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0141] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0142] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0143] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the defined scope.

Claims

1. A visual recognition method based on a pulse neural network, characterized in that: The method comprises: Pulse coding is performed on video sample data of a preset dynamic target to obtain a pulse sequence; Input the pulse sequence into a preset pulse neural network for residual calculation to obtain the pulse network output feature; extract long-term and short-term time series features from the pulse network output feature, and fuse the extracted long-term and short-term time series features with the pulse network output feature to obtain the target fusion feature; Calculating the loss value of the target fusion feature using a preset loss function; Perform reverse iterative update on the spiking neural network according to the loss value, and when the loss value is less than a preset loss threshold, stop the reverse iterative update process to obtain a target long-term and short-term time series feature fusion model; The video stream data of the dynamic target to be identified is input into the target long-term and short-term temporal feature fusion model to obtain the identification result.

2. The method according to claim 1, characterized in that The preset pulse neural network includes at least two residual blocks and at least two long-term and short-term temporal feature extraction modules. The number of the residual blocks is the same as that of the long-term and short-term temporal feature extraction modules and they are in a one-to-one correspondence.

3. The method according to claim 2, characterized in that The step of inputting the pulse sequence into a preset pulse neural network for residual calculation to obtain a pulse network output feature, extracting long-term and short-term time series features from the pulse network output feature, and fusing the extracted long-term and short-term time series features with the pulse network output feature to obtain a target fusion feature specifically includes: Inputting the pulse sequence into a first residual block of the pulse neural network to obtain a first pulse output; Inputting the first pulse output into the first long-short term time series feature extraction module to obtain the first long-short term time series feature; The first long-term and short-term time series feature and the first pulse output are fused by an addition operation to obtain a first fused feature; The first fusion feature and the pulse sequence are fused by addition operation to obtain the first residual output; Starting from the first residual output, the target fusion feature is obtained by using the subsequent N-1 residual outputs according to the preset fusion process; The preset fusion process includes: Input the N-2th residual output to the N-1th residual block of the pulse neural network to obtain the N-1th pulse output; input the N-1th pulse output to the N-1th long-term and short-term time series feature extraction module to obtain the N-1th long-term and short-term time series feature; fuse the N-1th long-term and short-term time series feature with the N-1th pulse output through an addition operation to obtain the N-1th fused feature; fuse the N-1th fused feature and the N-2th residual output through an addition operation to obtain the N-1th residual output, wherein N is a preset value, representing an integer greater than 3; The N-1th residual output is input to the Nth residual block of the pulse neural network to obtain the Nth pulse output; the Nth pulse output is input to the Nth long-term and short-term time series feature extraction module to obtain the Nth long-term and short-term time series feature; the Nth long-term and short-term time series feature is fused with the N-1th pulse output through an addition operation to obtain the Nth fused feature, wherein the Nth fused feature is the target fused feature.

4. The method according to claim 3, characterized in that: The using of a preset loss function to calculate the loss value of the target fusion feature specifically includes: The target fusion features are calculated by an exponential function to obtain the prediction probabilities of each category corresponding to the video sample data of the preset dynamic target; The cross entropy loss function is used to calculate the difference information between the predicted probability and the true label corresponding to the video sample data of the preset dynamic target to determine the loss value.

5. The method according to any one of claims 2 to 4, characterized in that: The long-term and short-term time series feature extraction module includes at least one convolution kernel, and the long-term and short-term time series feature extraction module uses a Sigmoid function to perform data nonlinear operations.

6. The method according to claim 5, characterized in that The long-term and short-term time series feature extraction module also includes a fully connected layer, and the number of output neurons of the fully connected layer is the same as the number of each category of the preset sample data.

7. A visual recognition device based on a pulse neural network, characterized in that: The device comprises: A sample encoding module, used for pulse encoding the video sample data of a preset dynamic target to obtain a pulse sequence; A feature fusion module is used to input the pulse sequence into a preset pulse neural network for residual calculation to obtain the pulse network output feature, and to extract the long-term and short-term time series features of the pulse network output feature, and at the same time, to fuse the extracted long-term and short-term time series features with the pulse network output feature to obtain the target fusion feature; A loss calculation module, used to calculate the loss value of the target fusion feature using a preset loss function; An iterative training module, used for reverse iteratively updating the spiking neural network according to the loss value, and stopping the reverse iterative updating process when the loss value is less than a preset loss threshold, to obtain a target long-term and short-term time series feature fusion model; The model recognition module is used to input the video stream data of the dynamic target to be recognized into the target long-term and short-term temporal feature fusion model to obtain the recognition result.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the visual recognition method based on a pulse neural network according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the visual recognition method based on a pulse neural network according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the visual recognition method based on a pulse neural network according to any one of claims 1 to 6.

Images