Visual recognition method and device based on pulse neural network

Abstract

The application relates to the technical field of target identification, and discloses a visual identification method and device based on a pulse neural network, which comprises the following steps: pulse coding of video sample data of a preset dynamic target to obtain a pulse sequence, residual calculation of the preset pulse neural network, long-short-term time sequence feature extraction of pulse network output features, fusion of the extracted long-short-term time sequence features and the pulse network output features to obtain target fusion features, loss value calculation of the target fusion features, reverse iteration update of the pulse neural network, and obtaining of a target long-short-term time sequence feature fusion model; and input of video stream data of a dynamic target to be identified into the long-short-term time sequence feature fusion model to obtain an identification result. The application overcomes the defect that the existing pulse neural network cannot effectively utilize time sequence information for dynamic data identification when capturing and identifying dynamic data by combining long-short-term time sequence feature extraction, and improves the visual identification accuracy for dynamic data.

Description

Technical Field

[0001] This invention relates to the field of target recognition technology, and more specifically to a visual recognition method and apparatus based on spiking neural networks. Background Technology

[0002] In existing spiking neural networks (SNNs), the network transmits pulse voltages between different time points, resulting in very limited transmission of temporal information.

[0003] Since spiking neural networks (SNNs) primarily focus on whether pulse signals occur rather than directly processing continuous time information, they cannot effectively capture the temporal dynamics inherent in the data. As a result, when processing time-series information, SNNs may not be able to fully capture short-term burst features and long-term trends. Furthermore, due to the lack of sufficient time-series information extraction mechanisms, they are slow to react to dynamic changes and cannot effectively distinguish key dynamic patterns, thus affecting the recognition performance. Summary of the Invention

[0004] In view of this, the present invention provides a visual recognition method and apparatus based on spiking neural networks to solve the problem of poor recognition effect of dynamic data, thereby improving the accuracy of visual recognition of dynamic data.

[0005] In a first aspect, the present invention provides a visual recognition method based on a spiking neural network, the method comprising:

[0006] Pulse coding is performed on video sample data of a preset dynamic target to obtain a pulse sequence;

[0007] The pulse sequence is input into a preset spiking neural network for residual calculation to obtain the output features of the spiking network. Long and short-term time series features are extracted from the output features of the spiking network. At the same time, the extracted long and short-term time series features are fused with the output features of the spiking network to obtain the target fused features.

[0008] The loss value of the target fusion feature is calculated using a preset loss function;

[0009] The spiking neural network is iteratively updated in reverse based on the loss value. When the loss value is less than a preset loss threshold, the inverse iterative update process is stopped to obtain the target long-term and short-term time series feature fusion model.

[0010] The video stream data of the dynamic target to be identified is input into the target's long-term and short-term temporal feature fusion model to obtain the identification result.

[0011] This invention converts video sample data into pulse sequences. Compared to directly processing pulse signals in existing technologies, pulse sequences obtained through pulse coding can improve the resolution of dynamic information. The output features of the pulse network are obtained through residual calculation, and long and short-term temporal features are extracted to obtain target fusion features. These target fusion features enhance the temporal dimension of feature representation, solving the problem that pulse neural networks cannot effectively capture the temporal dynamic features contained in the data. In this embodiment, the target fusion features combine long and short-term temporal features with the characteristics of the pulse signal itself. Therefore, by continuously updating the loss value in the pulse network signal using the target fusion features, the pulse network is updated, resulting in a target long and short-term temporal feature fusion model. The final target long and short-term temporal feature fusion model has the ability to efficiently identify dynamic targets. When the video stream data of the identified dynamic target is input into the target long and short-term temporal feature fusion model, a more accurate identification result can be obtained, solving the problem of poor dynamic data recognition and thus improving the accuracy of visual recognition of dynamic data.

[0012] In one optional implementation, the preset spiking neural network includes at least two residual blocks and at least two long short-term time series feature extraction modules, wherein the number of residual blocks and long short-term time series feature extraction modules are the same and they are in a one-to-one correspondence.

[0013] In this embodiment of the invention, multiple residual blocks and long and short-term time-series feature extraction modules are used to form a preset spiking neural network in a one-to-one correspondence. Different residual blocks and time-series feature modules can realize the fusion of residual information and long and short-term time-series features, so that the final target fusion feature can more comprehensively represent the information of the dynamic target.

[0014] In one optional implementation, the step of inputting the pulse sequence into a preset spiking neural network for residual calculation to obtain the output features of the spiking network, and extracting long and short-term time-series features from the output features of the spiking network, and simultaneously fusing the extracted long and short-term time-series features with the output features of the spiking network to obtain the target fused features, specifically includes:

[0015] The pulse sequence is input into the first residual block of the spiking neural network to obtain the first pulse output.

[0016] The first pulse output is input to the first long-short-term time series feature extraction module to obtain the first long-short-term time series feature.

[0017] The first fused feature is obtained by fusing the first long-short time series feature and the first pulse output through an addition operation;

[0018] The first fusion feature and the pulse sequence are fused together using an addition operation to obtain the first residual output;

[0019] Starting from the first residual output, the target fusion feature is obtained by using the subsequent N-1 residual outputs according to the preset fusion process;

[0020] The preset fusion process includes:

[0021] The (N-2)th residual output is input to the (N-1)th residual block of the spiking neural network to obtain the (N-1)th pulse output; the (N-1)th pulse output is input to the (N-1)th long short-term time series feature extraction module to obtain the (N-1)th long short-term time series feature; the (N-1)th long short-term time series feature and the (N-1)th pulse output are fused by addition to obtain the (N-1)th fused feature; the (N-1)th fused feature and the (N-2)th residual output are fused by addition to obtain the (N-1)th residual output, where N is a preset value, representing an integer greater than 3;

[0022] The (N-1)th residual output is input to the Nth residual block of the spiking neural network to obtain the Nth pulse output; the Nth pulse output is input to the Nth long-short-term time series feature extraction module to obtain the Nth long-short-term time series feature; the Nth long-short-term time series feature and the (N-1)th pulse output are fused by addition to obtain the Nth fused feature, wherein the Nth fused feature is the target fused feature.

[0023] This invention integrates the output features of a spiking network with long and short-term temporal features, and is equivalent to adding temporal feature extraction to each residual block. This ensures that features at different levels are enhanced by long and short-term temporal features, thereby improving the accuracy of recognition. By combining a spiking neural network with a temporal feature extraction module, the model learns the temporal features of dynamic targets.

[0024] In one optional implementation, calculating the loss value of the target fusion feature using a preset loss function specifically includes:

[0025] The predicted probabilities of each category corresponding to the video sample data of the preset dynamic target are obtained by calculating the target fusion features using an exponential function.

[0026] The difference between the predicted probability and the real label corresponding to the video sample data of the preset dynamic target is calculated using the cross-entropy loss function to determine the loss value.

[0027] This invention uses an exponential function to calculate the predicted probability and a cross-entropy loss function to calculate the difference between the predicted probability and the true label. This is equivalent to continuously updating the model through target fusion features, thereby improving the dynamic target recognition capability and further improving the accuracy of visual recognition.

[0028] In one optional implementation, the long-short-term time series feature extraction module includes at least one convolutional kernel, and the long-short-term time series feature extraction module uses the Sigmoid function to perform nonlinear data operations.

[0029] In this embodiment of the invention, the long-short-term time series feature extraction module includes at least one convolutional kernel, and the long-short-term time series feature extraction module uses the Sigmoid function to perform nonlinear data operations. By combining the convolutional kernel and the Sigmoid function, the long-short-term time series feature extraction module can more accurately fuse the temporal information in dynamic targets, structurally enhancing the feature expression effect in the time dimension, so that the generated target fusion features can better reflect the data features of dynamic targets, thereby improving the accuracy of dynamic target recognition.

[0030] In one optional implementation, the long and short-term time series feature extraction module further includes a fully connected layer, wherein the number of output neurons of the fully connected layer is the same as the number of each category of the preset sample data.

[0031] The long and short-term time series feature extraction module of this invention includes a fully connected layer. The number of output neurons in this layer is the same as the number of each category in the preset sample data. The fully connected layer can map the extracted long and short-term time series features to a higher-dimensional space, making the features between different categories more obvious. Since the number of output neurons is the same as the number of categories, each neuron only needs to learn to recognize the features of a specific category, thereby improving the accuracy of identifying dynamic targets in different types.

[0032] Secondly, the present invention provides a visual recognition device based on a spiking neural network, the device comprising:

[0033] The sample encoding module is used to perform pulse encoding on video sample data of a preset dynamic target to obtain a pulse sequence;

[0034] The feature fusion module is used to input the pulse sequence into a preset spiking neural network for residual calculation to obtain the output features of the spiking network, and to extract long and short-term time series features from the output features of the spiking network. At the same time, the extracted long and short-term time series features are fused with the output features of the spiking network to obtain the target fused features.

[0035] The loss calculation module is used to calculate the loss value of the target fusion feature using a preset loss function;

[0036] The iterative training module is used to perform reverse iterative updates on the spiking neural network based on the loss value. When the loss value is less than a preset loss threshold, the reverse iterative update process is stopped to obtain the target long-term and short-term time series feature fusion model.

[0037] The model recognition module is used to input the video stream data of the dynamic target to be recognized into the target's long and short-term temporal feature fusion model to obtain the recognition result.

[0038] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the visual recognition method based on a spiking neural network as described in the first aspect or any corresponding embodiment thereof.

[0039] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to execute the visual recognition method based on a spiking neural network according to the first aspect or any corresponding embodiment thereof.

[0040] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the visual recognition method based on a spiking neural network according to the first aspect or any corresponding embodiment described above. Attached Figure Description

[0041] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating a visual recognition method based on a spiking neural network according to an embodiment of the present invention.

[0043] Figure 2 This is a schematic diagram of short-term temporal feature extraction of a visual recognition method based on a spiking neural network according to an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of long-term temporal feature extraction of a visual recognition method based on a spiking neural network according to an embodiment of the present invention;

[0045] Figure 4 This is a schematic diagram of the structure of a target long- and short-term temporal feature fusion model of a visual recognition method based on a spiking neural network according to an embodiment of the present invention;

[0046] Figure 5 This is a flowchart illustrating another visual recognition method based on a spiking neural network according to an embodiment of the present invention;

[0047] Figure 6This is a flowchart illustrating another visual recognition method based on a spiking neural network according to an embodiment of the present invention;

[0048] Figure 7 This is a structural block diagram of a visual recognition device based on a spiking neural network according to an embodiment of the present invention;

[0049] Figure 8 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] In existing spiking neural networks (SNNs), the network transmits pulse voltages between different time points, resulting in very limited transmission of temporal information.

[0052] Spiking neural networks (SNNs) cannot effectively identify dynamic data, mainly because they rely on the presence or absence of pulse signals rather than recognizing the timing of signal appearance and disappearance.

[0053] Therefore, when information changes suddenly in a short period of time, spiking neural networks will ignore short-term dynamic changes. Since they cannot accurately capture changes over time, SNNs will inevitably have difficulty recognizing key dynamic patterns, thus affecting the final recognition results.

[0054] This invention provides a visual recognition method based on spiking neural networks, which can be applied to visual recognition scenarios based on spiking neural networks, such as video stream data that needs to identify short-term dynamic change features. By combining long and short-term temporal feature extraction, more accurate dynamic pattern recognition can be achieved, thereby improving the model's ability and accuracy in recognizing dynamic video streams. This overcomes the shortcomings of existing spiking neural networks in that they cannot effectively utilize temporal information for accurate dynamic data recognition when capturing and recognizing dynamic data.

[0055] According to embodiments of the present invention, a visual recognition method based on a spiking neural network is provided. It should be noted that the steps shown in the flowcharts in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0056] This embodiment provides a visual recognition method based on a spiking neural network, which can be used in the aforementioned computer. Figure 1 This is a flowchart of a visual recognition method based on a spiking neural network according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0057] Step S101: Pulse coding is performed on the video sample data of the preset dynamic target to obtain a pulse sequence.

[0058] It should be noted that pulse coding refers to converting continuous video data into a discrete pulse sequence. The purpose of this embodiment is to use a coding mechanism to transform dynamic video sample data into a sequence of pulses, allowing the spiking neural network to mimic the processing methods of a biological nervous system to process the input dynamic video stream data. Specifically, pulse coding can be Poisson coding or temporal difference coding, etc.

[0059] The video sample data for the preset dynamic target can be RGB images or dynamic data streams in DVS format. Specifically, one type is static RGB images, such as those from the Cifar10 and ImageNet datasets; the other is dynamic event stream datasets, such as the Cifar10-DVS and DVS-Gesture datasets. These dynamic datasets are converted from their corresponding static datasets. In this embodiment, both types of video sample data differ from traditional video data and can be obtained from real-time captured dynamic videos or from preset datasets and databases.

[0060] Step S102: Input the pulse sequence into a preset spiking neural network for residual calculation to obtain the output features of the spiking network. Extract long and short-term time series features from the output features of the spiking network. Simultaneously, fuse the extracted long and short-term time series features with the output features of the spiking network to obtain the target fused features.

[0061] It should be noted that spiking neural networks are highly biomimetic neural network models capable of simulating the human brain's ability to process visual information. Residual computation refers to the method of using residuals to reduce the vanishing gradient problem, which can improve model training performance.

[0062] Specifically, the process of extracting short-term and long-term time series features is implemented by the short-term and long-term time series feature extraction module. The pulse sequence is input into a spiking neural network and processed through multiple residual blocks. The output features of the spiking network already contain time series information, but the original spiking neural network cannot process this time series information. In this embodiment, the short-term and long-term time series feature extraction module further extracts the short-term and long-term information from the time series information. The extracted short-term and long-term time series features are fused with the output features of the spiking network. This fusion represents the combination of the short-term and long-term dynamic change features of the input data. The short-term and long-term time series feature extraction module includes a short-term time series feature extraction module and a long-term time series feature extraction module.

[0063] For example, such as Figure 2 As shown, in short-term temporal feature extraction, the module first uses a shift operation to fuse the pulse features of adjacent time points, which can capture the dynamic changes between adjacent time points. Then, a convolution operation is used to extract short-term temporal features from these fused features. The extracted short-term temporal features are then fused with the original pulse features through multiplication to finally generate new pulse features. Specifically, Feature Fusion represents the process of combining features from different time points; C / 2channels shift left refers to shifting the channels of the feature map, moving the first half of the feature map to the left; Temporal conV is the convolution operation performed on the fused features to extract short-term temporal features; truncate refers to removing unnecessary parts during feature processing; and Multiplication fuses the extracted short-term temporal features with the original pulse features through multiplication.

[0064] For example, such as Figure 3As shown, in long-term temporal feature extraction, the module performs a mean operation on the pulse features along the channel dimension, fusing features from all time points together. This comprehensively considers information from all time points to extract long-term temporal features. The extracted long-term temporal features are then fused with new pulse features through broadcast multiplication to obtain the final pulse features. The use of broadcast multiplication makes the fusion process more flexible, effectively combining long-term temporal information with current features. Specifically, Feature Fusion represents the process of combining features from different time points; Mean represents the mean operation used for dimensionality reduction of time series data; Temporal conv represents temporal convolution, capturing local features of the data in the time dimension; and Broadcast multiplication represents the fusion of extracted long-term temporal features with current pulse features. Broadcast multiplication can automatically expand smaller-dimensional features to align them with larger-dimensional features, thereby achieving effective feature combination and better integrating long-term temporal information with current features.

[0065] Step S103: Calculate the loss value of the target fusion feature using a preset loss function.

[0066] Specifically, the loss value of the target fusion feature is calculated using a preset loss function. The choice of loss function can be mean squared error, cross-entropy calculation, etc., depending on the specific model training task. In this embodiment, the target fusion feature integrates long and short-term temporal features, which can improve the model's ability and accuracy in recognizing dynamic video stream data.

[0067] Step S104: Perform reverse iterative update on the spiking neural network based on the loss value. When the loss value is less than a preset loss threshold, stop the reverse iterative update process to obtain the target long-short-term time series feature fusion model.

[0068] Specifically, based on the calculated loss value, the weights of the spiking neural network are updated using the backpropagation algorithm. When the loss value is lower than the preset loss threshold, training stops, indicating that the model has converged and can perform the task of video stream recognition.

[0069] Step S105: Input the video stream data of the dynamic target to be identified into the target's long and short-term temporal feature fusion model to obtain the identification result.

[0070] Specifically, a pre-trained long-short-term temporal feature fusion model is used to identify the input video stream data. Since the target long-short-term temporal feature fusion model trained in this embodiment has learned the temporal features of the video stream, it can more accurately identify dynamic targets.

[0071] For example, refer to Figure 4When a canine-related dynamic video stream is input, it is first converted into multiple frames as input. In the specific target video stream data recognition task, the input multi-frame images first undergo preliminary feature extraction through the Conv1 layer, and then enter different stages (stage 1 to stage 4) in MultiStep Blocks, with each stage further processing feature fusion. The Conv and SN layers are used together to extract long and short temporal features from the video stream images, enhancing information by simulating the spiking behavior of neurons. The enhanced feature map data stream then passes through the next layer, applying LM and GM layers to reduce the dimensionality of the feature maps while retaining key recognition information. Simultaneously, pooling operations (FC and Mean) further simplify the feature map data volume and highlight more critical features, generating the final output probability distribution. The output result determines the category of the target object in the dynamic video data; for example, "Class: Dandie Dinmont" indicates that the model predicts the canine species in the input video stream data to be "Dandie Dinmont".

[0072] This invention converts video sample data into pulse sequences. Compared to directly processing pulse signals in existing technologies, pulse sequences obtained through pulse coding can improve the resolution of dynamic information. The output features of the pulse network are obtained through residual calculation, and long and short-term temporal features are extracted to obtain target fusion features. These target fusion features enhance the temporal dimension of feature representation, solving the problem that pulse neural networks cannot effectively capture the temporal dynamic features contained in the data. In this embodiment, the target fusion features combine long and short-term temporal features with the characteristics of the pulse signal itself. Therefore, by continuously updating the loss value in the pulse network signal using the target fusion features, the pulse network is updated, resulting in a target long and short-term temporal feature fusion model. The final target long and short-term temporal feature fusion model has the ability to efficiently identify dynamic targets. When the video stream data of the identified dynamic target is input into the target long and short-term temporal feature fusion model, a more accurate identification result can be obtained, solving the problem of poor dynamic data recognition and thus improving the accuracy of visual recognition of dynamic data.

[0073] This embodiment provides a visual recognition method based on a spiking neural network, which can be used in the aforementioned computer. Figure 5 This is a flowchart of a visual recognition method based on a spiking neural network according to an embodiment of the present invention, such as... Figure 5 As shown, the process includes the following steps:

[0074] Step S501: Pulse coding is performed on the video sample data of the preset dynamic target to obtain a pulse sequence. For details, please refer to [link to relevant documentation]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0075] Step S502: Input the pulse sequence into a preset spiking neural network for residual calculation to obtain the output features of the spiking network, and extract long and short-term time series features from the output features of the spiking network. At the same time, fuse the extracted long and short-term time series features with the output features of the spiking network to obtain the target fused features.

[0076] In one optional implementation, the preset spiking neural network includes at least two residual blocks and at least two long short-term time series feature extraction modules, wherein the number of residual blocks and long short-term time series feature extraction modules are the same and they are in a one-to-one correspondence.

[0077] In this embodiment of the invention, multiple residual blocks and long and short-term time-series feature extraction modules are used to form a preset spiking neural network in a one-to-one correspondence. Different residual blocks and time-series feature modules can realize the fusion of residual information and long and short-term time-series features, so that the final target fusion feature can more comprehensively represent the information of the dynamic target.

[0078] Specifically, step S502 above includes:

[0079] Step S5021: Input the pulse sequence into the first residual block of the spiking neural network to obtain the first pulse output.

[0080] Step S5022: Input the first pulse output to the first long-short-term time series feature extraction module to obtain the first long-short-term time series feature.

[0081] Step S5023: The first long-short time series feature and the first pulse output are fused by addition to obtain the first fused feature.

[0082] Step S5024: The first fusion feature and the pulse sequence are fused by addition to obtain the first residual output.

[0083] Step S5025: Starting from the first residual output, the target fusion feature is obtained by using the subsequent N-1 residual outputs according to the preset fusion process;

[0084] The preset fusion process includes:

[0085] The (N-2)th residual output is input to the (N-1)th residual block of the spiking neural network to obtain the (N-1)th pulse output; the (N-1)th pulse output is input to the (N-1)th long short-term time series feature extraction module to obtain the (N-1)th long short-term time series feature; the (N-1)th long short-term time series feature and the (N-1)th pulse output are fused by addition to obtain the (N-1)th fused feature; the (N-1)th fused feature and the (N-2)th residual output are fused by addition to obtain the (N-1)th residual output, where N is a preset value, representing an integer greater than 3;

[0086] The (N-1)th residual output is input to the Nth residual block of the spiking neural network to obtain the Nth pulse output; the Nth pulse output is input to the Nth long-short-term time series feature extraction module to obtain the Nth long-short-term time series feature; the Nth long-short-term time series feature and the (N-1)th pulse output are fused by addition to obtain the Nth fused feature, wherein the Nth fused feature is the target fused feature.

[0087] For example, each residual block corresponds to a long-short-term time series feature extraction module. If there are three residual blocks and three long-short-term time series feature extraction modules, residual block A1 corresponds to long-short-term time series feature extraction module B1, residual block A2 corresponds to long-short-term time series feature extraction module B2, and residual block A3 corresponds to long-short-term time series feature extraction module B3. The following steps are required to obtain the target fused features:

[0088] (1) Input the pulse sequence into the residual block A1 to obtain the first pulse output; input the first pulse output into the long and short-term time series feature extraction module B1 to obtain the first long and short-term time series feature; fuse these two features into the first fused feature, and then fuse it with the pulse sequence to obtain the first residual output.

[0089] (2) The first residual output is used as input and passes through the residual block A2 to obtain the second pulse output; the second pulse output is input into the long and short-term time series feature extraction module B2 to obtain the second long and short-term time series feature; the two features are fused into the second fused feature, and the second fused feature is fused with the first residual output to obtain the second residual output.

[0090] (3) The second residual output is used as input and passes through the residual block A3 to obtain the third pulse output; the third pulse output is input into the long and short-term time series feature extraction module B3 to obtain the third long and short-term time series feature; the two features are fused into the final target fusion feature.

[0091] Furthermore, the training parameters for the long-term and short-term time-series feature fusion model include: learning rate, batch size, and maximum number of training iterations. Stochastic gradient descent is used as the optimizer for the long-term and short-term time-series feature fusion model.

[0092] Furthermore, the long-short-term time series feature extraction module includes at least one convolutional kernel, and the long-short-term time series feature extraction module uses the Sigmoid function to perform nonlinear data operations.

[0093] It should be noted that, for data on the time axis, convolution kernel operations can efficiently extract correlation features between consecutive frames, helping the model understand the dynamic changes of training data in the short term. After the convolution operation, the features are non-linearly transformed by the Sigmoid function, which can amplify the significant features in the dynamically changing time signal, thereby accurately extracting the change information in the short and long term and improving the model's accuracy in recognizing dynamic targets.

[0094] In this embodiment of the invention, the long-short-term time series feature extraction module includes at least one convolutional kernel, and the long-short-term time series feature extraction module uses the Sigmoid function to perform nonlinear data operations. By combining the convolutional kernel and the Sigmoid function, the long-short-term time series feature extraction module can more accurately fuse the temporal information in dynamic targets, structurally enhancing the feature expression effect in the time dimension, so that the generated target fusion features can better reflect the data features of dynamic targets, thereby improving the accuracy of dynamic target recognition.

[0095] Furthermore, the long-short time series feature extraction module also includes a fully connected layer, the number of output neurons of the fully connected layer being the same as the number of each category of the preset sample data.

[0096] The long and short-term time series feature extraction module of this invention includes a fully connected layer. The number of output neurons in this layer is the same as the number of each category in the preset sample data. The fully connected layer can map the extracted long and short-term time series features to a higher-dimensional space, making the features between different categories more obvious. Since the number of output neurons is the same as the number of categories, each neuron only needs to learn to recognize the features of a specific category, thereby improving the accuracy of identifying dynamic targets in different types.

[0097] Step S503: Calculate the loss value of the target fused feature using a preset loss function. For details, please refer to [link to relevant documentation]. Figure 1 Step S103 of the illustrated embodiment will not be described again here.

[0098] Step S504: Perform reverse iterative updates on the spiking neural network based on the loss value. When the loss value is less than a preset loss threshold, stop the reverse iterative update process to obtain the target long-term and short-term time-series feature fusion model. For details, please refer to [link to details]. Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0099] Step S505: Input the video stream data of the dynamic target to be identified into the target's long-term and short-term temporal feature fusion model to obtain the identification result. For details, please refer to [link to relevant documentation]. Figure 1 Step S105 of the illustrated embodiment will not be described again here.

[0100] This invention integrates the output features of a spiking network with long and short-term temporal features, and is equivalent to adding temporal feature extraction to each residual block. This ensures that features at different levels are enhanced by long and short-term temporal features, thereby improving the accuracy of recognition. By combining a spiking neural network with a temporal feature extraction module, the model learns the temporal features of dynamic targets.

[0101] This embodiment provides a visual recognition method based on a spiking neural network, which can be used in the aforementioned computer. Figure 6 This is a flowchart of a visual recognition method based on a spiking neural network according to an embodiment of the present invention, such as... Figure 6 As shown, the process includes the following steps:

[0102] Step S601: Pulse coding is performed on the video sample data of the preset dynamic target to obtain a pulse sequence. For details, please refer to [link to details]. Figure 1 Step S101 of the illustrated embodiment will not be described again here.

[0103] Step S602: The pulse sequence is input into a preset spiking neural network for residual calculation to obtain the output features of the spiking network. Long-term and short-term time-series features are extracted from the output features of the spiking network. Simultaneously, the extracted long-term and short-term time-series features are fused with the output features of the spiking network to obtain the target fused features. For details, please refer to [link to details]. Figure 1 Step S102 of the illustrated embodiment will not be described again here.

[0104] Step S603: Calculate the loss value of the target fused feature using a preset loss function. Specifically, step S603 includes:

[0105] Step S6031: Calculate the target fusion features using an exponential function to obtain the predicted probabilities of each category corresponding to the video sample data of the preset dynamic target.

[0106] Step S6032: Calculate the difference between the predicted probability and the real label corresponding to the video sample data of the preset dynamic target using the cross-entropy loss function, and determine the loss value.

[0107] It should be noted that the exponential function can be the Softmax function, used as the activation function of the output layer of a multi-class classification model to transform the raw scores output by the network into a normalized probability distribution. In this embodiment, the loss function is the cross-entropy loss function. The principle behind calculating the cross-entropy loss function is that if the probability predicted by the model is close to the true class, the loss value will be small; if the difference is large, the loss value will be large. Therefore, by minimizing the cross-entropy loss, the model can continuously adjust its parameters and gradually improve the accuracy of predictions.

[0108] Step S604: Perform reverse iterative updates on the spiking neural network based on the loss value. When the loss value is less than a preset loss threshold, stop the reverse iterative update process to obtain the target long-term and short-term time-series feature fusion model. For details, please refer to [link to details]. Figure 1 Step S104 of the illustrated embodiment will not be described again here.

[0109] Step S605: Input the video stream data of the dynamic target to be identified into the target's long-term and short-term temporal feature fusion model to obtain the identification result. For details, please refer to [link to relevant documentation]. Figure 1 Step S105 of the illustrated embodiment will not be described again here.

[0110] The long and short-term time series feature extraction module of this invention includes a fully connected layer. The number of output neurons in this layer is the same as the number of each category in the preset sample data. The fully connected layer can map the extracted long and short-term time series features to a higher-dimensional space, making the features between different categories more obvious. Since the number of output neurons is the same as the number of categories, each neuron only needs to learn to recognize the features of a specific category, thereby improving the accuracy of identifying dynamic targets in different types.

[0111] This embodiment also provides a visual recognition device based on a spiking neural network. This system is used to implement the above embodiments and preferred embodiments, and details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0112] This embodiment provides a visual recognition device based on a spiking neural network, such as... Figure 7 As shown, it includes:

[0113] The sample encoding module 701 is used to perform pulse encoding on video sample data of a preset dynamic target to obtain a pulse sequence;

[0114] The feature fusion module 702 is used to input the pulse sequence into a preset spiking neural network for residual calculation to obtain the output features of the spiking network, and to extract long and short-term time series features from the output features of the spiking network. At the same time, the extracted long and short-term time series features are fused with the output features of the spiking network to obtain the target fused features.

[0115] The loss calculation module 703 is used to calculate the loss value of the target fusion feature using a preset loss function;

[0116] The iterative training module 704 is used to perform reverse iterative updates on the spiking neural network based on the loss value. When the loss value is less than a preset loss threshold, the reverse iterative update process is stopped to obtain the target long-term and short-term time series feature fusion model.

[0117] The model recognition module 705 is used to input the video stream data of the dynamic target to be recognized into the target's long and short-term temporal feature fusion model to obtain the recognition result.

[0118] In one optional implementation, the preset spiking neural network includes at least two residual blocks and at least two long short-term time series feature extraction modules, wherein the number of residual blocks and long short-term time series feature extraction modules are the same and they are in a one-to-one correspondence.

[0119] In some alternative implementations, the feature fusion module 702 includes:

[0120] A pulse input unit is used to input the pulse sequence into the first residual block of the spiking neural network to obtain the first pulse output.

[0121] The timing feature extraction unit is used to input the first pulse output to the first long-short-term timing feature extraction module to obtain the first long-short-term timing feature.

[0122] The fusion feature unit is used to fuse the first long-short-term time series feature and the first pulse output through an addition operation to obtain the first fused feature;

[0123] The residual fusion unit is used to obtain the target fusion feature by using the subsequent N-1 residual outputs according to a preset fusion process, starting from the first residual output;

[0124] The preset fusion process includes:

[0125] The (N-2)th residual output is input to the (N-1)th residual block of the spiking neural network to obtain the (N-1)th pulse output; the (N-1)th pulse output is input to the (N-1)th long short-term time series feature extraction module to obtain the (N-1)th long short-term time series feature; the (N-1)th long short-term time series feature and the (N-1)th pulse output are fused by addition to obtain the (N-1)th fused feature; the (N-1)th fused feature and the (N-2)th residual output are fused by addition to obtain the (N-1)th residual output, where N is a preset value, representing an integer greater than 3;

[0126] The (N-1)th residual output is input to the Nth residual block of the spiking neural network to obtain the Nth pulse output; the Nth pulse output is input to the Nth long-short-term time series feature extraction module to obtain the Nth long-short-term time series feature; the Nth long-short-term time series feature and the (N-1)th pulse output are fused by addition to obtain the Nth fused feature, wherein the Nth fused feature is the target fused feature.

[0127] In one optional implementation, the loss calculation module 703 specifically includes:

[0128] The prediction probability calculation unit is used to calculate the target fusion features through an exponential function to obtain the prediction probability of each category corresponding to the video sample data of the preset dynamic target;

[0129] The loss value calculation unit is used to calculate the difference between the predicted probability and the real label corresponding to the video sample data of the preset dynamic target using the cross-entropy loss function, and to determine the loss value.

[0130] In one optional implementation, the long-short-term time series feature extraction module includes at least one convolutional kernel, and the long-short-term time series feature extraction module uses the Sigmoid function to perform nonlinear data operations.

[0131] In one optional implementation, the long and short-term time series feature extraction module further includes a fully connected layer, wherein the number of output neurons of the fully connected layer is the same as the number of each category of the preset sample data.

[0132] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0133] In this embodiment, the visual recognition device based on spiking neural networks is presented in the form of functional units. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0134] This invention also provides a computer device having the above-described features. Figure 7 The visual recognition device shown is based on a spiking neural network.

[0135] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 8 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 8 Take a processor 10 as an example.

[0136] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0137] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0138] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0139] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0140] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0141] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0142] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0143] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the defined scope.

Claims

1. A method of visual recognition based on a spiking neural network, characterized in that, The method comprises: pulse coding video sample data of a preset dynamic target to obtain a pulse sequence; inputting the pulse sequence into a preset pulse neural network to perform residual calculation and obtain pulse network output features; performing long-short term time sequence feature extraction on the pulse network output features, and fusing the extracted long-short term time sequence features with the pulse network output features to obtain target fusion features; calculating a loss value of the target fusion features by using a preset loss function; performing reverse iterative updating on the pulse neural network according to the loss value, and stopping the reverse iterative updating process when the loss value is less than a preset loss threshold value, to obtain a target long-short term time sequence feature fusion model; inputting video stream data of a dynamic target to be recognized into the target long-short term time sequence feature fusion model to obtain a recognition result.

2. The method of claim 1, wherein, The preset pulse neural network comprises at least two residual blocks and at least two long-short term time sequence feature extraction modules, and the number of the residual blocks is the same as that of the long-short term time sequence feature extraction modules and in one-to-one correspondence.

3. The method of claim 2, wherein, The pulse sequence is inputted into a first residual block of the pulse neural network to obtain a first pulse output; the first pulse output is inputted into a first long-short term time sequence feature extraction module to obtain a first long-short term time sequence feature; the first long-short term time sequence feature and the first pulse output are fused by addition operation to obtain a first fusion feature; the first fusion feature and the pulse sequence are fused by addition operation to obtain a first residual output; the target fusion features are obtained by using N-1 subsequent residual outputs according to a preset fusion process from the first residual output; the preset fusion process comprises: inputting an N-2 residual output into an N-1 residual block of the pulse neural network to obtain an N-1 pulse output; inputting the N-1 pulse output into an N-1 long-short term time sequence feature extraction module to obtain an N-1 long-short term time sequence feature; fusing the N-1 long-short term time sequence feature and the N-1 pulse output by addition operation to obtain an N-1 fusion feature; fusing the N-1 fusion feature and the N-2 residual output by addition operation to obtain an N-1 residual output, wherein N is a preset value representing an integer greater than 3; inputting the N-1 residual output into an N residual block of the pulse neural network to obtain an N pulse output; inputting the N pulse output into an N long-short term time sequence feature extraction module to obtain an N long-short term time sequence feature; fusing the N long-short term time sequence feature and the N-1 residual output by addition operation to obtain an N fusion feature, wherein the N fusion feature is the target fusion feature. ​ ​ ​ ​ ​ ​ ​ ​ 4. The method of claim 3, wherein, The loss value of the target fusion feature is calculated by using a preset loss function, and specifically includes: The target fusion feature is calculated by an exponential function to obtain prediction probabilities of each category corresponding to the video sample data of the preset dynamic target; The difference information between the prediction probabilities and the real labels corresponding to the video sample data of the preset dynamic target is calculated by using a cross-entropy loss function to determine the loss value.

5. The method according to any one of claims 2-4, characterized in that, The long-short-term time sequence feature extraction module includes at least one convolution kernel, and the long-short-term time sequence feature extraction module adopts a Sigmoid function for data nonlinear operation.

6. The method of claim 5, wherein, The long-short-term time sequence feature extraction module further includes a fully connected layer, and the number of output neurons of the fully connected layer is the same as the number of each category of the preset sample data.

7. A visual recognition device based on a spiking neural network, characterized in that, The device includes: A sample encoding module encodes the video sample data of the preset dynamic target to obtain a pulse sequence; A feature fusion module inputs the pulse sequence into a preset pulse neural network to perform residual calculation, obtains pulse network output features, performs long-short-term time sequence feature extraction on the pulse network output features, and fuses the extracted long-short-term time sequence features with the pulse network output features to obtain target fusion features; A loss calculation module calculates a loss value of the target fusion feature by using a preset loss function; An iterative training module performs reverse iterative updating on the pulse neural network according to the loss value, stops the reverse iterative updating process when the loss value is less than a preset loss threshold, and obtains a target long-short-term time sequence feature fusion model; A model identification module inputs video stream data of a dynamic target to be identified into the target long-short-term time sequence feature fusion model to obtain an identification result.

8. A computer device, comprising: It includes: A memory and a processor are communicatively connected between each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the visual identification method based on the pulse neural network in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, and the computer instructions are used to make the computer execute the visual identification method based on the pulse neural network in any one of claims 1-6.

10. A computer program product, characterised in that, It includes computer instructions, and the computer instructions are used to make the computer execute the visual identification method based on the pulse neural network in any one of claims 1-6.

Images