Small target object recognition method and device based on pulse neural network
By using a dual-stream fusion model in small-target object recognition, combining pulsed neural networks and residual networks, the problem of information loss in convolutional neural networks in low-resolution target recognition is solved, and the recognition accuracy and feature extraction capabilities are improved.
Patent Information
- Application Number
- CN202410123207.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-01-29
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-01-29
AI Technical Summary
When processing target objects with low resolution and small size, convolutional neural networks are difficult to capture subtle features of the image, resulting in low recognition performance. The deep convolutional networks are seriously depleted in spatial information during feature extraction, resulting in small target objects. Poor recognition effect.
Using a small target object recognition method based on pulse neural network, combining pulse neural network and residual network through a dual-stream fusion model, the pulse neural network is used to extract time series features from the image, and the residual network extracts residual features, and fuses through Concat operation. Features, iterative training is performed using cross-entropy loss function to update network parameters to improve recognition accuracy.
The accuracy of small-target object recognition is improved, and through the combination of dynamic processing and static processing, richer image feature information is obtained, the recognition performance of the network is enhanced, and information loss is reduced.
Smart Images

Figure CN118072079B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of pattern recognition technology, and in particular to the field of small target object recognition. Background Art
[0002] Although deep convolutional networks have achieved great success in most image processing tasks, they are often limited in processing low-resolution and small-sized target objects.
[0003] In related technologies, due to the limited number of pixels in object images, there is little information available for analysis and recognition. Information is lost in the process of extracting image features using traditional convolutional neural networks, making it difficult for the network to capture subtle features of the image, resulting in low network recognition performance. When using deep convolutional networks, spatial information is also lost in the feature extraction process, resulting in the gradual loss of spatial details of the original image, which in turn leads to poor recognition of small target objects.
[0004] Spiking neural networks are gaining more and more attention from scholars because they are closer to the operating mechanism of the human brain. A popular method for building spiking neural networks is the transformation from deep convolutional networks. As a highly biologically simulated neural network model, spiking neural networks simulate the human brain's ability to process visual information and show significant potential in small target recognition tasks. Summary of the invention
[0005] The present invention provides a small target object recognition method and device based on a pulse neural network, which is used to solve the defects of the prior art in extracting small target object features, such as information loss, which leads to the gradual loss of spatial details of the original image and poor small target object recognition effect, thereby improving the accuracy of object recognition.
[0006] The present invention provides a small target object recognition method based on a pulse neural network, comprising:
[0007] Acquire an image of the object to be measured;
[0008] Classify and identify the image of the object to be tested based on the dual-stream fusion model to obtain a classification and identification result;
[0009] Among them, the dual-stream fusion model is constructed based on a pulse neural network and a residual network, and the dual-stream fusion model is trained by taking sample object images as training samples and fusion features as training features; the fusion features are determined based on the output features of the pulse neural network and the output features of the residual network, and the pulse neural network and the residual network respectively include a plurality of residual blocks arranged in sequence, and the features of the current residual block input of the pulse neural network are the sum of the features output by the previous residual block of the pulse neural network and the features output by the residual block corresponding to the previous residual block in the residual network.
[0010] According to the small target object recognition method based on pulse neural network provided by the present invention, the dual-stream fusion model is trained by the following steps:
[0011] Pulse encoding is performed on the preprocessed sample object image to obtain an image pulse sequence;
[0012] Dynamically processing the image pulse sequence based on the pulse neural network to obtain the pulse neural network output feature, and statically processing the preprocessed sample image based on the residual network to obtain the residual network output feature;
[0013] The pulse neural network output feature and the residual network output feature are combined through a Concat operation to obtain the fusion feature;
[0014] The loss value corresponding to the fusion feature is calculated according to the cross entropy loss function, and the pulse neural network and the residual network are reversely iterated and updated using the loss value to update the loss value, and the dual-stream fusion model is obtained when the updated loss value is less than the target threshold.
[0015] According to the small target object recognition method based on the pulse neural network provided by the present invention, the pulse neural network and the residual network both include N residual blocks, where N is an integer greater than 2;
[0016] The dynamically processing the image pulse sequence based on the pulse neural network to obtain the pulse neural network output feature, and the statically processing the preprocessed sample object image based on the residual network to obtain the residual network output feature includes:
[0017] Inputting the image pulse sequence into the first residual block of the pulse neural network to obtain a first pulse output, inputting the preprocessed sample object image into the first residual block of the residual network to obtain a first residual output, and performing an Add operation on the first pulse output and the first residual output to obtain a first operation feature;
[0018] Inputting the first operation feature into the second residual block of the spiking neural network to obtain a second pulse output, inputting the first residual output into the second residual block of the residual network to obtain a second residual output, and performing an Add operation on the second pulse output and the second residual output to obtain a second operation feature, and so on;
[0019] Until the N-1th operation feature is input into the Nth residual block of the pulse neural network to obtain the pulse neural network output feature, the N-1th residual output is input into the Nth residual block of the residual network to obtain the residual network output feature.
[0020] According to the small target object recognition method based on the spiking neural network provided by the present invention, the reverse iterative update of the spiking neural network is achieved by the following steps:
[0021] The spiking neural network is reversely iterated and updated using a substitute gradient; wherein the substitute gradient includes:
[0022]
[0023] Among them, α is the learning rate and x is the loss value.
[0024] According to the small target object recognition method based on spiking neural network provided by the present invention, the preprocessed sample object image is obtained by the following steps:
[0025] Performing data amplification on the sample object image and adjusting the size of the amplified image to obtain a target sample image;
[0026] The target sample image is normalized to obtain the preprocessed sample image.
[0027] According to the small target object recognition method based on pulse neural network provided by the present invention, the dual-stream fusion model also includes multiple fully connected layers;
[0028] The pulse neural network includes at least one first convolution kernel and a plurality of residual blocks, wherein the plurality of residual blocks are determined based on a plurality of second convolution kernels; the activation layers included in each residual block are all set as LIF neurons;
[0029] The convolution kernel in the residual network is the same as the first convolution kernel in the pulse neural network, the residual block in the residual network is the same as the residual block in the pulse neural network, and the ReLU activation function is used in the residual network to perform data nonlinear operations, and the LIF neuron is used as the activation function in the pulse neural network to perform data nonlinear operations.
[0030] The present invention also provides a small target object recognition device based on a pulse neural network, comprising:
[0031] An image acquisition module, used to acquire an image of the object to be measured;
[0032] A small target object recognition module is used to classify and recognize the image of the object to be tested based on a dual-stream fusion model to obtain a classification and recognition result;
[0033] Among them, the dual-stream fusion model is constructed based on a pulse neural network and a residual network, and the dual-stream fusion model is trained by taking sample object images as training samples and fusion features as training features; the fusion features are determined based on the output features of the pulse neural network and the output features of the residual network, and the pulse neural network and the residual network respectively include a plurality of residual blocks arranged in sequence, and the features of the current residual block input of the pulse neural network are the sum of the features output by the previous residual block of the pulse neural network and the features output by the residual block corresponding to the previous residual block in the residual network.
[0034] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for recognizing small target objects based on a pulse neural network as described above is implemented.
[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described small target object recognition methods based on a pulse neural network.
[0036] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned small target object recognition methods based on pulse neural networks.
[0037] The small target object recognition method and device based on pulse neural network provided by the present invention recognize the image of the object to be tested through a dual-stream fusion model, specifically utilize the pulse neural network to extract the pulse neural network output features including multiple time series from the image, utilize the residual network to extract the residual network output features from the image, so as to obtain richer image feature information, and realize iterative training of the network and image classification through the fusion features of the residual network output features and the pulse neural network output features, thereby improving the accuracy of small target object recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0039] Figure 1 This is one of the flow charts of the small target object recognition method based on the pulse neural network provided by the present invention;
[0040] Figure 2This is the second flow chart of the small target object recognition method based on the pulse neural network provided by the present invention;
[0041] Figure 3 It is a structural schematic diagram of a dual-stream fusion model provided by the present invention;
[0042] Figure 4 It is a structural schematic diagram of a small target object recognition device based on a pulse neural network provided by the present invention;
[0043] Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION
[0044] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0045] Combine the following Figure 1-Figure 4 The present invention describes a small target object recognition method and device based on a pulse neural network.
[0046] Figure 1 FIG. 1 is one of the flow charts of the small target object recognition method based on the pulse neural network provided by the present invention, such as Figure 1 As shown, the small target object recognition method based on pulse neural network includes the following steps:
[0047] Step 110: Acquire an image of the object to be measured.
[0048] In this step, the image of the object to be measured includes a small target object, wherein the small target object refers to a target object with a smaller size in the image, for example, a tiny object, a human face, or a vehicle in the image.
[0049] In this embodiment, the image of the object to be measured can be obtained from an image captured in real time, or can be obtained from an image data set or a database.
[0050] Step 120, classify and identify the image of the object to be tested based on the dual-stream fusion model to obtain a classification and identification result; wherein the dual-stream fusion model is constructed based on a pulse neural network and a residual network, and the dual-stream fusion model is trained by using sample object images as training samples and fusion features as training features; the fusion features are determined based on the output features of the pulse neural network and the output features of the residual network, the pulse neural network and the residual network respectively include a plurality of residual blocks arranged in sequence, and the features of the current residual block input of the pulse neural network are the sum of the features output by the previous residual block of the pulse neural network and the features output by the residual block corresponding to the previous residual block in the residual network.
[0051] In this step, the two-stream fusion model is a two-stream network structure integrating a spiking neural network (Spiking Neural Network, SNN) and a residual network. Among them, the spiking neural network, as a highly biologically simulated neural network model, can simulate the ability of the human brain to process visual information. When processing the pulse sequence corresponding to the image, it not only statically processes the spatial features of the image, but also dynamically interprets the information from the time dimension. By simulating this dynamic processing capability, the spiking neural network can improve the recognition accuracy of objects while maintaining low energy consumption.
[0052] In this embodiment, the dual-stream fusion model includes a pulse neural network branch, a residual network (ResNet) branch and a fully connected layer; specifically, the pulse neural network branch is used to extract the pulse neural network output features containing time series information from the image of the object to be tested, and the residual network branch is used to extract the residual network output features from the image of the object to be tested. After the residual network output features and the pulse neural network output features are feature fused, the output prediction value is calculated through the fully connected layer, the pulse neural network and the residual network are iteratively trained in turn, and when the network converges, a usable dual-stream fusion model is obtained for subsequent small target object recognition.
[0053] In this embodiment, the pulse neural network and the residual network are set to include three residual blocks respectively, then the input of the second residual block of the pulse neural network includes the output of the first residual block of the pulse neural network and the output of the first residual block of the residual network, and the input features of the third residual block of the pulse neural network include the output of the second residual block of the pulse neural network and the output of the second residual block of the residual network. Specifically, these two outputs are added to obtain the input of the next residual block of the pulse neural network; correspondingly, the input of the second residual block of the residual network is the output of the first residual block of the residual network, the input of the third residual block of the residual network is the output of the second residual block of the residual network, and so on.
[0054] The small target object recognition method based on pulse neural network provided by the embodiment of the present invention recognizes the image of the object to be tested through a dual-stream fusion model. Specifically, the pulse neural network is used to extract the pulse neural network output features including multiple time series from the image, and the residual network is used to extract the residual network output features from the image, so as to obtain richer image feature information, and the iterative training of the network and the image classification are realized through the fusion features of the residual network output features and the pulse neural network output features, thereby improving the accuracy of object recognition.
[0055] In some embodiments, a dual-stream fusion model is constructed by the following steps: pulse encoding the preprocessed sample object image to obtain an image pulse sequence; dynamically processing the image pulse sequence based on a pulse neural network to obtain a pulse neural network output feature, and statically processing the preprocessed sample object image based on a residual network to obtain a residual network output feature; merging the pulse neural network output feature and the residual network output feature through a Concat operation to obtain a fusion feature; calculating the loss value corresponding to the fusion feature according to the cross entropy loss function, and using the loss value to reversely iterate and update the pulse neural network and the residual network to update the loss value, and obtaining a dual-stream fusion model when the updated loss value is less than the target threshold.
[0056] In this embodiment, the image preprocessing operation includes operations such as image expansion, image normalization, and outlier removal.
[0057] In this embodiment, pulse coding includes performing Poisson coding or other types of coding on the image; specifically, the time step T is set to N, where N is a natural number greater than 1 (for example, N=16), and within the interval, the probability distribution of the number of pulses generated by Poisson coding is as follows:
[0058]
[0059] Among them, λ is the reciprocal of each pixel value after image normalization, k is the number of pulses occurring in the interval [t, t+τ], and τ is the time interval.
[0060] In this embodiment, the cross entropy loss function is expressed as follows:
[0061] Loss=-(y·log(p)+(1-y)·log(1-p));
[0062] Among them, y is the corresponding label in the input network of the sample, and p is the probability value of each prediction result obtained after the final model output is classified by the softmax classifier.
[0063] Figure 2This is the second flow chart of the small target object recognition method based on the pulse neural network provided by the present invention. Figure 2 In the illustrated embodiment, the neural network training steps include:
[0064] (1) Initializing the network parameters of the spiking neural network and the residual network (corresponding to the neural network parameters);
[0065] (2) performing Poisson encoding on the preprocessed sample object image to obtain a pulse sequence, and inputting the pulse sequence into a pulse neural network to extract the pulse neural network output features of the image, and inputting the preprocessed sample object image into a residual network to extract the residual network output features of the image, and outputting the recognition result corresponding to the image after fusing the residual network output features and the pulse neural network output features;
[0066] (3) The loss value (corresponding error) between the recognition result obtained by calculating the loss function and the actual result of the training image;
[0067] (4) Using the loss value to perform back propagation updates on the spiking neural network and the residual network, i.e., to achieve iterative updates of the loss value;
[0068] (5) If the updated loss value is greater than or equal to the target threshold, return to step (4) to continue training; until the updated loss value is less than the target threshold, a dual-stream fusion model is obtained.
[0069] In this embodiment, the reverse iterative update of the spiking neural network is implemented by the following steps: reverse iterative update of the spiking neural network is performed using a substitute gradient; wherein the substitute gradient includes:
[0070]
[0071] Among them, α is the learning rate and x is the loss value.
[0072] In this embodiment, the ReLU activation function is used in the reverse iterative update process of the residual network to prevent gradient vanishing and gradient exploding during network training.
[0073] The small target object recognition method based on pulse neural network provided by the embodiment of the present invention performs pulse encoding on the preprocessed sample object image, and inputs it into the pulse neural network to extract the pulse neural network output features of the pulse sequence, inputs the preprocessed sample object image into the residual network to extract the residual network output features, calculates the loss value corresponding to the fusion feature according to the cross entropy loss function, and reversely iterates and updates the pulse neural network and the residual network to obtain a dual-stream fusion model, thereby improving the feature representation capability and thus improving the classification performance of the dual-stream fusion model.
[0074] In some embodiments, the pulse neural network and the residual network both include N residual blocks, where N is an integer greater than 2; dynamically processing the image pulse sequence based on the pulse neural network to obtain the pulse neural network output feature, and statically processing the preprocessed sample object image based on the residual network to obtain the residual network output feature includes: inputting the image pulse sequence into the first residual block of the pulse neural network to obtain a first pulse output, inputting the preprocessed sample object image into the first residual block of the residual network to obtain a first residual output, and performing an Add operation on the first pulse output and the first residual output to obtain a first operation feature; inputting the first operation feature into the second residual block of the pulse neural network to obtain a second pulse output, inputting the first residual output into the second residual block of the residual network to obtain a second residual output, and performing an Add operation on the second pulse output and the second residual output to obtain a second operation feature, and so on; until the N-1th operation feature is input into the Nth residual block of the pulse neural network to obtain the pulse neural network output feature, and the N-1th residual output is input into the Nth residual block of the residual network to obtain the residual network output feature.
[0075] In this embodiment, each residual block in the pulse neural network can perform image processing operations for multiple time steps. For example, after the residual block performs image processing for 16 time steps, the outputs of the 16 time steps are merged through an Add operation, and the residual network supplements the information of the pulse neural network, that is, the 1st to N-1 pulse outputs and the 1st to N-1 residual outputs are respectively added and used as the inputs of the 2nd to N residual blocks of the pulse neural network, which are also merged through an Add operation. Finally, the outputs of the pulse residual network branch and the residual network branch are fused through a Concat operation to achieve comprehensive feature processing and obtain corresponding fused features.
[0076] In this embodiment, the number of N can be set according to user needs.
[0077] Figure 3 is a schematic diagram of the structure of the dual-stream fusion model provided by the present invention. Figure 3 In the illustrated embodiment, when N=3, the step of obtaining the fusion feature includes:
[0078] (1) inputting the image pulse sequence into the first residual block of the spiking neural network to obtain a first pulse output, and inputting the preprocessed sample object image into the first residual block of the residual network to obtain a first residual output;
[0079] (2) using the first pulse output and the first residual output as inputs of the second residual block of the pulse neural network through an Add operation, and then obtaining a second pulse output, and inputting the first residual output into the second residual block of the residual network to obtain a second residual output;
[0080] (3) using the second pulse output and the second residual output as inputs of the second residual block of the spiking neural network through an Add operation to obtain a spiking neural network output feature, and inputting the second residual output into the third residual block of the residual network to obtain a residual network output feature;
[0081] (4) The residual network output features and the pulse neural network output features are fused through the Concat operation to obtain the fused features.
[0082] The small target object recognition method based on pulse neural network provided by the embodiment of the present invention extracts the corresponding pulse neural network output features by setting up a pulse neural network (including the information supplement of the pulse neural network branch by the corresponding layer output of the residual network branch, that is, the Add operation), and then combines the residual network output feature information of the residual network output to construct a fusion feature with stronger representation ability. On the one hand, it relies on the pulse neural network to control power consumption, and at the same time improves the performance of the dual-stream fusion model.
[0083] In some embodiments, the preprocessed sample object image is obtained by the following steps: data amplification of the sample object image and adjustment of the amplified image size to obtain a target sample image; normalization of the target sample image to obtain a preprocessed sample object image.
[0084] In this embodiment, the number of sample object images is expanded through data augmentation, and data augmentation can be achieved in the following ways: such as flipping, rotating, translating, scaling, noise perturbation, brightness contrast transformation, etc., or using a generative adversarial network to expand the number of samples.
[0085] In this embodiment, the target sample image is related to the data size standard allowed by the network.
[0086] In this embodiment, the normalization process of the image is achieved by the following formula:
[0087]
[0088] Among them, z norm is the sample object image after preprocessing, z is the target sample image, min(z) is the minimum pixel value in z, and max(z) is the maximum pixel value in z.
[0089] The small target object recognition method based on pulse neural network provided by the embodiment of the present invention ensures the number of samples required for network training by performing data amplification and normalization processing on the sample object images, reduces the problem that the network is prone to overfitting, and reduces the problem of data imbalance between samples, thereby improving the sample quality and facilitating the robustness of subsequent model training.
[0090] In some embodiments, the dual-stream fusion model also includes multiple fully connected layers; the pulse neural network includes at least one first convolution kernel and multiple residual blocks, and the multiple residual blocks are determined based on multiple second convolution kernels; the activation layer included in each residual block is set to LIF neurons; the convolution kernel in the residual network is the same as the first convolution kernel in the pulse neural network, the residual block in the residual network is the same as the residual block in the pulse neural network, and the ReLU activation function is used in the residual network for data nonlinear operations, and the LIF neuron is used as the activation function in the pulse neural network for data nonlinear operations.
[0091] In this embodiment, the current equation of the LIF (Leaky Integrate-and-Fire) neuron model is expressed as:
[0092] I(t)=I R +I C ;
[0093] Where I(t) is the current passing through the LIF neuron model at time t, IR is the current passing through the resistor R, IR is the current passing through the resistor R, and IC is the current passing through the capacitor C; according to the above formula, we can derive:
[0094]
[0095] Among them, U(t) is the voltage across the LIF neuron model at time t, U res is the resting potential, U 0 is the membrane potential at t = 0, τ m is the membrane time constant, where τ m =RC.
[0096] In this embodiment, the sizes of the first convolution kernel and the second convolution kernel can be selected according to user needs. For example, the first convolution kernel is a 5*5 convolution kernel, and the second convolution is at least one of a 3*3 convolution kernel and a 1*1 convolution.
[0097] In this embodiment, the pulse neural network includes a 5*5 convolution kernel and three residual blocks, which can ensure the effective transmission of key features, wherein each residual block is composed of two 3*3 convolution kernels and one 1*1 convolution kernel, and the activation function adopts LIF neuron, which can balance biological rationality and practicality.
[0098] In this embodiment, the residual network includes a 5*5 convolution kernel and three residual blocks to ensure the effective transmission of key features. Each residual block consists of two 3*3 convolution kernels and one 1*1 convolution kernel. ReLU is used as the activation function to effectively solve the problems of gradient disappearance and gradient explosion.
[0099] In this embodiment, the dual-stream fusion model also includes three fully connected layers, the number of output neurons in each fully connected layer is 1024, 256, and the number of categories (C), respectively, and a ReLU activation function is used after each fully connected layer.
[0100] In this embodiment, during the training of the dual-stream fusion model, the relevant parameters include: using stochastic gradient descent (SGD) as the optimizer, setting the learning rate to 0.1, the sample batch batch_size to 32, and the maximum number of training epochs to 30.
[0101] The small target object recognition method based on the pulse neural network provided by the embodiment of the present invention constructs a pulse neural network and a corresponding residual network through at least one first convolution kernel and multiple residual blocks. The activation function in the pulse neural network is constructed using LIF neurons, and the convolution kernel in the residual network uses the ReLU activation function to perform data nonlinear operations. While ensuring the effective transmission of key features, it alleviates the gradient vanishing and gradient exploding problems during network training.
[0102] The small target object recognition device based on the pulse neural network provided by the present invention is described below. The small target object recognition device based on the pulse neural network described below and the small target object recognition method based on the pulse neural network described above can be referenced to each other.
[0103] Figure 4 : is a schematic diagram of the structure of a small target object recognition device based on a pulse neural network provided by the present invention. Figure 4 As shown, the small target object recognition device based on pulse neural network includes: an image acquisition module 410 and a small target object recognition module 420.
[0104] An image acquisition module 410 is used to acquire an image of an object to be measured;
[0105] A small target object recognition module 420 is used to classify and recognize the image of the object to be tested based on the dual-stream fusion model to obtain a classification and recognition result;
[0106] Among them, the dual-stream fusion model is constructed based on the pulse neural network and the residual network. The dual-stream fusion model is trained by taking the sample object image as the training sample and the fusion feature as the training feature; the fusion feature is determined based on the output feature of the pulse neural network and the output feature of the residual network. The pulse neural network and the residual network respectively include multiple residual blocks arranged in sequence. The feature of the current residual block input of the pulse neural network is the sum of the feature output of the previous residual block of the pulse neural network and the feature output of the residual block corresponding to the previous residual block in the residual network.
[0107] The small target object recognition device based on the pulse neural network provided by the embodiment of the present invention recognizes the image of the object to be tested through a dual-stream fusion model. Specifically, the pulse neural network extracts the pulse neural network output features including multiple time series from the image, and uses the residual network to extract the residual network output features from the image, so as to obtain richer image feature information, and realizes iterative training of the network and image classification through the fusion features of the residual network output features and the pulse neural network output features, thereby improving the accuracy of object recognition.
[0108] Figure 5 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute a small target object recognition method based on a pulse neural network, the method comprising: obtaining an image of an object to be tested; classifying and identifying the image of the object to be tested based on a dual-stream fusion model to obtain a classification recognition result; wherein the dual-stream fusion model is constructed based on a pulse neural network and a residual network, the dual-stream fusion model is obtained by training with a sample object image as a training sample and with a fusion feature as a training feature; the fusion feature is determined based on the output feature of the pulse neural network and the output feature of the residual network, the pulse neural network and the residual network respectively include a plurality of residual blocks arranged in sequence, and the feature of the current residual block input of the pulse neural network is the sum of the feature output of the previous residual block of the pulse neural network and the feature output of the residual block corresponding to the previous residual block in the residual network.
[0109] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0110] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the small target object recognition method based on the pulse neural network provided by the above methods, and the method includes: obtaining an image of the object to be tested; classifying and identifying the image of the object to be tested based on a dual-stream fusion model to obtain a classification and recognition result; wherein the dual-stream fusion model is constructed based on a pulse neural network and a residual network, and the dual-stream fusion model is trained by using sample object images as training samples and fusion features as training features; the fusion features are determined based on the output features of the pulse neural network and the output features of the residual network, and the pulse neural network and the residual network respectively include multiple residual blocks arranged in sequence, and the features of the current residual block input of the pulse neural network are the sum of the features output by the previous residual block of the pulse neural network and the features output by the residual block corresponding to the previous residual block in the residual network.
[0111] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the small target object recognition method based on the pulse neural network provided by the above-mentioned methods, the method comprising: obtaining an image of the object to be tested; classifying and identifying the image of the object to be tested based on a dual-stream fusion model to obtain a classification and recognition result; wherein the dual-stream fusion model is constructed based on a pulse neural network and a residual network, and the dual-stream fusion model is trained by using sample object images as training samples and using fusion features as training features; the fusion features are determined based on the output features of the pulse neural network and the output features of the residual network, the pulse neural network and the residual network respectively include a plurality of residual blocks arranged in sequence, and the features of the current residual block input of the pulse neural network are the sum of the features output by the previous residual block of the pulse neural network and the features output by the residual block corresponding to the previous residual block in the residual network.
[0112] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0113] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0114] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A small target object recognition method based on pulse neural network, characterized in that: include: Acquire an image of the object to be measured; Classify and identify the image of the object to be tested based on the dual-stream fusion model to obtain a classification and identification result; Among them, the dual-stream fusion model is constructed based on a pulse neural network and a residual network, and the dual-stream fusion model is trained by taking sample object images as training samples and fusion features as training features; the fusion features are determined based on the output features of the pulse neural network and the output features of the residual network, and the pulse neural network and the residual network respectively include a plurality of residual blocks arranged in sequence, and the features of the current residual block input of the pulse neural network are the sum of the features output by the previous residual block of the pulse neural network and the features output by the residual block corresponding to the previous residual block in the residual network. The dual-stream fusion model is trained by the following steps: Pulse encoding is performed on the preprocessed sample object image to obtain an image pulse sequence; Dynamically processing the image pulse sequence based on the pulse neural network to obtain the pulse neural network output feature, and statically processing the preprocessed sample image based on the residual network to obtain the residual network output feature; The pulse neural network output feature and the residual network output feature are combined through a Concat operation to obtain the fusion feature; Calculating the loss value corresponding to the fusion feature according to the cross entropy loss function, using the loss value to reversely iterate and update the spiking neural network and the residual network to update the loss value, and obtaining the dual-stream fusion model when the updated loss value is less than a target threshold; the spiking neural network and the residual network each include N residual blocks, where N is an integer greater than 2; The dynamically processing the image pulse sequence based on the pulse neural network to obtain the pulse neural network output feature, and the statically processing the preprocessed sample object image based on the residual network to obtain the residual network output feature includes: Inputting the image pulse sequence into the first residual block of the pulse neural network to obtain a first pulse output, inputting the preprocessed sample object image into the first residual block of the residual network to obtain a first residual output, and performing an Add operation on the first pulse output and the first residual output to obtain a first operation feature; Inputting the first operation feature into the second residual block of the spiking neural network to obtain a second pulse output, inputting the first residual output into the second residual block of the residual network to obtain a second residual output, and performing an Add operation on the second pulse output and the second residual output to obtain a second operation feature, and so on; Until the N-1th operation feature is input into the Nth residual block of the pulse neural network to obtain the pulse neural network output feature, the N-1th residual output is input into the Nth residual block of the residual network to obtain the residual network output feature.
2. The small target object recognition method based on pulse neural network according to claim 1 is characterized in that: The reverse iterative update of the spiking neural network is achieved by the following steps: The spiking neural network is reversely iterated and updated using a substitute gradient; wherein the substitute gradient includes: ; Among them, α is the learning rate and x is the loss value.
3. The small target object recognition method based on pulse neural network according to claim 1 is characterized in that: The preprocessed sample object image is obtained by the following steps: Performing data amplification on the sample object image and adjusting the size of the amplified image to obtain a target sample image; The target sample image is normalized to obtain the preprocessed sample image.
4. The small target object recognition method based on pulse neural network according to claim 1 is characterized in that: The two-stream fusion model also includes multiple fully connected layers; The pulse neural network includes at least one first convolution kernel and a plurality of residual blocks, wherein the plurality of residual blocks are determined based on a plurality of second convolution kernels; the activation layers included in each residual block are all set as LIF neurons; The convolution kernel in the residual network is the same as the first convolution kernel in the pulse neural network, the residual block in the residual network is the same as the residual block in the pulse neural network, and the ReLU activation function is used in the residual network to perform data nonlinear operations, and the LIF neuron is used as the activation function in the pulse neural network to perform data nonlinear operations.
5. A small target object recognition device based on a spiking neural network corresponding to the small target object recognition method based on a spiking neural network as claimed in claim 1, characterized in that: include: An image acquisition module, used to acquire an image of the object to be measured; A small target object recognition module is used to classify and recognize the image of the object to be tested based on a dual-stream fusion model to obtain a classification and recognition result; Among them, the dual-stream fusion model is constructed based on a pulse neural network and a residual network, and the dual-stream fusion model is trained by taking sample object images as training samples and fusion features as training features; the fusion features are determined based on the output features of the pulse neural network and the output features of the residual network, and the pulse neural network and the residual network respectively include a plurality of residual blocks arranged in sequence, and the features of the current residual block input of the pulse neural network are the sum of the features output by the previous residual block of the pulse neural network and the features output by the residual block corresponding to the previous residual block in the residual network.
6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the small target object recognition method based on the pulse neural network as described in any one of claims 1 to 4 is implemented.
7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the small target object recognition method based on a pulse neural network as claimed in any one of claims 1 to 4 is implemented.
8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the small target object recognition method based on a pulse neural network as claimed in any one of claims 1 to 4 is implemented.
Citation Information
Patent Citations
Efficient and high-precision training algorithm for spiking neural network
CN114186672A
Classification identification method based on convolutional neural network and multi-modal fusion
CN115438708A
