Pulse-driven target detection methods, devices, electronic equipment, and media
Patent Information
- Application Number
- CN202311758021.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-19
- Publication Date
- 2026-09-01
- Estimated Expiration
- 2043-12-19
AI Technical Summary
[0004]本发明提供一种脉冲驱动的目标检测方法、装置、电子设备及介质，用以解决现有技术中SNN领域中针对小目标检测性能较差的问题
[0039] The pulse-driven target detection method, apparatus, electronic device, and medium provided by this invention can achieve high-precision target detection of images by constructing a target detection network. At the same time, the target detection network is a pure addition operation, which is more in line with the event-driven characteristics of spiking neural networks and greatly reduces the energy consumption of the target detection network.
Smart Images
Figure CN118015325B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to a pulse-driven target detection method, apparatus, electronic device, and medium. Background Technology
[0002] Feature Pyramid Networks (SNNs) exhibit superior performance compared to Artificial Neural Networks (ANNs) in terms of low energy consumption, event-driven characteristics, and biological plasticity. With the development of deep learning, SNNs have also borrowed methods from ANNs, such as gradient backpropagation and residual learning, making it possible to train large-scale SNNs.
[0003] A region-based deep learning object detection algorithm (Mask Region-based Convolutional Neural Network, Mask-RCNN) has demonstrated extremely high performance in object detection and has been widely used in object detection and instance segmentation. However, due to the incompatibility between the purely 0-1 computational nature of spiking neural networks and traditional Mask-RCNN operations, migrating the Mask-RCNN architecture to SNNs is quite difficult. Recently, some research has migrated the YOLO architecture from the object detection field to the SNN field; however, this detection method still suffers from problems such as poor performance in detecting small objects. Summary of the Invention
[0004] This invention provides a pulse-driven target detection method, device, electronic device, and medium to solve the problem of poor performance for small target detection in the field of SNN in the prior art.
[0005] This invention provides a pulse-driven target detection method, comprising:
[0006] Acquire the image to be detected;
[0007] The image to be detected is input into a target detection network for target detection, and the category information of the target object in the image to be detected, the confidence information corresponding to the category information, and the position information of the target object in the image to be detected are determined.
[0008] The target detection network is constructed based on a backbone network, a first spiking neural network module based on a feature pyramid network, a second spiking neural network module based on a region candidate network, and a candidate box regression module.
[0009] The backbone network consists of a first convolution-based spiking neural network block, a second convolution-based spiking neural network block, a third spiking neural network block based on a Transformer structure, and a fourth spiking neural network block based on a Transformer structure, connected sequentially.
[0010] The backbone network is used to extract features from the image to be detected, and obtain first feature maps of different sizes;
[0011] The first spiking neural network module is used to sequentially perform feature fusion and feature extraction on the feature maps of different scales to obtain second feature maps of different sizes;
[0012] The second spiking neural network module is used to determine the candidate box prediction result based on the second feature maps of different sizes;
[0013] The candidate box regression module is used to determine the category information, the confidence information, and the location information based on the candidate box prediction results.
[0014] In some embodiments, the backbone network is connected to the first spiking neural network module through a first LIF neuron, and the first spiking neural network module is connected to the second spiking neural network module through a second LIF neuron.
[0015] The first LIF neuron is used to pulse the first feature maps of different sizes to obtain pulsed first feature maps;
[0016] The second LIF neuron is used to pulse the second feature maps of different sizes to obtain pulsed second feature maps.
[0017] In some embodiments, the first spiking neural network module includes a feature fusion network layer and a feature extraction network layer, the feature fusion network layer and the feature extraction network layer being connected through a third LIF neuron; the third LIF neuron pulses the third feature map to obtain a pulsed third feature map.
[0018] The feature fusion network layer is used to adjust the number of output channels of the backbone network to the same dimension, and to upsample the first feature maps of different sizes in sequence, and to perform feature fusion based on the upsampled first feature maps to obtain the third feature map;
[0019] The feature extraction network layer is used to perform local feature extraction on the pulsed third feature map using 3×3 convolution to obtain the second feature map of different sizes.
[0020] In some embodiments, the feature fusion network layer consists of four pointwise convolutions;
[0021] The four pointwise convolutions are respectively connected to the first spiking neural network block, the second spiking neural network block, the third spiking neural network block, and the fourth spiking neural network block.
[0022] In some embodiments, the second spiking neural network module includes a 3x3 convolution, a 1x1 convolution with 3 channels, and a 1x1 convolution with 12 channels.
[0023] In some embodiments, determining the category of the target object, the confidence information, and the location information based on the candidate box prediction results includes:
[0024] Based on the candidate box prediction results, the target region is pooled and fixed to obtain the fourth feature map;
[0025] The fourth feature map is reduced in dimensionality using a first fully connected layer to obtain a dimensionality-reduced fourth feature map.
[0026] The dimensionality-reduced fourth feature map is input into the second fully connected layer and the third fully connected layer respectively to obtain the category information and confidence information output by the second fully connected layer, and the location information output by the third fully connected layer.
[0027] The present invention also provides a pulse-driven target detection device, comprising:
[0028] The acquisition module is used to acquire the image to be detected;
[0029] The detection module is used to input the image to be detected into the target detection network for target detection, and to determine the category information of the target object in the image to be detected, the confidence information corresponding to the category information, and the position information of the target object in the image to be detected.
[0030] The target detection network is constructed based on a backbone network, a first spiking neural network module based on a feature pyramid network, a second spiking neural network module based on a region candidate network, and a candidate box regression module.
[0031] The backbone network consists of a first convolution-based spiking neural network block, a second convolution-based spiking neural network block, a third spiking neural network block based on a Transformer structure, and a fourth spiking neural network block based on a Transformer structure, connected sequentially.
[0032] The backbone network is used to extract features from the image to be detected, and obtain first feature maps of different sizes;
[0033] The first spiking neural network module is used to sequentially perform feature fusion and feature extraction on the feature maps of different scales to obtain second feature maps of different sizes;
[0034] The second spiking neural network module is used to determine the candidate box prediction result based on the second feature maps of different sizes;
[0035] The candidate box regression module is used to determine the category information, the confidence information, and the location information based on the candidate box prediction results.
[0036] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the pulse-driven target detection method as described above.
[0037] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the pulse-driven target detection method as described above.
[0038] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the pulse-driven target detection method as described above.
[0039] The pulse-driven target detection method, apparatus, electronic device, and medium provided by this invention can achieve high-precision target detection of images by constructing a target detection network. At the same time, the target detection network is a pure addition operation, which is more in line with the event-driven characteristics of spiking neural networks and greatly reduces the energy consumption of the target detection network. Attached Figure Description
[0040] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0041] Figure 1 This is a flowchart illustrating the pulse-driven target detection method provided by the present invention;
[0042] Figure 2 This is a schematic diagram of the overall structure of the target detection network in the pulse-driven target detection method provided by the present invention.
[0043] Figure 3 This is one of the schematic diagrams of the substructure of the target detection network in the pulse-driven target detection method provided by the present invention;
[0044] Figure 4 This is the second schematic diagram of the substructure of the target detection network in the pulse-driven target detection method provided by the present invention;
[0045] Figure 5 This is the third schematic diagram of the substructure of the target detection network in the pulse-driven target detection method provided by the present invention;
[0046] Figure 6 This is the fourth schematic diagram of the substructure of the target detection network in the pulse-driven target detection method provided by the present invention;
[0047] Figure 7 This is a schematic diagram of the structure of the pulse-driven target detection device provided by the present invention;
[0048] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0050] The following is combined with Figures 1-8 This invention describes a pulse-driven target detection method, apparatus, electronic device, and medium.
[0051] The execution subject of the pulse-driven target detection method provided by this invention can be an electronic device, a component in an electronic device, an integrated circuit, or a chip. The electronic device can be a mobile electronic device or a non-mobile electronic device. For example, a mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc., while a non-mobile electronic device can be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This invention does not impose specific limitations.
[0052] The technical solution of the present invention will be described in detail below using a computer executing the pulse-driven target detection method provided by the present invention as an example.
[0053] Figure 1 This is a flowchart illustrating the pulse-driven target detection method provided by the present invention. (Refer to...) Figure 1 The pulse-driven target detection method provided by the present invention includes steps 110 and 120.
[0054] Step 110: Obtain the image to be detected;
[0055] Step 120: Input the image to be detected into the target detection network to perform target detection, and determine the category information of the target object in the image to be detected, the confidence information corresponding to the category information, and the location information of the target object in the image to be detected;
[0056] The object detection network is constructed based on a backbone network, a first spiking neural network module based on a feature pyramid network, a second spiking neural network module based on a region candidate network, and a candidate box regression module.
[0057] The backbone network consists of a first convolution-based spiking neural network block, a second convolution-based spiking neural network block, a third spiking neural network block based on a Transformer structure, and a fourth spiking neural network block based on a Transformer structure, connected sequentially.
[0058] The backbone network is used to extract features from the image to be detected, obtaining first feature maps of different sizes;
[0059] The first spiking neural network module is used to sequentially perform feature fusion and feature extraction on feature maps of different scales to obtain second feature maps of different sizes;
[0060] The second spiking neural network module is used to determine the candidate box prediction results based on second feature maps of different sizes;
[0061] The candidate box regression module is used to determine category information, confidence information, and location information based on the candidate box prediction results.
[0062] Figure 2 This is a schematic diagram of the overall structure of the target detection network in the pulse-driven target detection method provided by this invention. The following is in conjunction with... Figures 3-6 ,right Figure 2 The overall structure of the target detection network in the document will be explained in detail.
[0063] Reference Figures 3-6The target detection network provided by the present invention includes: a backbone network A0, a first spiking neural network module A1 based on a feature pyramid network, a second spiking neural network module A2 based on a region candidate network, and a candidate box regression module A3.
[0064] like Figure 3 As shown, the backbone network A0 is composed of downsampling convolutional layers, a first convolutional spiking neural network block (Conv-based SNN block), another downsampling convolutional layer (Conv-based SNN block), another downsampling convolutional layer (Conv-based SNN block), a second spiking neural network block based on a Transformer structure, convolutional layers (conv), and another second spiking neural network block based on a Transformer structure (Transformer-based SNN block). Both the downsampling convolutional layers and the convolutional layers are used to increase the number of channels.
[0065] The backbone network A0 is used to receive the image to be detected and to extract features from the image at multiple scales, thereby obtaining feature maps with four different numbers of channels and resolutions.
[0066] Let the side length of the input image to be detected be H in the backbone network A0. Then the size of the first feature map a4 output by the first convolution-based first spiking neural network block is H / 4, the size of the first feature map a3 output by the second convolution-based first spiking neural network block is H / 8, the size of the first feature map a2 output by the first Transformer-based second spiking neural network block is H / 16, and the size of the first feature map a1 output by the second Transformer-based second spiking neural network block is H / 16.
[0067] like Figure 4 As shown, the first spiking neural network module A1 based on the Feature Pyramid Network (FPN) includes a feature fusion network layer B1 and a feature extraction network layer B2. The feature fusion network layer B1 includes four FPN Conv1 layers, and the feature extraction network layer B2 includes four FPN Conv2 layers and a downsampling layer.
[0068] The first spiking neural network module A1 is used to extract features from the first feature map output by the backbone network A0 in different dimensions.
[0069] like Figure 5 As shown, the second spiking neural network module A2 based on the Region Proposal Network (RPN) includes a convolutional layer C1, a prediction layer C2, and a post-processing layer C3. The convolutional layer C1 includes five RPN Convs.
[0070] The second spiking neural network module A2 is used to obtain the scores and coordinates of the initial candidate boxes.
[0071] like Figure 6 As shown, the candidate box regression module A3 includes fully connected layers D1, D2, and D3.
[0072] The candidate box regression module A3 is used to further classify and regress the candidate box prediction results to obtain the final category information, confidence information and location information of the target object in the image to be detected.
[0073] It should be noted that the object detection network is a high-performance, pure addition-driven pulse-driven module based on Mask-RCNN, designed for neuromorphic chips. The entire computation process contains no multiplication operations; it is a full addition operation and possesses pulse-driven characteristics.
[0074] The pulse-driven target detection method provided by this invention can achieve high-precision target detection in the image to be detected by constructing a target detection network. At the same time, the target detection network is a pure addition operation, which is more in line with the event-driven characteristics of the spiking neural network and greatly reduces the energy consumption of the target detection network.
[0075] In some embodiments, the backbone network A0 is connected to the first spiking neural network module A1 through a first LIF neuron, and the first spiking neural network module A1 is connected to the second spiking neural network module A2 through a second LIF neuron.
[0076] The first LIF neuron is used to pulse the first feature map of different sizes to obtain the pulsed first feature map;
[0077] The second LIF neuron is used to pulse the second feature maps of different sizes to obtain pulsed second feature maps.
[0078] It should be noted that the LIF (Leaky Integrate and Fire) neuron is a simplified neuron model based on the fundamental assumption that the neuron's membrane potential is stable in the absence of external stimuli.
[0079] Leaky refers to the gradual decrease in voltage when a potential difference exists across the cell membrane (leakage); integrated refers to the integration of the received pulse sequence when an external current is injected into the neuron; and firing refers to the firing of a pulse when the membrane voltage exceeds a set threshold. The LIF neuron model describes the electrical activity of neurons through an integration-leakage mechanism. Specifically, the LIF neuron model consists of two phases: an integration phase and a leakage phase. In the integration phase, the neuron's membrane potential gradually increases due to the influence of external current and membrane potential leakage current. When the membrane potential exceeds a threshold, the neuron enters the leakage phase and generates an action potential. In the leakage phase, the neuron's membrane potential rapidly decreases, undergoes a recovery period, and then returns to a stable baseline potential.
[0080] In this invention, both the first LIF neuron and the second LIF neuron are used to pulse the output feature map.
[0081] In some embodiments, the first spiking neural network module A1 includes a feature fusion network layer B1 and a feature extraction network layer B2, which are connected by a third LIF neuron; the third LIF neuron pulses the third feature map to obtain a pulsed third feature map.
[0082] The feature fusion network layer B1 is used to adjust the number of output channels of the backbone network A0 to the same dimension, and to upsample the first feature maps of different sizes in sequence. Based on the upsampled first feature maps, feature fusion is performed to obtain the third feature map.
[0083] The feature extraction network layer B2 is used to extract local features from the pulsed third feature map using 3×3 convolutions, resulting in second feature maps of different sizes.
[0084] like Figure 4 As shown, the feature extraction process of the first spiking neural network module A1 is divided into two parts.
[0085] The first part is the feature fusion network layer B1, which extracts features from the backbone network A0 with different numbers of channels and resolutions, and adjusts the number of channels to the same dimension to facilitate further feature fusion.
[0086] The first feature maps a1, a2, a3, and a4 obtained from the backbone network A0 have sizes of H / 16, H / 16, H / 8, and H / 4, respectively. Starting with the first feature map (a1) with the highest downsampling degree, each feature map is upsampled to twice the size of its original feature map and added to the upper-layer feature maps to form new third feature maps a1', a2', a3', and a4'.
[0087] For example: the third feature map a2' is obtained by upsampling the first feature map a1 and adding it to the first feature map a2; the third feature map a3' is obtained by upsampling the third feature map a2' and adding it to the first feature map a3; the third feature map a4' is obtained by upsampling the third feature map a3' and adding it to the first feature map a4. It should be noted that the third feature map a1' is obtained by directly inputting the first feature map a1 into the feature fusion network layer B1.
[0088] The second part involves the feature extraction network layer B2, which performs local feature extraction using 3x3 convolutions on each feature map obtained from the feature fusion network layer B1. For the third feature map (a1') with the highest downsampling degree, a second 2x2 max pooling is performed to generate a new feature map of size H / 32. As described above, the first spiking neural network module A1 outputs five sets of second feature maps with sizes of H / 32, H / 16, H / 16, H / 8, and H / 4, respectively. These second feature maps will be used to predict targets of different sizes.
[0089] In some embodiments, the second spiking neural network module A2 includes a 3x3 convolution, a 1x1 convolution with 3 channels, and a 1x1 convolution with 12 channels.
[0090] like Figure 5 As shown, the prediction process of the second spiking neural network module A2 is divided into three parts.
[0091] The first part involves convolutional layer C1 performing 3x3 convolutions again on each group of second feature maps output by feature extraction network layer B2, while maintaining the original number of channels, to further extract features.
[0092] The second part involves prediction layer C2 applying a 1x1 convolution with 3 output channels to each feature map output by convolutional layer C1, obtaining a preliminary category prediction score for the candidate object. Then, it applies a 1x1 convolution with 12 output channels to each feature map output by convolutional layer C1, obtaining the bounding box position prediction result for the candidate object. Prediction layer C2 corresponds to... Figure 5 The candidate box module in the middle.
[0093] The third part, post-processing layer C3, reshapes the initial category prediction scores of each group into one-dimensional vectors and passes them through a sigmoid layer to obtain the prediction score. Based on this score, 1000 candidate boxes are retained for each group. The results of the five groups are fused to obtain a total of 5000 candidate boxes. Non-maximum suppression is applied to these 5000 candidate boxes, retaining 1000 candidate boxes. Post-processing layer C3 corresponds to... Figure 5 The Non-Maximum Suppression (NMS) module in the code.
[0094] In some embodiments, the feature fusion network layer B1 consists of four pointwise convolutions;
[0095] The four pointwise convolutions are connected to the first spiking neural network block, the second spiking neural network block, the third spiking neural network block, and the fourth spiking neural network block, respectively.
[0096] In actual implementation, the first spiking neural network block (Conv-based SNN block) in the backbone network A0 is connected to the first pointwise convolution (FPN Conv1) of the feature fusion network layer B1 through the first LIF neuron;
[0097] The second spiking neural network block (Conv-based SNN block) in the backbone network A0 is connected to the second pointwise convolution (FPN Conv1) of the feature fusion network layer B1 through the first LIF neuron;
[0098] The third spiking neural network block (Transformer-based SNN block) in the backbone network A0 is connected to the third pointwise convolution (FPN Conv1) of the feature fusion network layer B1 through the first LIF neuron;
[0099] The fourth spiking neural network block (Transformer-based SNN block) in the backbone network A0 is connected to the fourth pointwise convolution (FPN Conv1) of the feature fusion network layer B1 through the first LIF neuron.
[0100] The first pointwise convolution (FPN Conv1) of the feature fusion network layer B1 is connected to the first convolutional layer (FPN Conv2) in the feature extraction network layer B2 through the third LIF neuron;
[0101] The second pointwise convolution (FPN Conv1) of the feature fusion network layer B1 is connected to the second convolutional layer (FPN Conv2) in the feature extraction network layer B2 through the third LIF neuron;
[0102] The third pointwise convolution (FPN Conv1) of the feature fusion network layer B1 is connected to the third convolutional layer (FPN Conv2) in the feature extraction network layer B2 through the third LIF neuron;
[0103] The fourth pointwise convolution (FPN Conv1) of the feature fusion network layer B1 is connected to the fourth convolutional layer (FPN Conv2) in the feature extraction network layer B2 through the third LIF neuron;
[0104] Each convolutional layer in the feature extraction network layer B2 consists of a 3x3 convolution, a 1x1 convolution with 3 channels, and a 1x1 convolution with 12 channels.
[0105] The first convolutional layer (FPN Conv2) in the feature extraction network layer B2 is connected to the first convolutional layer (RPN Conv) in the second spiking neural network module A2 through the second LIF neuron;
[0106] The second convolutional layer (FPN Conv2) in the feature extraction network layer B2 is connected to the second convolutional layer (RPN Conv) in the second spiking neural network module A2 through the second LIF neuron;
[0107] The third convolutional layer (FPN Conv2) in the feature extraction network layer B2 is connected to the third convolutional layer (RPN Conv) in the second spiking neural network module A2 through the second LIF neuron;
[0108] The fourth convolutional layer (FPN Conv2) in the feature extraction network layer B2 is connected to the fourth convolutional layer (RPN Conv) in the second spiking neural network module A2 through the second LIF neuron;
[0109] The fourth convolutional layer (FPN Conv2) in the feature extraction network layer B2 also passes through a downsampling layer and a second LIF neuron, and is connected to the fifth convolutional layer (RPNConv) in the second spiking neural network module A2.
[0110] In some embodiments, determining the category, confidence information, and location information of the target object based on the candidate box prediction results includes:
[0111] Based on the candidate box prediction results, the target region is pooled and fixed to obtain the fourth feature map;
[0112] The fourth feature map is reduced in dimensionality using the first fully connected layer to obtain the dimensionality-reduced fourth feature map.
[0113] The dimensionality-reduced fourth feature map is input into the second fully connected layer and the third fully connected layer respectively to obtain the category information and confidence information output by the second fully connected layer, and the position information output by the third fully connected layer.
[0114] like Figure 6 As shown, the regression prediction process of the candidate box regression module A3 is divided into three parts.
[0115] Module D1 is based on the candidate box prediction results obtained from post-processing layer C3. For the four sets of features with lower downsampling degree among the five sets of features obtained from convolutional layer C1, the ROIAlign method is used to pool and fix the corresponding regions to obtain the fourth feature map with shape [1000, C, 7, 7].
[0116] Module D2 is the fourth feature map obtained from module D1, which is reshaped into the form of [1000, Cx7x7] and the dimension is reduced to 1024 by using a fully connected layer.
[0117] Module D3 utilizes the 1024-dimensional features obtained from Module D2, employing two fully connected layers. The first fully connected layer obtains the target object's category and confidence information, while the second fully connected layer obtains the target bounding box information, i.e., the target object's location information.
[0118] In practical implementation, this invention uses Meta-Spikeformer as the backbone network, and performs object detection performance tests on the COCO dataset for models with two parameter ranges: 34.9M and 75.0M. The 75.0M model achieved a map size of 0.512 only at a time step of 1, surpassing all other SNN models. Specific results are shown in Table 1.
[0119] Table 1
[0120]
[0121] Where Methods represents the method, Architecture represents the architecture, Spike represents the spike drive, Param represents the number of parameters, Power represents the energy consumption, Step represents the time step, and mAP@0.5 (Mean Average Precision) represents the average prediction accuracy when the Intersection over Union (IoU) threshold is 0.5.
[0122] Based on the experimental results recorded in the table above, it can be seen that the pulse-driven target detection module based on Mask-RCNN used in this invention currently has the best performance. However, when designing pulse neuromorphic chips in the future, the chip architecture can be designed reasonably according to the importance of energy consumption and performance in specific scenarios.
[0123] The pulse-driven target detection device provided by the present invention is described below. The pulse-driven target detection device described below and the pulse-driven target detection method described above can be referred to in correspondence.
[0124] Figure 7This is a schematic diagram of the structure of the pulse-driven target detection device provided by the present invention. (Refer to...) Figure 7 The pulse-driven target detection device provided by the present invention includes:
[0125] The acquisition module 710 is used to acquire the image to be detected;
[0126] Detection module 720 is used to input the image to be detected into a target detection network for target detection, and to determine the category information of the target object in the image to be detected, the confidence information corresponding to the category information, and the position information of the target object in the image to be detected.
[0127] The target detection network is constructed based on a backbone network, a first spiking neural network module based on a feature pyramid network, a second spiking neural network module based on a region candidate network, and a candidate box regression module.
[0128] The backbone network consists of a first convolution-based spiking neural network block, a second convolution-based spiking neural network block, a third spiking neural network block based on a Transformer structure, and a fourth spiking neural network block based on a Transformer structure, connected sequentially.
[0129] The backbone network is used to extract features from the image to be detected, and obtain first feature maps of different sizes;
[0130] The first spiking neural network module is used to sequentially perform feature fusion and feature extraction on the feature maps of different scales to obtain second feature maps of different sizes;
[0131] The second spiking neural network module is used to determine the candidate box prediction result based on the second feature maps of different sizes;
[0132] The candidate box regression module is used to determine the category information, the confidence information, and the location information based on the candidate box prediction results.
[0133] The pulse-driven target detection device provided by this invention can achieve high-precision target detection in the image to be detected by constructing a target detection network. At the same time, the target detection network is a pure addition operation, which is more in line with the event-driven characteristics of the spiking neural network and greatly reduces the energy consumption of the target detection network.
[0134] In some embodiments, the backbone network is connected to the first spiking neural network module through a first LIF neuron, and the first spiking neural network module is connected to the second spiking neural network module through a second LIF neuron.
[0135] The first LIF neuron is used to pulse the first feature maps of different sizes to obtain pulsed first feature maps;
[0136] The second LIF neuron is used to pulse the second feature maps of different sizes to obtain pulsed second feature maps.
[0137] In some embodiments, the first spiking neural network module includes a feature fusion network layer and a feature extraction network layer, the feature fusion network layer and the feature extraction network layer being connected through a third LIF neuron; the third LIF neuron pulses the third feature map to obtain a pulsed third feature map.
[0138] The feature fusion network layer is used to adjust the number of output channels of the backbone network to the same dimension, and to upsample the first feature maps of different sizes in sequence, and to perform feature fusion based on the upsampled first feature maps to obtain the third feature map;
[0139] The feature extraction network layer is used to perform local feature extraction on the pulsed third feature map using 3×3 convolution to obtain the second feature map of different sizes.
[0140] In some embodiments, the feature fusion network layer consists of four pointwise convolutions;
[0141] The four pointwise convolutions are respectively connected to the first spiking neural network block, the second spiking neural network block, the third spiking neural network block, and the fourth spiking neural network block.
[0142] In some embodiments, the second spiking neural network module includes a 3x3 convolution, a 1x1 convolution with 3 channels, and a 1x1 convolution with 12 channels.
[0143] In some embodiments, the detection module 720 is specifically used for:
[0144] Based on the candidate box prediction results, the target region is pooled and fixed to obtain the fourth feature map;
[0145] The fourth feature map is reduced in dimensionality using a first fully connected layer to obtain a dimensionality-reduced fourth feature map.
[0146] The dimensionality-reduced fourth feature map is input into the second fully connected layer and the third fully connected layer respectively to obtain the category information and confidence information output by the second fully connected layer, and the location information output by the third fully connected layer.
[0147] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a pulse-driven target detection method, which includes:
[0148] Acquire the image to be detected;
[0149] The image to be detected is input into a target detection network for target detection, and the category information of the target object in the image to be detected, the confidence information corresponding to the category information, and the position information of the target object in the image to be detected are determined.
[0150] The target detection network is constructed based on a backbone network, a first spiking neural network module based on a feature pyramid network, a second spiking neural network module based on a region candidate network, and a candidate box regression module.
[0151] The backbone network consists of a first convolution-based spiking neural network block, a second convolution-based spiking neural network block, a third spiking neural network block based on a Transformer structure, and a fourth spiking neural network block based on a Transformer structure, connected sequentially.
[0152] The backbone network is used to extract features from the image to be detected, and obtain first feature maps of different sizes;
[0153] The first spiking neural network module is used to sequentially perform feature fusion and feature extraction on the feature maps of different scales to obtain second feature maps of different sizes;
[0154] The second spiking neural network module is used to determine the candidate box prediction result based on the second feature maps of different sizes;
[0155] The candidate box regression module is used to determine the category information, the confidence information, and the location information based on the candidate box prediction results.
[0156] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0157] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the pulse-driven target detection method provided by the above methods, the method comprising:
[0158] Acquire the image to be detected;
[0159] The image to be detected is input into a target detection network for target detection, and the category information of the target object in the image to be detected, the confidence information corresponding to the category information, and the position information of the target object in the image to be detected are determined.
[0160] The target detection network is constructed based on a backbone network, a first spiking neural network module based on a feature pyramid network, a second spiking neural network module based on a region candidate network, and a candidate box regression module.
[0161] The backbone network consists of a first convolution-based spiking neural network block, a second convolution-based spiking neural network block, a third spiking neural network block based on a Transformer structure, and a fourth spiking neural network block based on a Transformer structure, connected sequentially.
[0162] The backbone network is used to extract features from the image to be detected, and obtain first feature maps of different sizes;
[0163] The first spiking neural network module is used to sequentially perform feature fusion and feature extraction on the feature maps of different scales to obtain second feature maps of different sizes;
[0164] The second spiking neural network module is used to determine the candidate box prediction result based on the second feature maps of different sizes;
[0165] The candidate box regression module is used to determine the category information, the confidence information, and the location information based on the candidate box prediction results.
[0166] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the pulse-driven target detection method provided by the methods described above, the method comprising:
[0167] Acquire the image to be detected;
[0168] The image to be detected is input into a target detection network for target detection, and the category information of the target object in the image to be detected, the confidence information corresponding to the category information, and the position information of the target object in the image to be detected are determined.
[0169] The target detection network is constructed based on a backbone network, a first spiking neural network module based on a feature pyramid network, a second spiking neural network module based on a region candidate network, and a candidate box regression module.
[0170] The backbone network consists of a first convolution-based spiking neural network block, a second convolution-based spiking neural network block, a third spiking neural network block based on a Transformer structure, and a fourth spiking neural network block based on a Transformer structure, connected sequentially.
[0171] The backbone network is used to extract features from the image to be detected, and obtain first feature maps of different sizes;
[0172] The first spiking neural network module is used to sequentially perform feature fusion and feature extraction on the feature maps of different scales to obtain second feature maps of different sizes;
[0173] The second spiking neural network module is used to determine the candidate box prediction result based on the second feature maps of different sizes;
[0174] The candidate box regression module is used to determine the category information, the confidence information, and the location information based on the candidate box prediction results.
[0175] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0176] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0177] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A pulse-driven target detection method, characterized in that, include: Acquire the image to be detected; The image to be detected is input into a target detection network for target detection, and the category information of the target object in the image to be detected, the confidence information corresponding to the category information, and the position information of the target object in the image to be detected are determined. The target detection network is constructed based on a backbone network, a first spiking neural network module based on a feature pyramid network, a second spiking neural network module based on a region candidate network, and a candidate box regression module. The backbone network consists of a first convolution-based spiking neural network block, a second convolution-based spiking neural network block, a third spiking neural network block based on a Transformer structure, and a fourth spiking neural network block based on a Transformer structure, connected sequentially. The backbone network is used to extract features from the image to be detected, and obtain first feature maps of different sizes; The first spiking neural network module is used to sequentially perform feature fusion and feature extraction on the feature maps of different sizes to obtain second feature maps of different sizes; The second spiking neural network module is used to determine the candidate box prediction result based on the second feature maps of different sizes; The candidate box regression module is used to determine the category information, the confidence information, and the location information based on the candidate box prediction results. The first spiking neural network module includes a feature fusion network layer and a feature extraction network layer, which are connected by a third LIF neuron; the third LIF neuron pulses the third feature map to obtain a pulsed third feature map. The feature fusion network layer is used to adjust the number of output channels of the backbone network to the same dimension, and to upsample the first feature maps of different sizes in sequence, and to perform feature fusion based on the upsampled first feature maps to obtain the third feature map; The feature extraction network layer is used to perform local feature extraction on the pulsed third feature map using 3×3 convolution to obtain the second feature maps of different sizes; The feature fusion network layer consists of four FPN Conv1 layers, and the feature extraction network layer consists of four FPN Conv2 layers and one downsampling layer.
2. The pulse-driven target detection method according to claim 1, characterized in that, The backbone network is connected to the first spiking neural network module through a first LIF neuron, and the first spiking neural network module is connected to the second spiking neural network module through a second LIF neuron. The first LIF neuron is used to pulse the first feature maps of different sizes to obtain pulsed first feature maps; The second LIF neuron is used to pulse the second feature maps of different sizes to obtain pulsed second feature maps.
3. The pulse-driven target detection method according to claim 1, characterized in that, The feature fusion network layer consists of four pointwise convolutions; The four pointwise convolutions are respectively connected to the first spiking neural network block, the second spiking neural network block, the third spiking neural network block, and the fourth spiking neural network block.
4. The pulse-driven target detection method according to claim 1, characterized in that, The second spiking neural network module includes a 3x3 convolution, a 1x1 convolution with 3 channels, and a 1x1 convolution with 12 channels.
5. The pulse-driven target detection method according to claim 1, characterized in that, The step of determining the category of the target object, the confidence information, and the location information based on the candidate box prediction results includes: Based on the candidate box prediction results, the target region is pooled and fixed to obtain the fourth feature map; The fourth feature map is reduced in dimensionality using a first fully connected layer to obtain a dimensionality-reduced fourth feature map. The dimensionality-reduced fourth feature map is input into the second fully connected layer and the third fully connected layer respectively to obtain the category information and confidence information output by the second fully connected layer, and the location information output by the third fully connected layer.
6. A pulse-driven target detection device, characterized in that, include: The acquisition module is used to acquire the image to be detected; The detection module is used to input the image to be detected into the target detection network for target detection, and to determine the category information of the target object in the image to be detected, the confidence information corresponding to the category information, and the position information of the target object in the image to be detected. The target detection network is constructed based on a backbone network, a first spiking neural network module based on a feature pyramid network, a second spiking neural network module based on a region candidate network, and a candidate box regression module. The backbone network consists of a first convolution-based spiking neural network block, a second convolution-based spiking neural network block, a third spiking neural network block based on a Transformer structure, and a fourth spiking neural network block based on a Transformer structure, connected sequentially. The backbone network is used to extract features from the image to be detected, and obtain first feature maps of different sizes; The first spiking neural network module is used to sequentially perform feature fusion and feature extraction on the feature maps of different sizes to obtain second feature maps of different sizes; The second spiking neural network module is used to determine the candidate box prediction result based on the second feature maps of different sizes; The candidate box regression module is used to determine the category information, the confidence information, and the location information based on the candidate box prediction results. The first spiking neural network module includes a feature fusion network layer and a feature extraction network layer, which are connected by a third LIF neuron; the third LIF neuron pulses the third feature map to obtain a pulsed third feature map. The feature fusion network layer is used to adjust the number of output channels of the backbone network to the same dimension, and to upsample the first feature maps of different sizes in sequence, and to perform feature fusion based on the upsampled first feature maps to obtain the third feature map; The feature extraction network layer is used to perform local feature extraction on the pulsed third feature map using 3×3 convolution to obtain the second feature maps of different sizes; The feature fusion network layer consists of four FPN Conv1 layers, and the feature extraction network layer consists of four FPN Conv2 layers and one downsampling layer.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the pulse-driven target detection method as described in any one of claims 1 to 5.
8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the pulse-driven target detection method as described in any one of claims 1 to 5.
9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the pulse-driven target detection method as described in any one of claims 1 to 5.