Edge-end cooperative target detection method based on spiking neural network

Abstract

The invention relates to an edge-end collaborative target detection method based on a spiking neural network, and belongs to the technical field of artificial intelligence, big data and computer vision. The invention provides an improved SpikeYOLO model aiming at the problems of high energy consumption, large computing resource consumption and low edge-end cooperation efficiency when end-side equipment performs target detection, and on the basis of an original target detection model, spiking neurons are used for replacing artificial neurons, so that the parameter quantity of the model and the operation energy consumption of the model are remarkably reduced. Besides, aiming at the problem of performance reduction possibly caused when complex calculation of a YOLO module is directly converted into SNN, a reverse residual structure is introduced to optimize model design and ensure that key features are not lost in a transmission process, so that the recognition effect of an end side equipment model is within an acceptable precision attenuation range.

Description

Technical Field

[0001] The present invention belongs to the fields of artificial intelligence, big data and computer vision technology, and specifically relates to an edge collaborative target detection method based on a pulse neural network. Background Art

[0002] With the development of edge-side unmanned technology, unmanned equipment such as drones and electronic dogs are increasingly being used in modern military operations. These unmanned devices mainly rely on target detection technology to achieve effective perception and analysis of the surrounding environment, forming the technical basis for element recognition, unmanned system operation, and unmanned equipment application in complex geographical environments. Especially in the field of military operations and intelligence collection, drones can detect key targets in battlefield images and use edge intelligence models to identify important military targets in complex battlefield environments, providing key information for situation analysis and decision support.

[0003] Although the application of edge devices has made significant progress, it still faces a series of challenges in actual operation, including high consumption of computing resources and difficulty in edge-end collaboration. Traditional deep learning-based object detection models, such as the YOLO series, usually require a lot of computing resources and power supply to achieve relatively satisfactory detection results. However, edge devices are often affected by latency, unstable network connections, and bandwidth limitations, which greatly limit the actual effectiveness of object detection technology. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] The present invention proposes an end-side intelligent target detection method, which mainly solves two problems. First, it aims to reduce the number of model parameters and model operation energy consumption in order to solve the problems of high energy consumption, large consumption of computing resources and low efficiency of edge-end collaboration when the end-side device performs target detection. Second, it aims to solve the problem that the performance may be degraded when the complex calculation of the YOLO module is directly converted to SNN, ensuring that the key features will not be lost during the transmission process, so that the recognition effect of the end-side device model is within an acceptable accuracy attenuation range.

[0006] (II) Technical solution

[0007] In order to solve the above technical problems, the present invention provides.

[0008] (III) Beneficial effects

[0009] The present invention provides a method for edge collaborative target detection based on a spiking neural network, the main advantages of which are reflected in the following aspects:

[0010] (1) An integer leaky integrate and fire (I-LIF) neuron is designed to replace artificial neurons. It uses discrete pulse signals as the input of the pulse neuron. The pulse neuron will only emit discrete pulses when the accumulated input signal reaches or exceeds the set threshold, which greatly reduces the consumption of storage and computing resources. It is particularly suitable for resource-constrained edge devices.

[0011] (2) A feature extraction module that combines the YOLOv8 classic architecture with the metaSNN block strategy in Meta-SpikeFormer is designed. It can promote efficient feature extraction and fusion, while ensuring an acceptable precision loss range. At the same time, it effectively reduces the number of model parameters and energy consumption. Compared with YOLOv8, it reduces redundant calculations and improves the inference speed of the model.

[0012] (3) We designed an inverse residual structure to optimize the internal feature transfer of the model, which minimized the loss of key feature information during the transfer process. This effectively solved the problem that the complex YOLO module is difficult to train when converted to SNN. By combining shallow information with deep information, we narrowed the performance gap between ANN and SNN in target detection tasks, thereby improving the target detection accuracy and stability of SpikeYOLO in complex scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 This is a model structure diagram of the present invention. DETAILED DESCRIPTION

[0014] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with the drawings and examples.

[0015] The present invention proposes a lightweight, low-energy target detection method based on a spiking neural network (Spiking Neural Network, SNN), which aims to significantly reduce the demand for computing resources and energy consumption without sacrificing or only slightly losing accuracy, and can show lower energy consumption and higher energy efficiency when processing dynamic information. The present invention proposes a terminal-side intelligent target detection method that integrates a spiking neural network. Specifically, it mainly solves two problems. First, in view of the high energy consumption, large computing resource consumption and low efficiency of edge-end collaboration when the terminal device performs target detection, an improved SpikeYOLO model is proposed. On the basis of the original target detection model, spiking neurons are used to replace artificial neurons, which significantly reduces the number of model parameters and model operation energy consumption. Second, in view of the problem that the performance may be degraded when the complex calculation of the YOLO module is directly converted to SNN, a reverse residual structure is introduced to optimize the model design to ensure that the key features will not be lost during the transmission process, so that the recognition effect of the terminal device model is within an acceptable accuracy attenuation range.

[0016] The overall framework of SpikeYOLO is as follows Figure 1 As shown in the figure, in general, SpikeYOLO consists of three different stages, namely the spike neuron design stage, network input, macro and micro design stages. In the first stage, this method introduces I-LIF neurons to replace the original artificial neurons, retaining the ability of LIF neurons to efficiently propagate information in the spatiotemporal domain. By allowing neurons to emit integer-valued pulses during the training stage, the representation limitations brought by traditional binary pulse outputs are overcome. I-LIF neurons can provide deeper feedback during the training process, improve the representation ability of the model, and ensure the low power consumption characteristics of the model during actual deployment. In the second stage, this method uses static images to repeat and use them as input for each time step, so that static images can utilize the spatiotemporal processing capabilities of SNN, and introduce neuromorphic preprocessing processes for optimization, making full use of the pulse-driven characteristics and their advantages in event stream processing, including low resource requirements, high temporal resolution, and high robustness, thereby improving the overall performance of the model. Finally, this method constructs a feature extraction scheme suitable for SNN by combining the macro design of YOLOv8 with the micro design of Meta-SpikeFormer. First, the classic architecture of YOLOv8, such as feature pyramid network and fast spatial pyramid pooling, is retained to promote efficient feature extraction and fusion. At the same time, in order to solve the spike degradation problem caused by complex calculations in the YOLO module when directly converted to the SNN version, the reverse residual structure is introduced to optimize the feature transfer inside the model, reduce information loss, and improve model performance.

[0017] Given a static image input I, where I∈R C×H×W, whose dimensions are the number of channels C, height H and width W. The goal of the SpikeYOLO task of the present invention is to perform target detection on the input image sequence I so that the model can accurately identify and locate each target in the image. The specific steps are described as follows:

[0018] Step 1: Spiking Neuron Design

[0019] Spiking neurons propagate information in the spatial and temporal domains by mimicking the way biological neurons transmit information. This process can be described as converting the membrane potential of spiking neurons into binary pulses. However, this method has inherent quantization errors, which severely limits the representation of the model. The present invention uses I-LIF neurons, spiking neurons, to replace the artificial neurons of the existing SpikeYOLO model to achieve the purpose of reducing energy consumption, thereby improving model performance.

[0020] LIF neurons are the most commonly used model for building spiking neural networks. They have achieved a good balance between biological complexity and computational rationality. However, in some specific tasks, the simple structure of LIF neurons does not seem to be competent. The neural dynamics of LIF can be described as:

[0021]

[0022] Where t is the time step, U[t] is the membrane potential, which integrates the temporal information H[t-1] and the spatial information X[t]. Θ(·) is a step function that takes the value 1 when x ≥ 0 and takes the value 0 otherwise. If U[t] exceeds the firing threshold V th , the spiking neuron will fire a pulse S[t], and U[t] will decrease accordingly. Otherwise, H[t] will remain unchanged. In addition, U[t] will decay to H[t] by a factor of the decay constant β, where β represents the decay constant. For simplicity, we focus on the second formula and represent the spiking neuron layer as SN(·), whose input is the membrane potential tensor U and output is the pulse tensor S. Obviously, this binary output is difficult to cope with fine control and learning. I-LIF (Integer Leaky Integrate-and-Fire) neurons emit integer values ​​during training (I-LIF neurons simulate the discharge behavior of neurons. When the potential reaches a certain threshold, I-LIF neurons will emit a spike. The integer value refers to the signal output by the I-LIF neuron each time the spike discharges), and convert the integer value into a 0 / 1 pulse during inference. Specifically, in I-LIF, the above formula can be rewritten as:

[0023] S[t]=Clip(round(U[t]),0,D) (2)

[0024] Where round(·) is the integer symbol, Clip(x,min,max) means truncating x to the range [min,max], and D is a hyperparameter, which represents the maximum integer value emitted by I-LIF. During the training phase, the above formula (2) is not a continuous function, which makes its derivative a step function, which may lead to unstable training. Previous studies have proposed several alternative gradient functions, which are mainly used to process binary pulse outputs. The present invention always uses a rectangular window as an alternative function. For simplicity, only the gradients of neurons activated in the range [0,D] are retained, and all other neurons are ignored.

[0025] Spiking neurons process and transmit input signals in the inference stage, where the introduction of integer values ​​requires additional multiplication-addition operations (MACs), which may reduce the energy efficiency of SNNs. Therefore, it is necessary to convert integer values ​​into binary spikes. Specifically, the input to the spiking neuron in the l+1th layer can be represented as S l [t], which is a pulse sequence. The present invention expands the time step T to T×D and converts the integer value S l [t,d] is converted into a pulse sequence S l [t], satisfies:

[0026]

[0027] Therefore, the output of the spiking neuron in the l+1th layer is:

[0028]

[0029] Since the matrix multiplication function is a linear operator, we can get:

[0030]

[0031] Thus, the pulse sequence S l [t] contains only 0s and 1s, so all MAC operations can be converted into sparse accumulate operations (ACs), which ensures pulse driving during inference.

[0032] Step 2: Design an improved SpikeYOLO model based on the spiking neuron, and perform model inference to obtain target detection results

[0033] The improved SpikeYOLO model designed by the present invention includes an input layer, an SNN convolution layer, a downsampling layer, an SNN module and an output layer. The input layer is implemented by the pulse neuron, the input is image data, and the output is an encoded pulse signal; the SNN convolution layer extracts spatial features and temporal features, the input is a pulse signal from the previous layer, and the output is a feature map processed by convolution; the downsampling layer reduces the amount of data by pooling operation while maintaining significant features, the input is the feature map output by the SNN convolution layer, and the output is a feature map with lower resolution; the SNN module generates preliminary prediction results based on feature information in the feature map, the input is the feature map from the downsampling layer, and the output is the preliminary prediction results of the detection box and category; the output layer converts the pulse signal into conventional detection box information, the input is the prediction result from the SNN module, and the output is a bounding box, target category label and confidence score, and the bounding box represents the detected target position.

[0034] The input of the improved SpikeYOLO model (SNNs network) of the present invention can be expressed as I∈R T×C×H×W , where T represents the time step, C represents the number of channels, and H×W represents the spatial resolution. For the input of static images, in order to utilize the spatiotemporal capabilities of SNNs, it is common practice to repeat the static image and use it as the input for each time step T. This approach is called direct input encoding, in which the spiking neurons in the input layer of the SNNs network encode the continuous values ​​of the input static image into a pulse signal, and the spiking neurons are the spiking neurons designed in step 1. For the input of neuromorphic data, that is, data generated by a dynamic vision sensor (DVS), a pulse is generated only when the logarithmic change of the light intensity at the pixel exceeds a preset threshold. The event-based stream can be expressed as (xn,yn,tn,pn), capturing the spatial coordinates (x,y), timestamp t, and polarity p for each event, where p∈{-1,1} indicates whether the light intensity increases or decreases. Neuromorphic vision has many advantages, such as low resource requirements, high temporal resolution, and high robustness. The pulse-driven nature of SNNs makes them naturally suitable for processing event streams. The general strategy of neuromorphic preprocessing is to aggregate the event stream into a frame format within a fixed time window. The present invention follows this operation. Specifically, the total input window length is T×dt, where dt and T are the time resolution and time step, respectively. In this step, the improved SpikeYOLO model receives the input image and encodes it into a spike signal.

[0035] The improved SpikeYOLO model proposed in the present invention combines the macro design of YOLOv8 with the micro design of Meta-SpikeFormer. The present invention observes that the complex calculations in the YOLO module lead to spike degradation when directly converted to the SNN version, resulting in excessive performance loss. Therefore, while maintaining the overall design principles of the YOLO architecture, the present invention combines the reverse residual structure to achieve performance closeness.

[0036] In object detection, the network is based on the input image sequence Output the category and location of each object. Assume that the input image sequence has N targets and the output It can be calculated as:

[0037] B=Model(X) (6)

[0038] Each of the B n Contains information about confidence f, category c, center coordinates (x, y) and target size (w, h). Model(·) refers to the improved SpikeYOLO model proposed in this invention.

[0039] SpikeYOLO is a variant of the YOLO framework and is more suitable for feature extraction schemes of SNNs. Specifically, YOLOv8 is a classic single-stage detection framework that divides the image into multiple grids, each of which is independently responsible for predicting an object. Some classic designs, such as the feature pyramid network in YOLOv8, play a vital role in promoting efficient feature extraction and fusion. In contrast, its feature extraction modules, such as C2F, perform the operation of extracting features from the same set of feature maps multiple times. This kind of module can enhance feature extraction in ANNs, but it does not work well in SNNs. As a compromise, the present invention retains the classic architecture in YOLOv8 while combining the strategy of the metaSNN block in Meta-SpikeFormer. The structure is as follows Figure 1 shown.

[0040] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for edge collaborative target detection based on a spiking neural network, characterized in that: Given a static image input I, where I∈R C×H×W , whose dimensions are the number of channels C, height H and width W. The goal of this method is to design a SpikeYOLO model to detect targets in the input image sequence I, identify and locate each target in the image. The specific steps are as follows: Step 1: Spiking Neuron Design Spiking neurons propagate information in the spatial and temporal domains by mimicking the way biological neurons transmit information. I-LIF neurons, a type of spiking neuron, are used as artificial neurons in the SpikeYOLO model. Step 2: Based on the pulse neuron design, a SpikeYOLO model is obtained, and model inference is performed to obtain target detection results.

2. The method according to claim 1, characterized in that In step 1, the neural dynamics of LIF are described as: Where t is the time step, U[t] is the membrane potential, which integrates the temporal information H[t-1] and the spatial information X[t]; Θ(·) is a step function that takes the value 1 when x ≥ 0 and takes the value 0 otherwise; if U[t] exceeds the firing threshold V th , the spiking neuron will emit a pulse S[t], and U[t] will decrease accordingly; otherwise, H[t] will remain unchanged; in addition, U[t] decays to H[t] by a factor of the decay constant β, where β represents the decay constant; I-LIF neurons emit integer values ​​during training. I-LIF neurons simulate discharge behavior. When the potential reaches a certain threshold, I-LIF neurons emit a spike. The integer value is the signal output by the I-LIF neuron each time the spike discharges. I-LIF neurons are used to process and transmit input signals in the inference stage. In the inference stage, I-LIF neurons convert the integer value into a binary pulse of 0 / 1. In I-LIF, the formula for S[t] is rewritten as: S[t]=Clip(round(U[t]),0,D) (2) Wherein round(·) is the integer symbol, Clip(x,min,max) means clipping x to the range [min,max], and D is a hyperparameter, which represents the maximum integer value emitted by I-LIF. In the training stage, the present invention always uses a rectangular window as a substitute function of formula (2). In addition, only the gradients of neurons activated in the range [0,D] are retained, and all other neurons are ignored.

3. The method according to claim 2, characterized in that In step 1, the method used by I-LIF neurons to convert integer values ​​into binary pulses in the inference phase is as follows: the pulse neuron input to the l+1th layer is represented by S l [t], which is a pulse train that expands the time step T to T×D and replaces the integer value S l [t,d] is converted into a pulse sequence S l [t], satisfies: Therefore, the output of the spiking neuron in the l+1th layer is: The matrix multiplication function is a linear operator, and we get: Thus, the pulse sequence S l [t] contains only 0 and 1.

4. The method according to claim 1, characterized in that The SpikeYOLO model designed in step 2 is as follows: it includes an input layer, an SNN convolutional layer, a downsampling layer, an SNN module and an output layer; the input layer is implemented using the pulse neuron, the input is image data, and the output is an encoded pulse signal; the SNN convolutional layer extracts spatial features and temporal features, the input is the pulse signal from the previous layer, and the output is a feature map processed by convolution; the downsampling layer reduces the amount of data through pooling operations while maintaining significant features, the input is the feature map output by the SNN convolutional layer, and the output is a feature map with a lower resolution; the SNN module generates preliminary prediction results based on the feature information in the feature map, the input is the feature map from the downsampling layer, and the output is the preliminary prediction results of the detection box and category; the output layer converts the pulse signal into conventional detection box information, the input is the prediction result from the SNN module, and the output is the bounding box, target category label and confidence score, and the bounding box represents the detected target position.

5. The method according to claim 4, characterized in that The input of the SpikeYOLO model is represented as I∈R T×C×H×W , where T represents the time step, C represents the number of channels, and H×W represents the spatial resolution. For the input of a static image, the spike neurons in the input layer of the model encode the continuous values ​​of the input static image into a spike signal; In target process detection, the model is based on the input image sequence Output the category and position of each object. Assuming that the input image sequence has N targets, output The calculation formula is: B=Model(X) (6) Among them, each B n Contains information about the confidence f, category c, center coordinates (x, y), and object size (w, h). Model(·) refers to the SpikeYOLO model.

6. The method according to claim 5, characterized in that For the input of neuromorphic data, i.e. data generated by a dynamic vision sensor, a pulse is generated only when the logarithmic change of the light intensity at a pixel exceeds a preset threshold.

7. The method according to claim 1, characterized in that The SpikeYOLO model combines the macro design of YOLOv8 with the micro design of Meta-SpikeFormer.

8. The method according to any one of claims 1 to 7, characterized in that This method is used in the fields of artificial intelligence, big data, and computer vision technology.

9. A system for implementing the method according to any one of claims 1 to 7.

10. The system according to claim 9, characterized in that The system is used in the fields of artificial intelligence, big data, and computer vision technology.

Images