Construction method for detecting embedded edge computing system in real time

Abstract

The invention provides a construction method for detecting an embedded edge computing system in real time, and relates to the technical field of communication.The method comprises the steps that images of cracks, rust, spalling and air holes of a high-definition pavement are obtained and divided; constructing a spiking neural network model SNNs by using an LIF neuron model, and training the spiking neural network model SNNs by using the divided training set; customizing an LIF neuron model, replacing the LIF neuron model in the trained spiking neural network model SNNs with the custom LIF neuron model, and converting the LIF neuron model into an open neural network exchange ONNX format; and deploying the pulse neural network model SNNs subjected to format conversion to a resource-limited embedded edge computing device for processing a real-time task. According to the invention, the problems of high power consumption and high delay of the ANNs in real-time processing tasks on an embedded system and an edge computing device are solved.

Description

Technical Field

[0001] The present invention belongs to the field of communication technology, and in particular relates to a method for constructing a real-time detection embedded edge computing system. Background Art

[0002] In recent years, with the development of neural networks, spiking neural networks (SNNs), as a new generation of bio-inspired neural networks, have shown great potential in fields such as healthcare, intelligent driving, and industrial detection due to their low power consumption and unique information encoding methods. In particular, in the power-sensitive and resource-constrained embedded systems and edge computing fields, the sparsity and event-driven nature of spiking neural networks (SNNs) bring significant advantages in low power consumption and high inference speed. Currently, traditional artificial neural networks (ANNs) have been widely used in embedded systems and edge computing, but their high power consumption and high latency pose challenges to the processing of real-time tasks. Compared to ANNs, spiking neural networks (SNNs), which have lower power consumption, faster inference speed, and are more suitable for processing real-time tasks, are still at a relatively early stage of application in these fields. Summary of the Invention

[0003] In response to the above-mentioned deficiencies in the prior art, the present invention provides a method for constructing a real-time detection embedded edge computing system, which solves the problems of high power consumption and high latency in real-time processing tasks of artificial neural networks (ANNs) on embedded systems and edge computing devices.

[0004] In order to achieve the above objectives, the technical solution adopted by the present invention is: a method for constructing a real-time detection embedded edge computing system, comprising the following steps:

[0005] S1. Obtain high-definition road surface defect images and divide them into proportions to construct a dataset.

[0006] S2. Using the LIF neuron model to construct a spiking neural network model SNNs, and using the divided data set to train the spiking neural network model SNNs;

[0007] S3. Customize the LIF neuron model and use it to replace the LIF neuron model in the trained spiking neural network model SNNs, and convert it into the Open Neural Network Exchange (ONNX) format;

[0008] S4. Deploy the converted spike neural network model (SNNs) to resource-constrained embedded edge computing devices to process real-time image classification tasks and complete the construction of a real-time detection embedded edge computing system.

[0009] Furthermore, the membrane potential update expression of the LIF neuron model in S2 is as follows:

[0010] V(t)=β·V(t-1)+WI(t-1)-S(t-1)·θ

[0011] Where V(t) represents the instantaneous membrane potential at the current time step t, β represents the leakage coefficient of the membrane potential, V(t-1) represents the instantaneous membrane potential at the previous time step, W represents the weight, I(t-1) represents the input at the previous time step, S(t-1) represents the output pulse at the previous time step, and θ represents the threshold.

[0012] Furthermore, the expression of the output pulse of the LIF neuron model in S2 is as follows:

[0013]

[0014] Where S(t) represents the output pulse at the current time step.

[0015] Furthermore, the membrane potential update expression of the custom LIF neuron model in S3 is as follows:

[0016] V tmp (t) = V mem -S'(t-1)·V thresh

[0017] Among them, V tmp (t) represents the temporary membrane potential at time step t, V mem represents the actual membrane potential at time step t, S'(t-1) represents the output pulse at the previous time step, and V thresh Indicates the threshold for pulse emission.

[0018] Furthermore, the expression of the output pulse of the custom LIF neuron model in S3 is as follows:

[0019]

[0020] Where S'(t) represents the output pulse at the current time step t.

[0021] Beneficial effects of the present invention:

[0022] 1. This invention aims to build a complete, reasonable, and efficient embedded system to open up a new practical path for the deployment and application of spiking neural networks (SNNs). While ensuring model performance, it fully utilizes the low power consumption and low latency advantages of spiking neural networks (SNNs), deploying them on resource-constrained embedded edge computing devices to handle real-time processing tasks. The main contributions of this invention are as follows: (1) It solves the operator conflict problem that occurs when the spiking neural network (SNN) model is converted to the Open Neural Network Exchange (ONNX) format; (2) It breaks through the high cost and low portability problems of existing spiking neural network (SNN) hardware implementation methods; (3) It improves the portability, interoperability, and performance of spiking neural networks (SNNs) in edge computing environments.

[0023] 2. The present invention solves the problem of operator conflict during model conversion by customizing the LIF neuron model and manually adding external corresponding operators, so that spiking neural networks (SNNs) can be successfully converted to the ONNX format and deployed on embedded systems.

[0024] 3. Due to the sparsity and event-driven nature of spiking neural networks (SNNs), the introduction of a custom LIF neuron model further reduces system power consumption. The system significantly reduces energy consumption and inference time while maintaining high performance, which is particularly important for resource-constrained embedded systems and edge computing devices.

[0025] 4. By leveraging ONNX’s cross-framework deployment capabilities, the system is deployed across frameworks on the hardware platform, effectively improving the portability and interoperability of spiking neural networks (SNNs).

[0026] 5. The embedded edge computing system of the present invention combines the advantages of hardware and software, and can achieve fast response and high-precision real-time image classification tasks, meeting the real-time detection requirements in the edge computing environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Flow chart of the method of the present invention.

[0028] Figure 2 It is a training process record diagram of the present invention. DETAILED DESCRIPTION

[0029] The specific embodiments of the present invention are described below to facilitate understanding of the present invention by those skilled in the art. However, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the appended claims, these changes are obvious, and all inventions and creations utilizing the concepts of the present invention are protected.

[0030] Example

[0031] The basic idea of this invention is to replace the official LIF neuron model in the trained spiking neural network (SNN) model with a custom LIF neuron model, convert it to ONNX, and deploy it on the development board to perform real-time image classification tasks. After deployment, the performance of the model is evaluated using four key indicators: Accuracy, Precision, Recall, and F1 Score. The energy consumption and real-time performance of the model are evaluated based on the average power consumption and average inference time of the model when inferring a single image. Figure 1 As shown, the present invention provides a method for constructing a real-time detection embedded edge computing system, and its implementation method is as follows:

[0032] S1. Obtain high-definition road surface defect images and divide them into proportions to construct a dataset.

[0033] In this embodiment, high-definition images of four road surface defects, namely cracks, rust, spalling, and pores, are obtained and divided into a data set in a ratio of 8:2 for subsequent spiking neural network model (SNNs) training.

[0034] In this embodiment, a high-definition industrial camera is used to capture images of four road surface defects, namely, cracks, rust, spalling, and pores. Through image augmentation and image enhancement, an experimental data set of 1,550 images is formed, and the data set is divided into a training set and a test set in a ratio of 8:2 to ensure the quality and consistency of the data.

[0035] S2. Use the LIF neuron model to construct a spiking neural network model SNNs, and use the divided data set to train the spiking neural network model SNNs;

[0036] In this embodiment, during the training process, the hyperparameter training round epoch = 100, the learning rate lr = 0.0005, the Adam optimizer is used, and the loss function selects the cross entropy loss function CrossEntropyloss. The training process of the model is recorded as follows: Figure 2 As shown in the figure, the four constructed SNNs network models CNN-SNN, VGG-SNN, DenseNet-SNN, and ResNet-SNN all performed well during the training process. As the number of training rounds increased, the classification accuracy of the model on the training set and the validation set increased steadily, and finally converged to a higher level, indicating that the model has strong classification learning ability. During the training process, there was no significant difference in the performance of the training set and the validation set, and no obvious overfitting phenomenon was observed, indicating that the model has strong generalization ability. Therefore, Figure 2 The effectiveness of the constructed SNNs model was verified.

[0037] In this embodiment, a spiking neural network model (SNNs) is constructed using the LIF neuron model. The membrane potential update formula and output pulse formula of the LIF neuron model can be expressed by formula (1) and formula (2).

[0038] V(t)=β·V(t-1)+WI(t-1)-S(t-1)·θ (1)

[0039]

[0040] Where V(t) represents the instantaneous membrane potential at the current time step t, β represents the leakage coefficient of the membrane potential, V(t-1) represents the instantaneous membrane potential at the previous time step, W represents the weight, I(t-1) represents the input at the previous time step, S(t-1) represents the output pulse at the previous time step, θ represents the threshold, and S(t) represents the output pulse at the current time step. When the membrane potential value is higher than the threshold θ, the neuron will release a pulse.

[0041] In this embodiment, a basic convolutional spiking neural network (CNN) was constructed by adding LIF neurons to the basic convolutional neural network (CNN), processing each time step in the final fully connected layer, and recording the pulse and membrane potential state at each time step. Its overall structure and operations at different stages are shown in Table 1. At the same time, the same operations were used in the CNN variants residual network (ResNet), densely connected network (DenseNet), and VGG to construct the residual spiking neural network (ResNet-SNN), densely connected spiking neural network (DenseNet-SNN), and VGG spiking neural network (VGG-SNN), and subsequently trained. Table 1 shows the overall structure and operation table of the CNN-SNN.

[0042] Table 1

[0043] stage Input size operate Output size Phase 1 3x224x224 Convolution + Max Pooling 16x112x112 Phase 2 16x112x112 Convolution + Max Pooling 32x56x56 Phase 3 32x56x56 pave 1x100352 Phase 4 1x100352 Fully connected 1x512 Phase 5 1x512 LIF neuron manipulation 1x512 Phase 6 1x512 Fully connected 1x4 Phase 7 1x4 LIF neuron manipulation 1x4

[0044] S3. Customize the LIF neuron model and use it to replace the LIF neuron model in the trained spiking neural network model SNNs, and convert it to ONNX format;

[0045] In this example, a custom LIF neuron model was used. After comparing its performance with the official LIF neuron model to ensure its feasibility, the official LIF neuron model was replaced in the trained spiking neural network (SNN) model. This model was converted to the ONNX format and deployed. The membrane potential update formula and output pulse formula of the custom neuron model can be described by Equations (3) and (4).

[0046] V tmp (t) = Vmem -S'(t-1)·V thresh (3)

[0047]

[0048] Among them, V tmp (t) represents the temporary membrane potential at time step t, V mem represents the actual membrane potential at time step t, S'(t-1) represents the output pulse at the previous time step, and V thresh represents the threshold of pulse emission, and S'(t) represents the output pulse at the current time step t.

[0049] In this embodiment, the customized LIF neuron model is highly flexible, with parameters adjustable as needed. It can be easily integrated into existing deep learning frameworks, making it easier to understand and implement. Floating-point numbers are used to calculate the model, thereby improving stability and accuracy. By using the customized LIF neuron model, this method not only ensures the success rate of the SNN conversion, but also completes the model conversion to ONNX format while ensuring performance and speed.

[0050] S4. Deploy the converted spike neural network model SNNs to resource-constrained embedded edge computing devices to process real-time tasks and complete the construction of a real-time detection embedded edge computing system.

[0051] In this embodiment, the spike neural network model SNNs after format conversion is saved locally on the NVIDIA JETSON AGX XAVIER module development board, and its powerful AI computing capability is used in conjunction with the open neural network exchange ONNX format model inference engine onnxruntime to perform model inference. In this way, the model is deployed on resource-constrained embedded edge computing devices to process real-time image classification tasks, completing the construction of a real-time detection embedded edge computing system.

[0052] In this example, after setting up the development board environment, a GT256 infrared pod camera was connected via a network cable to achieve high-precision detection. Leveraging the powerful computing capabilities of the NVIDIA JETSON AGX XAVIER development board, image data was processed and analyzed in real time. By combining software and hardware, a fast-response, accurate, and reliable real-time image classification edge computing embedded system was constructed.

[0053] In this embodiment, according to the neuron performance comparison experimental results in Table 2, under the same batch of generated data, the customized LIF neuron (Custom LIF) occupies slightly more memory than the official LIF neuron (Leaky), but has a faster running speed and shows good stability in multiple runs, which can well meet the requirements of fast response and high real-time performance of this system. Table 2 is a performance comparison table of LIF neuron models.

[0054] Table 2

[0055]

[0056]

[0057] In this embodiment, from the comparative experimental results of the models in Table 3 after conversion to the ONNX format and deployment on the development board, in terms of performance, the four network models all performed well. Among them, although the CNN-SNN, VGG-SNN, and DenseNet-SNN network models experienced some slight performance degradation after conversion to the ONNX format, they still maintained good performance in key performance indicators. In particular, the ResNet-SNN network model, because it is particularly suitable for processing this task and dataset, still maintains extremely high performance indicators after conversion to the ONNX format and deployment locally; in terms of energy consumption, the energy consumption performance of the four network models has been significantly improved after conversion to the ONNX format and deployment on the development board. The experimental results show that these models have achieved significant energy consumption reductions while maintaining high performance, which is particularly important for resource-constrained embedded systems. Table 3 is a performance comparison table before and after model conversion and deployment.

[0058] Table 3

[0059]

[0060]

[0061]

[0062] In summary, the present invention solves the problems of high power consumption and high latency in real-time processing tasks of artificial neural networks (ANNs) on embedded systems and edge computing devices.

Claims

1. A method for constructing a real-time detection embedded edge computing system, characterized in that: The following steps are involved: S1. Obtain high-definition road surface defect images and divide them into proportions to construct a dataset. S2. Using the LIF neuron model to construct a spiking neural network model SNNs, and using the divided data set to train the spiking neural network model SNNs; S3. Customize the LIF neuron model and use it to replace the LIF neuron model in the trained spiking neural network model SNNs, and convert it into the Open Neural Network Exchange (ONNX) format; S4. Deploy the converted spike neural network model (SNNs) to resource-constrained embedded edge computing devices to process real-time image classification tasks and complete the construction of a real-time detection embedded edge computing system.

2. The method for constructing a real-time detection embedded edge computing system according to claim 1, characterized in that: The membrane potential update expression of the LIF neuron model in S2 is as follows: V(t)=β·V(t-1)+WI(t-1)-S(t-1)·θ Where V(t) represents the instantaneous membrane potential at the current time step t, β represents the leakage coefficient of the membrane potential, V(t-1) represents the instantaneous membrane potential at the previous time step, W represents the weight, I(t-1) represents the input at the previous time step, S(t-1) represents the output pulse at the previous time step, and θ represents the threshold.

3. The method for constructing a real-time detection embedded edge computing system according to claim 2, characterized in that: The expression of the output pulse of the LIF neuron model in S2 is as follows: Where S(t) represents the output pulse at the current time step.

4. The method for constructing a real-time detection embedded edge computing system according to claim 1, wherein: The membrane potential update expression of the custom LIF neuron model in S3 is as follows: V tmp (t)=V mem -S'(t-1)·V thresh Among them, V tmp (t) represents the temporary membrane potential at time step t, V mem represents the actual membrane potential at time step t, S'(t-1) represents the output pulse at the previous time step, and V thresh Indicates the threshold for pulse emission.

5. The method for constructing a real-time detection embedded edge computing system according to claim 1, wherein: The expression of the output pulse of the custom LIF neuron model in S3 is as follows: Where S'(t) represents the output pulse at the current time step t.

Images