A hybrid semantic communication system based on spiking neural network

Abstract

The present invention belongs to the field of semantic communication technology, and specifically is a hybrid semantic communication system based on a pulse neural network. The present invention uses an ANN model for semantic extraction on the server side; then the semantic information is discretized in the spatiotemporal dimension and converted into a data format suitable for SNN model processing; the received semantic information is processed using an easy-to-deploy and low-power SNN model on the edge side to achieve efficient and high-precision semantic understanding. And through the intelligent allocation submodule, different frequency feature optimization allocation methods can be used to match the corresponding pulse neuron to trigger calculation only when a pulse event is detected, and the threshold conditions for multiple signal triggering of a single pulse neuron can be set to achieve up to three levels of trigger-type power consumption optimization. In combination with the use of FPGA, high-efficiency performance is maintained when processing large-scale data. The present invention is particularly suitable for low-power and high-efficiency communication application scenarios.

Description

Technical Field

[0001] The present invention belongs to the field of semantic communication technology combining artificial intelligence and edge computing, and in particular relates to a hybrid semantic communication system based on a pulse neural network. Background Art

[0002] At present, with the widespread application of artificial intelligence in various fields, the traditional centralized computing model faces many challenges, especially in terms of latency, bandwidth and power consumption, and edge computing has gradually become a solution. However, with the increase in the number of computing nodes, the communication cost is also rising, and bandwidth expansion and latency reduction become more difficult in extreme environments, and information transmission efficiency is limited. The emergence of semantic communication has further improved communication efficiency, but the existing technology has problems such as low information compression efficiency and limited hardware resources at the edge.

[0003] Problems or defects of existing semantic communication technologies include:

[0004] 1. Low information compression efficiency: The classic semantic communication architecture consists of two parts: semantic extraction and semantic understanding. The mainstream implementation method of semantic extraction is to use artificial neural networks (ANN) for semantic extraction and information compression. Although it improves efficiency compared to traditional communication methods, it is difficult to further improve compression efficiency due to the full-precision extraction characteristics of the ANN model.

[0005] 2. Hardware-unfriendly, difficult to deploy on resource-constrained edge devices: The semantic understanding part is implemented on the edge device, but the application of the ANN model requires a large amount of storage and computing resources, which brings great difficulties to the deployment of the ANN model for semantic understanding on the edge device with limited resources.

[0006] 3. High energy consumption: The complexity and high resource overhead of the ANN model lead to a significant increase in power consumption. Summary of the invention

[0007] In view of the above-mentioned problems or deficiencies, in order to solve the problems of low compression efficiency of existing semantic communication, difficulty in deployment at the resource-constrained edge, and high energy consumption, the present invention proposes a hybrid semantic communication system based on a pulse neural network. The present invention uses an ANN model to perform full-precision semantic extraction on the server side, and converts the extracted semantic information into a data format processed by a pulse neural network (SNN) through semantic discretization in the spatiotemporal dimension, further compresses the semantic information, and uses the SNN model on the edge for efficient semantic understanding. In this way, the lightweight and event-driven characteristics of the SNN model are fully utilized to achieve more efficient semantic extraction and semantic understanding at a low hardware cost, significantly reducing the computational complexity and energy consumption of the communication system, and being able to understand and process information more intelligently, thereby improving its applicability and reliability.

[0008] A hybrid semantic communication system based on a pulse neural network, comprising: an ANN model, a spatiotemporal semantic discretization module, an encoder, a channel, a decoder and an SNN model, such as Figure 1 shown.

[0009] The ANN model consists of an input layer, multiple convolutional layers and an output layer, and is used for full-precision semantic extraction and information compression. The original data enters the ANN model set on the server side through the input layer, and is processed by multiple convolutional layers to extract deep semantic information, which is output through the output layer.

[0010] The spatiotemporal dimension semantic discretization module performs format conversion and data compression on the full-precision semantic information output by the ANN model, converting it into a pulse data format that can be processed by the SNN model, further improving the compression efficiency; the converted semantic information is sent to the encoder.

[0011] The encoder encodes the semantic information after the semantic discretization of the spatiotemporal dimension into a transmittable data format, and then sends it from the server to the decoder at the edge via the channel, thereby achieving high-quality and efficient transmission of the semantic information.

[0012] The decoder decodes the received information and restores it to the state processed by the spatiotemporal dimension semantic discretization module.

[0013] The SNN model consists of N fully connected layers, and the SNN model deployment is implemented through the heterogeneous neuromorphic computing core at the edge to understand the semantic information decoded by the decoder.

[0014] The structure of the heterogeneous neuromorphic computing core is as follows Figure 3 As shown, it includes a memory module and N pulse calculation modules; a single pulse calculation module is used as a fully connected layer, and N pulse calculation modules are cascaded in sequence for processing, and the signal processed by the N pulse calculation modules is output or transmitted to other external units.

[0015] The memory module is used to temporarily store the pulse signal decoded from the decoder to ensure that the signal can be subsequently processed in an orderly manner.

[0016] The pulse calculation module includes an intelligent allocation submodule and M pulse neurons; the intelligent allocation submodule optimizes the allocation according to the frequency characteristics of the pulse signal and sends it to the designated pulse neuron for processing; after a single pulse neuron is triggered, the output pulse signal is sent to the intelligent allocation submodule of the next-level pulse calculation module or output to an external unit.

[0017] Through the cascade of N pulse calculation modules and the collaboration of intelligent allocation sub-modules, each pulse signal is optimally allocated to the corresponding pulse neuron for processing according to its frequency characteristics, ultimately realizing the fully connected operation of the SNN model.

[0018] Furthermore, the channel is wireless transmission or wired transmission.

[0019] Furthermore, a buffer is provided between the memory module and the first pulse computing module to improve the data access speed of the heterogeneous neuromorphic computing core and reduce the bandwidth pressure of the memory module.

[0020] Furthermore, a buffer is provided after the Nth pulse calculation module, and the data is output after being buffered by the buffer to improve the stability of data output.

[0021] Furthermore, in the pulse calculation module, each intelligent allocation submodule optimizes the allocation method according to different frequency characteristics, and sends the corresponding pulse signal to the designated pulse neuron. The calculation of the pulse neuron is triggered only when a pulse signal event is received, so as to realize the second-level trigger-type work, further reduce power consumption or realize more complex calculations.

[0022] Furthermore, the pulse neuron is implemented using a memristor, and the trigger threshold condition of the memristor is the accumulation result of at least three pulse signals, so as to further reduce power consumption or realize more complex operations.

[0023] Furthermore, the heterogeneous neuromorphic computing core is implemented using FPGA to understand the received semantic information.

[0024] The specific operation process of the present invention is as follows: after the data is collected from the sensor or input device, it is first transmitted to the server side; the server side uses the ANN model to perform semantic extraction and information compression; then the extracted semantic information is converted into a data format suitable for SNN model processing after semantic discretization in the spatiotemporal dimension, and then transmitted to the edge side through wireless communication and other means; finally, the edge side uses the SNN model to further process the received semantic information to achieve efficient semantic understanding and output.

[0025] The ANN model is deployed on the server side for full-precision semantic extraction, and combined with the spatiotemporal semantic discretization module, the semantic information is converted into a format that can be processed by the SNN model. This process significantly improves the efficiency of information compression and realizes efficient semantic extraction and information compression.

[0026] An easy-to-deploy and low-power SNN model is used at the edge, which has the characteristics of lightweight design, event-driven and sparse activation. Spiking neurons only trigger calculations when a spike event is detected, and within the rated time, a small number (1%-5%) of spiking neurons are activated, and a large number (95%-99%) of spiking neurons remain silent. Compared with the ANN model in which all neurons participate in the calculation, the power consumption of the system is further reduced, making it suitable for a variety of low-power application scenarios.

[0027] Furthermore, the SNN model is implemented using FPGA. The highly parallel fully connected computing of FPGA can process multiple tasks simultaneously, optimize the allocation of computing resources, reduce latency and energy consumption, greatly improve system throughput and reduce computing latency, and maintain efficient performance when processing large-scale data.

[0028] In summary, the present invention provides a semantic communication system for information compression and transmission that is efficient, low-power, and has powerful computing capabilities. It is suitable for a variety of application scenarios, including intelligent monitoring, the Internet of Things, edge computing, and mobile devices. In particular, it can provide excellent performance under the requirements of low power consumption and efficient communication, and has excellent engineering application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 is a system architecture block diagram of the present invention;

[0030] Figure 2 It is a structural block diagram of the ANN model and the SNN model used in the present invention;

[0031] Figure 3 This is a diagram of the heterogeneous neuromorphic core architecture of the present invention;

[0032] Figure 4 is an information processing flow chart of the present invention;

[0033] Figure 5 It is a structural block diagram of a hybrid semantic communication system based on a pulse neural network used in the embodiment;

[0034] Figure 6 It is the structural block diagram of the traditional ANN semantic communication system used in the comparative example;

[0035] Figure 7 The bandwidth overhead comparison curve of the two semantic communication systems of the embodiment and the comparative example is shown. DETAILED DESCRIPTION

[0036] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings and embodiments.

[0037] The following experiment compares the traditional ANN model semantic communication system with the hybrid semantic communication system based on pulse neural network proposed in this invention, and uses bandwidth overhead (the amount of data required to transmit to achieve the same goal) as an indicator to measure communication efficiency. The lower the bandwidth overhead, the higher the communication efficiency. The experiment uses the semantic communication system to input a portrait image and output 8 pose labels of the portrait at the output end. Figure 2 It is a structural block diagram of the ANN model and the SNN model used in the present invention; Figure 3 This is a diagram of the heterogeneous neuromorphic core architecture of the present invention; Figure 4It is an information processing flow chart of the present invention.

[0038] The structure of the hybrid semantic communication system based on pulse neural network designed in this embodiment is as follows: Figure 5 As shown in the figure, the server-side ANN model consists of 8 3×3 convolutional layers in series, and the edge-side SNN model consists of 4 fully connected layers. The number of spike neurons in the 4 fully connected layers is 64, 32, 32, and 8 respectively. The semantic information is transmitted via wireless Bluetooth. The comparison example is the traditional ANN model semantic communication system, with the structure as shown in the figure. Figure 6 As shown in the figure, the ANN model on the server side consists of 8 3×3 convolutional layers, and the ANN model on the edge side consists of 4 fully connected layers. The number of artificial neurons in the 4 fully connected layers is 64, 32, 32, and 8 respectively. The semantic information is transmitted via wireless Bluetooth. There are two differences between the two semantic communication structures used in the experiment:

[0039] 1. The hybrid semantic communication system based on pulse neural network has a spatiotemporal semantic discretization module on the server side, which is not available in the traditional ANN model semantic communication system.

[0040] 2. The hybrid semantic communication system based on pulse neural network uses SNN model for semantic understanding at the edge, and the traditional ANN model semantic communication system uses ANN model for semantic understanding at the edge. The network structure of the two is the same at the edge, but the difference is that the units that make up the fully connected layer are pulse neurons or artificial neurons.

[0041] Data processing flow of the hybrid semantic communication system based on spiking neural networks: A 784×761 preprocessed image is divided into 10 timesteps according to the time series, and the 10 timesteps are input into the system in sequence. First, the full-precision semantic information is extracted after 8 3×3 convolutions, and then converted into a pulse data format encoded according to the frequency after semantic discretization in the spatiotemporal dimension, and sent to the edge via wireless Bluetooth. At the edge, it passes through 4 layers of fully connected layers composed of spiking neurons for semantic understanding, and finally outputs an 8-bit unique hot code, representing one of the eight postures.

[0042] Traditional ANN model semantic communication system data processing flow: A 784×761 pre-processed image is divided into 10 timesteps according to the time series, and the 10 timesteps are input into the system in sequence. After 8 3×3 convolutions, the full-precision semantic information is extracted, and the extracted semantic information is directly sent to the edge via wireless Bluetooth. At the edge, it passes through 4 layers of fully connected layers composed of artificial neurons for semantic understanding, and finally outputs an 8-bit unique hot code, representing one of the eight postures.

[0043] The experimental results are as follows Figure 7 As shown, the bandwidth overhead comparison curves of the traditional ANN model semantic communication system (comparative example) and the hybrid semantic communication system based on pulse neural network proposed in the present invention (implementation) are shown. The experimental results show that in the first 4 timesteps, compared with the traditional semantic communication system, the bandwidth overhead of the semantic communication system proposed in the present invention is reduced by 96.8%, 93.7%, 90.6% and 87.5% respectively. After all 10 timesteps are transmitted, the bandwidth overhead is reduced by 68.7%.

[0044] The resource usage of the two semantic communication systems on the edge FPGA is shown in Table 1.

[0045] Table 1:

[0046] Resource Type ANN Model SNN Model Look-up Table (LUT) 9% 3% Flip-flop (FF) 4.5% 1.35% Digital Signal Processing Unit (DSP) 96% 45% On-chip memory (BRAM) 1.5% 0.7%

[0047] According to the data in Table 1, compared with the traditional ANN model semantic communication system, the hybrid semantic communication system based on pulse neural network proposed in the present invention occupies greatly less resources, which is conducive to deployment at the edge end with limited resources.

[0048] In terms of edge power consumption, the edge power consumption of the traditional ANN semantic communication system fluctuates between 800-1000mW, while the edge power consumption of the hybrid semantic communication system based on pulse neural network proposed in the present invention fluctuates between 40-100mW, reducing the power consumption by about 95%.

[0049] Through the analysis of the above embodiments and comparative examples, it can be seen that the present invention uses an ANN model for semantic extraction on the server side; then the semantic information is discretized in the spatiotemporal dimension and converted into a data format suitable for SNN model processing; the received semantic information is processed by an SNN model that is easy to deploy and has low power consumption at the edge to achieve efficient and high-precision semantic understanding. And through the different frequency feature optimization allocation methods of the intelligent allocation submodules of each layer, the corresponding pulse neurons can be used to trigger the calculation only when a pulse event is detected, and the threshold conditions for multiple signal triggering of a single pulse neuron can be set to achieve up to three levels of trigger-type power consumption optimization. In combination with the use of FPGA, the allocation of computing resources is optimized, latency and energy consumption are reduced, the system throughput is greatly improved, and the computing latency is reduced, and high-efficiency performance is maintained when processing large-scale data. Compared with the traditional semantic communication system, the hybrid semantic communication system based on the pulse neural network proposed in the present invention greatly improves the efficiency of information compression, reduces the edge end's occupancy of FPGA resources, reduces bandwidth overhead, and has a significant reduction in power consumption, which is particularly suitable for low-power and high-efficiency communication application scenarios.

Claims

1. A hybrid semantic communication system based on a spiking neural network, characterized in that: Including ANN model, spatiotemporal semantic discretization module, encoder, channel, decoder and SNN model; The ANN model consists of an input layer, multiple convolutional layers and an output layer, and is used for full-precision semantic extraction and information compression; the original data enters the ANN model set on the server side through the input layer, is processed by multiple convolutional layers, and deep semantic information is extracted and output through the output layer; The spatiotemporal dimension semantic discretization module performs format conversion and data compression on the full-precision semantic information output by the ANN model, converting it into a pulse data format that can be processed by the SNN model, and sending the converted semantic information to the encoder; The encoder encodes the semantic information after the semantic discretization of the spatiotemporal dimension into a transmittable data format, and then sends it from the server to the decoder at the edge via the channel; The decoder decodes the received information and restores it to the state processed by the spatiotemporal dimension semantic discretization module; The SNN model consists of N fully connected layers, and the SNN model deployment is realized through the heterogeneous neuromorphic computing core at the edge to understand the semantic information decoded by the decoder; The heterogeneous neuromorphic computing core includes a memory module and N pulse computing modules, with a single pulse computing module as a fully connected layer, and the N pulse computing modules are sequentially cascaded for processing, and the signal processed by the N pulse computing modules is output or transmitted to other external units; The memory module is used to temporarily store the pulse signal decoded from the decoder to ensure that the signal can be subsequently processed in an orderly manner; The pulse calculation module includes an intelligent allocation submodule and M pulse neurons; The intelligent allocation submodule optimizes the allocation according to the frequency characteristics of the pulse signal and sends it to the designated pulse neuron for processing; After a single pulse neuron is triggered, it sends the output pulse signal to the intelligent allocation submodule of the next level pulse calculation module or outputs it to an external unit; Through the cascade of N pulse calculation modules and the collaboration of intelligent allocation sub-modules, each pulse signal is optimally allocated to the corresponding pulse neuron for processing according to its frequency characteristics, ultimately realizing the fully connected operation of the SNN model.

2. The hybrid semantic communication system based on pulse neural network as claimed in claim 1, characterized in that: The channel is wireless transmission or wired transmission.

3. The hybrid semantic communication system based on pulse neural network as claimed in claim 1, characterized in that: A buffer is also provided between the memory module and the first pulse calculation module.

4. The hybrid semantic communication system based on pulse neural network as claimed in claim 1, characterized in that: A buffer is also set after the Nth pulse calculation module, and the data is output after being buffered by the buffer.

5. The hybrid semantic communication system based on pulse neural network as claimed in claim 1, characterized in that: In the pulse calculation module, each intelligent allocation submodule optimizes the allocation method according to different frequency characteristics and sends the corresponding pulse signal to the specified pulse neuron. The calculation of the pulse neuron is only triggered when a pulse signal event is received, so as to further reduce power consumption or realize more complex calculations.

6. The hybrid semantic communication system based on pulse neural network as claimed in claim 5, characterized in that: The pulse neuron is implemented by a memristor, and the trigger threshold condition of the memristor is the accumulation result of at least three pulse signals, so as to further reduce power consumption or realize more complex calculations.

7. The hybrid semantic communication system based on pulse neural network as claimed in claim 1, characterized in that: The heterogeneous neuromorphic computing core is implemented using FPGA.

Images