Distributed semantic processing of sensor data
By optimizing sensor networks using deep neural networks and joint source-channel coding, the problems of failure and high latency in traditional systems under low channel quality are solved, achieving efficient and robust sensor data processing.
Patent Information
- Application Number
- CN202380100505.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-18
- Publication Date
- 2026-02-13
AI Technical Summary
Traditional wireless sensor networks fail when channel quality falls below a threshold and have high computational latency, making it difficult to effectively process large amounts of complex sensor data.
A deep neural network (DNN) is used for semantic feature extraction and joint source-channel coding (JISC). Combined with distributed inference, the sensor equipment layout and fusion equipment are optimized to achieve the joint processing of semantic processing and channel coding.
Reduce communication costs and latency, improve the ability to recover from channel distortion, avoid the cliff effect, and enhance robustness to channel fluctuations.
Smart Images
Figure CN121532777A_ABST
Abstract
Description
Technical Field
[0001] This invention generally relates to the field of semantic network learning, and more particularly to sensor device arrangements and sensor fusion devices for distributed semantic processing of (accumulated) sensor data, corresponding operating methods and wireless sensor networks. Background Technology
[0002] A growing number of applications and services, such as robotics, autonomous driving, traffic management, and smart factories, rely on technologies like object recognition and computer vision. In these applications and services, multiple distributed sensors collect information about the environment to enable complex decision-making at a control center. However, the increasing volume and / or complexity of sensor data transmitted by sensors and processed by the control center makes efficient decision-making a highly challenging task.
[0003] Another challenge associated with wireless sensor networks is the volatility of channel characteristics. Traditional communication systems rely on the separation principle, where source and channel coding are performed independently in two steps. When channel quality falls below a certain threshold, such systems often fail completely because channel coding can no longer correct errors. This phenomenon is often referred to as the cliff effect. Furthermore, the two-step process introduces unnecessary computational delays, which are undesirable in many real-time applications. Therefore, treating the source encoder and channel encoder as independent components is generally no longer optimal. Summary of the Invention
[0004] The purpose of this invention is to overcome the above and other shortcomings by using sensor data semantic processing based on deep neural networks (DNN) and joint source-channel coding (JSCC).
[0005] The above and other objectives are achieved through the features of the independent claims. Other implementations are apparent from the dependent claims, the specification, and the drawings.
[0006] According to a first aspect, a sensor device arrangement is provided for distributed semantic processing of sensor data in a wireless sensor network. The sensor device arrangement includes a sensor component, a semantic feature extraction (SFE) coding component, a joint inference-source-channel (JISC) coding component, and a transceiver component. The sensor component generates sensor data of a target variable, the sensor data including one or more features. The SFE component infers a feature vector for semantic processing from the sensor data. The JISC coding component infers a JISC-coded feature vector from the inferred semantic processing feature vector based on a channel model of the uplink channel of the wireless sensor network. The transceiver component transmits the inferred JISC-coded feature vector to a sensor fusion device via the uplink channel for distributed semantic processing of accumulated sensor data in the wireless sensor network.
[0007] The sensor device arrangement combines the advantages of semantic data processing, joint source-channel coding, and NN-based distributed inference: Semantic data processing focuses on the semantic features / interpretation of sensor data, which can significantly reduce communication costs and latency, and improve the ability to recover from channel distortion.
[0008] Joint source-channel coding can avoid unnecessary computational delays and the cliff effect of traditional systems (i.e., when the channel quality falls below a certain threshold and the channel coding can no longer correct errors, these systems often fail completely), thereby improving robustness to channel fluctuations.
[0009] NN-based distributed inference involves different sensor devices processing input data by jointly considering the relevance of the data to the target and the relevance of data acquired by other edge devices. The resulting encoding includes only semantically meaningful information and is associated with joint source-channel coding, suitable for transmission over wireless channels. At the receiver, the distortion coding caused by the received channel is directly used to infer the variables of interest, rather than performing source reconstruction as is typically done. This can significantly reduce the amount of data to be transmitted and reduce computational latency.
[0010] As used herein, sensor device arrangement can refer to a composite sensor device in which its components can be arranged relative to each other in various ways. For example, all components can be arranged together (co-located), or one or more components can be arranged separately from the remaining components.
[0011] As used in this article, semantic processing can refer to processing data at the semantic level by focusing on its intended meaning (i.e., semantic features) rather than its exact representation.
[0012] As used in this paper, semantic feature extraction (SFE) encoding can refer to inferring feature vectors (i.e., one or more features) from sensor data.
[0013] As used in this paper, joint inference-source-channel (JISC) coding can refer to a combination of distributed inference (locally observed data samples based on the relevance of a given task), distributed source coding (locally observed data samples as well as data samples implicitly observed by other sensor devices), and channel coding (i.e., adaptation to the wireless channel).
[0014] As used in this article, a transceiver can refer to a combination of a transmitter and a receiver used for wireless communication.
[0015] As used in this article, inference can refer to the forward operation of a trained neural network.
[0016] As used in this article, a vector can refer to an n-tuple (i.e., a finite sequence of n numbers) representing the elements of a vector space.
[0017] As used in this paper, a channel model can refer to a mathematical representation of the adverse effects of a communication channel on signals propagating through that channel.
[0018] As used herein, an uplink channel can refer to a communication channel from any of the different sources of information in a wireless sensor network (i.e., the corresponding sensor device arrangement) toward the central entity of the wireless sensor network (i.e., the sensor fusion device).
[0019] As used in this article, sensor fusion can refer to combining sensor data or data derived from different sources so that the uncertainty of the resulting information is lower than that of information from a single source.
[0020] In one implementation, the SFE encoding component may include a deep neural network (DNN) for receiving the sensor data at its input; inferring a feature vector for semantic processing from the received sensor data; and performing a forward operation on the inferred feature vector at its output.
[0021] As used in this article, a deep neural network (DNN) can refer to an artificial neural network (ANN) that has multiple hidden layers between the input and output layers.
[0022] In one implementation, the JISC encoding component may include alternating sequences of feature modules and attention modules. The corresponding feature module may include a DNN for inferring semantic processing feature vectors from received feature vectors and outputting the inferred feature vectors. The corresponding attention module may include a DNN for inferring a weight vector indicating the feature relevance of the received feature vectors based on the channel model of the uplink channel of the wireless sensor network; and outputting a linear combination of the inferred weight vector and the received feature vectors.
[0023] In one implementation, the inferred weight vector may include a normalized vector.
[0024] As used in this article, normalization can refer to a vector of unit length (i.e., a unit vector).
[0025] In one implementation, the JISC coding component can also be used to receive channel-state information (CSI) from the sensor fusion device to define the channel model of the uplink channel of the wireless sensor network.
[0026] As used herein, channel-state information (CSI) can refer to information representing the current channel conditions, such as, in the case of channel estimation, comprehensive knowledge of the transmitted (i.e., training / pilot sequences) and received signals.
[0027] In one implementation, the SFE encoding component can also be used to infer a feature vector for semantic processing from the sensor data including labeled training data; send the inferred feature vector to the sensor fusion device; receive an error vector from the output of the sensor device arrangement from the sensor fusion device; and update the weights of its DNN based on the received error vector.
[0028] As used herein, labeled training data can refer to training data used for supervised learning, including exemplary input data and corresponding expected output data.
[0029] As used in this article, supervised learning can refer to the basic machine learning techniques of artificial neural networks, the goal of which is to learn general rules for mapping exemplary input data to desired output data.
[0030] In one implementation, the transceiver component can also be used to record all forward operations associated with the labeled training data; receive an error vector from the sensor fusion device at the output of the sensor device via the downlink channel of the wireless sensor network; and estimate the error vector at the output of the JISC encoding component based on the received error vector and the recorded forward operations. The JISC encoding component can also be used to update the weights of its DNN based on the estimated error vector.
[0031] As used in this paper, forward operations can refer to all operations performed by the corresponding transceiver components during inference, such as power normalization, quantization, signal modulation, etc. The idea is to estimate the error (backward) propagation based on the recorded forward operations.
[0032] As used herein, a downlink channel can refer to a communication channel from the central entity of a wireless sensor network (i.e., the sensor fusion device) toward any of the different sources of information in the wireless sensor network (i.e., the corresponding sensor device arrangements).
[0033] As used in this paper, the error vector can refer to the deviation between the actual output data of an artificial neural network and the expected output data specified by the labeled training data.
[0034] In one implementation, the transceiver components may include one or more of the following: a power normalization unit, a quantization unit, and an orthogonal frequency-division multiplexing (OFDM) modulation unit.
[0035] According to a second aspect, a sensor fusion apparatus is provided for distributed semantic processing of accumulated sensor data in a wireless sensor network. The sensor fusion apparatus includes a transceiver component and a joint inference-source-channel (JISC) decoding component. The transceiver component is used to receive, via an uplink channel of the wireless sensor network, corresponding channel-distortion JISC-encoded feature vectors of a target variable from multiple sensor devices. The JISC decoding component is used to infer an estimate of the target variable from the received feature vectors based on a channel model of the uplink channel of the wireless sensor network.
[0036] As used in this paper, joint inference-source-channel (JISC) decoding can refer to the combination of estimation of target variables inferred from received feature vectors (affected by the uplink channel), source decoding, and channel decoding.
[0037] In one implementation, the JISC decoding component can also be used to receive channel-state information (CSI) to define the channel model of the uplink channel of the wireless sensor network.
[0038] In one implementation, the sensor fusion device may further include an SFE decoding component for receiving feature vectors corresponding to the semantic processing of labeled training data from the plurality of sensor device arrangements; inferring an estimate of the target variable from the received feature vectors; calculating an error vector at the output of the SFE decoding component based on the target variable and the inferred estimate of the target variable; updating the weights of its DNN based on the calculated error vector; and sending the corresponding error vector to the corresponding sensor device arrangement at the output of the corresponding sensor device arrangement.
[0039] As used in this paper, semantic feature extraction (SFE) decoding can refer to the estimation of the target variable inferred from the received feature vector (unaffected by the uplink channel).
[0040] In one implementation, the transceiver component can also be used to record all forward operations associated with the labeled training data. The JISC decoding component can also be used to calculate an error vector at its output based on the target variable and the inferred estimate of the target variable; and update the weights of its DNN based on the calculated error vector. The transceiver component can also be used to estimate a corresponding error vector at the output of the corresponding sensor device deployment based on the calculated error vector, the recorded forward operations, and the channel model of the uplink channel; and transmit the corresponding error vector to the corresponding sensor device deployment via the downlink channel of the wireless sensor network.
[0041] In one implementation, the transceiver component may include an orthogonal frequency-division multiplexing (OFDM) demodulation unit.
[0042] According to a third aspect, a wireless sensor network for distributed semantic processing of sensor data is provided. The wireless sensor network includes an arrangement of multiple sensor devices according to the first aspect or any implementation thereof; and a sensor fusion device according to the second aspect or any implementation thereof, wherein the sensor fusion device and the corresponding sensor device arrangement are in wireless network communication.
[0043] According to a fourth aspect, a method is provided for operating a sensor device arrangement to perform distributed semantic processing on sensor data in a wireless sensor network. The method includes: generating the sensor data of a target variable, the sensor data including one or more features; inferring a feature vector for semantic processing from the sensor data; inferring a JISC-coded feature vector from the inferred feature vector based on a channel model of an uplink channel of the wireless sensor network; and transmitting the inferred JISC-coded feature vector to a sensor fusion device via the uplink channel for performing distributed semantic processing on accumulated sensor data in the wireless sensor network.
[0044] According to a fifth aspect, a method is provided for operating a sensor fusion device to perform distributed semantic processing on accumulated sensor data in a wireless sensor network. The method includes: receiving, via an uplink channel of the wireless sensor network, a corresponding channel-distorted JISC-coded feature vector of a target variable from a plurality of sensor devices; and inferring an estimate of the target variable from the received feature vector based on a channel model of the uplink channel of the wireless sensor network.
[0045] According to a sixth aspect, a computer program including program code is provided, which, when executed on a computer, performs the method described according to the fourth or fifth aspect or any implementation thereof. Attached Figure Description
[0046] The above aspects and implementations are explained below with reference to the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements.
[0047] The accompanying drawings should be considered as schematic illustrations, and the elements shown are not necessarily shown to scale. Rather, the various elements are shown to make their function and general purpose obvious to those skilled in the art.
[0048] Figure 1 Wireless sensor networks 1 and 2 according to the present invention are shown; Figure 2 Showing more details Figure 1 JISC encoding component 13; Figure 3 Showing more details Figure 2 Attention module 132; Figure 4 Showing more details Figure 1 Transceivers 14 and 24; Figures 5 to 7 The joint training of SFE encoding component 12 and SFE decoding component 22 is shown; Figures 8 to 10The joint training of JISC encoding component 13 and JISC decoding component 23 is shown; Figure 11 A flowchart is shown relating to the method 3 for operating the sensor device arrangement 1 and the method 4 for operating the sensor fusion device 2 according to the present invention. Detailed Implementation
[0049] In the following description, reference is made to the accompanying drawings, which form part of this invention, which illustrate by way of description specific aspects of implementations of the invention or aspects in which implementations of the invention may be used. It should be understood that implementations of the invention can be used in other aspects and include structural or logical variations not shown in the drawings. Therefore, the following detailed description should not be construed in a limiting sense, and the scope of the invention is defined by the appended claims.
[0050] For example, it should be understood that the disclosures relating to the described methods also apply to the apparatus or system corresponding to performing the methods, and vice versa. For instance, if one or more specific method steps are described, the corresponding apparatus may include one or more units (e.g., functional units) to perform the described one or more method steps (e.g., one unit performs one or more steps, or multiple units perform one or more of a plurality of steps respectively), even if such one or more units are not explicitly described or shown in the drawings. On the other hand, for example, if a specific apparatus is described based on one or more units (e.g., functional units), the corresponding method may include a step to perform the function of one or more units (e.g., one step performs the function of one or more units, or multiple steps perform the function of one or more of a plurality of units respectively), even if such one or more steps are not explicitly described or shown in the drawings. Furthermore, it should be understood that, unless otherwise expressly stated, features of the various exemplary implementations and / or aspects described herein can be combined with each other.
[0051] Figure 1 Wireless sensor networks 1 and 2 according to the present invention are shown.
[0052] Figure 1 The left side shows a multiple access network with multiple sensor devices arranged 1, suitable for accessing sensor data X in wireless sensor networks 1 and 2. k Perform distributed semantic processing; Figure 1 The sensor fusion device 2 is shown on the right, which is suitable for distributed semantic processing of accumulated sensor data in wireless sensor networks 1 and 2.
[0053] The sensor fusion device 2 and the corresponding sensor device arrangement 1 are in wireless network communication.
[0054] Uplink communication from the corresponding sensor device arrangement 1 to the sensor fusion device 2 is accomplished via a slow-fading wireless channel. We also consider (optionally) a low-bit-rate feedback channel from the sensor fusion device 2 to the corresponding sensor device arrangement 1 for transmitting uplink channel parameters.
[0055] The corresponding sensor device arrangement 1 can acquire / perceive some data related to the inference task assigned to the sensor fusion device 2.
[0056] Therefore, the corresponding sensor device arrangement 1 includes a sensor component 11, a semantic feature extraction (SFE) coding component 12, a joint inference-source-channel (JISC) coding component 13, and a transceiver component 14.
[0057] Sensor component 11 is used to generate sensor data X for the target variable Y. k Sensor data X k Includes one or more features k .
[0058] SFE component 12 is used to extract sensor data X k Inferring feature vectors for semantic processing k .
[0059] More specifically, the SFE encoding component 12 may include a trained deep neural network (DNN) for receiving sensor data X at its input. k From the received sensor data X k Inferring feature vectors for semantic processing k ; at its output end, the inferred feature vector k Perform a forward operation.
[0060] In other words, the SFE component 12 takes available source data as input and outputs semantic features that represent only these features of the source data relevant to a given inference task of the sensor fusion device 2. k This serves to (i) compress (by removing redundant information) and (ii) denoise (i.e., represent relevant features in a form more suitable for further processing). Semantic features kIt complements the features extracted by other sensor device arrangements 1 (this is achieved by appropriately training the DNN so that each sensor device arrangement 1 knows the useful information possessed by other sensor device arrangements 1 at the statistical level without explicit coordination between sensor device arrangements 1).
[0061] Please note that SFE encoding component 12 is an application layer function and does not consider the communication channel.
[0062] In contrast, JISC coding component 13 takes into account the physical layer functions of the communication channel.
[0063] Therefore, JISC decoding component 23 can also be used to receive channel-state information (CSI) H, N to define the channel model of the uplink channel of wireless sensor networks 1, 2.
[0064] Therefore, JISC coding component 13 is used to infer the semantic processing feature vector based on the channel model of the uplink channel of wireless sensor networks 1 and 2. k Inferring the JISC-encoded feature vector U k .
[0065] When the characteristics of the wireless channel change over time (e.g., due to a decrease in signal-to-noise ratio), the (optional) uplink channel parameters transmitted by the sensor fusion device 2 via the feedback channel are useful. Using these parameters, each JISC coding component 13 can better encode semantic features U by taking into account the quality of its own channel as well as the channel quality of other sensor device arrangements 1. k .
[0066] More specifically, JISC encoding component 13 will use the semantic features from SFE encoding component 12 k The input is (possibly) the uplink channel parameters, and the output is the JISC-coded U. k (It can further compress the input semantic features to match the channel capacity), making it suitable for transmission over the wireless channel, specifically by (i) protecting the most relevant semantic features (making them robust to channel distortion) and (ii) adapting to the characteristics of the wireless channel (e.g., channel noise, multipath propagation, interference, etc.), and complementing the JISC coding of other sensor device arrangements 1 (similar to SFE, which is achieved without explicit coordination between devices; by properly training the DNNs, they know which sensor device arrangement 1 is good at what task, and which sensor device arrangement 1 has good / bad communication channel quality).
[0067] Please note that the JISC encoding is U. kThere may be correlations in sensor device arrangement 1. This could occur when the JISC encoder learns implicit cooperation by utilizing the positive superposition and interference of wireless signals generated by other sensor device arrangements 1.
[0068] The following will combine Figure 2 and Figure 3 A more detailed explanation of the design / settings of JISC coding component 13.
[0069] Transceiver component 14 is used to transmit the inferred JISC-coded feature vector U via the uplink channel k The data is sent to sensor fusion device 2 for distributed semantic processing of the accumulated sensor data in a wireless sensor network.
[0070] Please note that no specific encoding and modulation method is assumed to encode JISC U. k Mapping to physical signals does not assume the signal transmission method on the wireless channel, nor the signal reception and demodulation method of the sensor fusion device 2.
[0071] Separating the SFE encoding component 12 and the JISC encoding component 13 has several advantages: First, this provides the necessary modularity because SFE encoding component 12 and JISC encoding component 13 are implemented at two different functional layers, and they can be provided by two different entities (e.g., the application provider provides SFE encoding component 12, and the network provider provides JISC encoding component 13). Each component in Component 12 and Component 13 can also be replaced without significantly affecting system performance, provided that the input / output distribution of Component 12 and Component 13 remains similar to before.
[0072] Secondly, this increases flexibility because the JISC coding component 13 can be easily retrained to new source and channel distributions (e.g., when the sensor component 11 of the SFE coding component 12 (e.g., a camera) is moved from an indoor to an outdoor environment, or when one of the sensor device arrangements in the sensor device arrangement 1 is missing). Furthermore, since the output of the SFE coding component 12 has already been denoised, retraining the JISC coding component 13 requires significantly fewer communication and computational resources than training it completely from scratch.
[0073] The sensor fusion device 2 includes a transceiver component 24 and a joint inference-source-channel (JISC) decoding component 23, and may also include a channel estimation component 25.
[0074] The transceiver component 24 is used to receive the corresponding channel-distorted JISC-coded feature vector Z of the target variable Y from multiple sensor devices arranged 1 via the uplink channel of the wireless sensor network.
[0075] JISC decoding component 23 is used to infer an estimate of the target variable Y from the received feature vector Z (which may be affected by channel distortion) based on the channel model of the uplink channel of the wireless sensor network. .
[0076] Note that there is no explicit reconstruction of the transmitted JISC encoding before it is fed to the JISC decoding component 23. This contrasts with the traditional joint source-channel coding (JSCC) framework.
[0077] The channel estimation component 25 can be used to estimate the channel model of the uplink channel of the wireless sensor networks 1 and 2, and provide the corresponding CSI H and N to the corresponding sensor device arrangement 1 and JISC decoding component 23.
[0078] Figure 2 Showing more details Figure 1 JISC encoding component 13.
[0079] As described above, the uplink channel parameters can be transmitted to the corresponding sensor device arrangement 1. This feedback information can help the JISC coding component 13 better adapt to fluctuations in channel conditions. However, how to use this additional information is not obvious. One possible design / setup could be based on the attention module.
[0080] More specifically, JISC encoding component 13 (and the same JISC decoding component 23) may include alternating sequences of feature modules 131 and attention modules 132.
[0081] Based on the above, the JISC coding component 13 can also be used to receive CSI H and N from the sensor fusion device 2 to define the channel model of the uplink channel of the wireless sensor networks 1 and 2.
[0082] The corresponding feature module 131 may include a DNN, especially a traditional fully connected DNN, for processing the received feature vector. k Inferring feature vectors for semantic processing k And output the inferred feature vector. k .
[0083] In contrast, the corresponding attention module 132 utilizes CSI H, N, which define the channel model.
[0084] Figure 3 Showing more details Figure 2 Attention module 132.
[0085] The corresponding attention module 132 may include a DNN 1321, a softmax component 1322, and a linear combination component 1323.
[0086] The DNN 1321 (especially a traditional fully connected DNN) and the softmax component 1322 work together to infer the feature vector indicating reception based on the channel model (e.g., denoted by CSI H, N) of the uplink channel of the wireless sensor networks 1 and 2. k The feature relevance weight vector w i .
[0087] The vector w of attention weights i This describes the relevance of each feature. For example, the attention weight vector w i It can include numbers between 0 and 1, where values close to 1 describe relevant features and values close to 0 describe irrelevant features.
[0088] Specifically, the inferred weight vector w i It can include normalized vectors.
[0089] Linear combination component 1323 is used for the inferred weight vector w i and the received feature vector k Perform a linear combination to obtain a new attention-weighted feature vector.
[0090] The attention module 132 controls the level of protection for the most relevant information. For example, if channel conditions are good, the attention module 132 tends to allow the transmission of all features. Conversely, when severe channel distortion occurs, the attention module 132 suppresses less relevant information to better protect core features. This allows for flexible adaptation to fluctuations in channel conditions.
[0091] Figure 4 Showing more details Figure 1 Transceivers 14 and 24.
[0092] Although no specific modulation and demodulation techniques are assumed, transceivers 12 and 24 can still be specified based on specific physical layer implementations, such as orthogonal frequency-division multiplexing (OFDM).
[0093] On one side of the sensor device arrangement 1, each JISC encoding component 13 can be trained to encode the SFE. k It is directly mapped to an OFDM symbol sequence represented as complex values in the in-phase / quadrature (I / Q) domain. Furthermore, the transceiver component 14 may include one or more of the following: a power normalization unit 141, a quantization unit 142, and an OFDM modulation unit 143.
[0094] Therefore, the output OFDM symbols can be normalized to keep the maximum signal power within some predefined limits, and / or, if certain fixed symbol constellations (e.g., QAM) are used, the OFDM symbols can be quantized. The generated OFDM symbols can be converted into baseband waveforms, modulated onto high-frequency subcarriers, and transmitted over a shared multiple access narrowband channel. Signal modulation can be performed using conventional IFFT-based modulation techniques, a cyclic prefix (CP) can be added to reduce inter-symbol interference (ISI), and sensor device arrangement 1 can transmit simultaneously on the same subcarrier in a synchronous manner. In this case, the transmitted OFDM signals will be superimposed to form a new combined OFDM signal.
[0095] On one side of the sensor fusion device 2, the transceiver component 24 may include an OFDM demodulation unit 241 for demodulating channel distortion OFDM symbols Z.
[0096] Therefore, traditional FFT-based demodulation techniques can be used to demodulate the signal, and the recovered channel-distorted OFDM symbol Z can be directly fed into the JISC decoding component 23 to infer the estimate of the target variable Y. .
[0097] Figures 5 to 7 The joint training of SFE encoding component 12 and SFE decoding component 22 is shown.
[0098] This first training process does not depend on the wireless channel, and therefore can be fully executed at the application layer in conjunction with the joint SFE decoding component 22.
[0099] In short, the SFE encoding component 12 is trained to infer / extract only those semantic features from the labeled training data that are relevant to the current context and complement the semantic features inferred by other sensor device arrangements 1. k.
[0100] During the forward propagation (see...) Figure 6 SFE encoding component 12 is based on sensor data X, including labeled training data. kPerform the above inference.
[0101] In other words, the SFE encoding component 12 can be used to process sensor data X, including labeled training data. k Inferring feature vectors for semantic processing k ; and the inferred feature vector k Send to sensor fusion device 2.
[0102] Please note that, especially for training, the sensor fusion device 2 may also include an SFE decoding component 22 for receiving semantically processed feature vectors of labeled training data from the multiple sensor device arrangement 1. k ; from the received feature vector k Inferring the estimate of the target variable Y Estimates based on target variable Y and inferences from target variable Y. The error vector is calculated at the output of the SFE decoding component 22. NN Based on the calculated error vector NN Update the weights of its DNN.
[0103] During the reverse transmission process (see...) Figure 7 The SFE decoding component 22 can also be used to send the corresponding error vector to the corresponding sensor device arrangement 1 at the output of the corresponding sensor device arrangement 1.
[0104] The SFE encoding component 12 can also be used to receive error vectors from the sensor fusion device 2 at the output of the sensor device arrangement 1; and update the weights of its DNN based on the received error vectors.
[0105] At the end of the first training process, the joint SFE decoding component 22 is discarded, and the weights of the DNN of the SFE encoding component 12 are frozen (these weights are no longer updated).
[0106] Figures 8 to 10 The joint training of JISC encoding component 13 and JISC decoding component 23 is shown.
[0107] In short, the JISC coding component 13 is trained to generate JISC codes suitable for transmission over a wireless channel. k Furthermore, the JISC decoding component 23 is trained to infer the target variable from the distorted data.
[0108] This second training process relies on wireless channels, whether real or simulated.
[0109] During the forward propagation (see...) Figure 9 The corresponding sensor device arrangement 1 and sensor fusion device 2 perform the above inference. However, please note that the inference performed by the SFE encoding component 12 of the corresponding sensor device arrangement 1 is based on sensor data X including labeled training data. k And the frozen weights of the first training step.
[0110] In other words, the training samples are propagated through the frozen SFE encoding component 12, JISC encoding component 13, and transceiver component 14. Next, the generated signal is simultaneously transmitted through the uplink channel and received at the sensor fusion device 2, where the transceiver component 24 demodulates the signal and feeds the retrieved data to the JISC decoding component 23 to infer the target variable.
[0111] To enable appropriate updates to the involved DNN, all forward operations are recorded and stored in the memory of the respective devices 1 and 2. In addition, sensor fusion device 2 can estimate the parameters of the wireless channel and (optionally) transmit them to all sensor device arrangements 1. For example, this can be based on... Figures 8 to 10 Additional record / estimation components 16, 26, and 27 are shown.
[0112] During the reverse transmission process (see...) Figure 10 JISC decoding component 23 can also be used for estimation based on the target variable Y and the inference of the target variable Y. The error vector (the value of the loss function) is calculated at the output of the JISC decoding component 23; the weights of its DNN are updated based on the calculated error vector.
[0113] Transceiver component 24 can also be used to, based on the calculated error vector, the recorded forward operation, and the channel model of the uplink channel, in the corresponding sensor device arrangement 1 (note that, through...). Figure 8 The output of the recording / estimation components 26 and 27 (dashed arrows) estimates the corresponding error vector; the corresponding error vector is then transmitted to the corresponding sensor device arrangement 1 via the downlink channel of the wireless sensor network.
[0114] Transceiver component 14 can also be used to receive an error vector from sensor fusion device 2 at the output of sensor device arrangement 1 via the downlink channel of the wireless sensor network; based on the received error vector and the recorded forward operation, in JISC encoding component 13 (note that, through...) Figure 8 The output of the recording / estimation component 16 (dashed arrow) estimates the error vector.
[0115] JISC encoding component 13 can also be used to update the weights of its DNN based on the estimated error vector.
[0116] Repeat the second training process described above until convergence. To improve the system's robustness to fluctuations in channel conditions, the DNN involved should ideally be trained under varying channel parameters.
[0117] Figure 11 A flowchart is shown relating to the method 3 for operating the sensor device arrangement 1 and the method 4 for operating the sensor fusion device 2 according to the present invention.
[0118] Illustrative examples describe wireless sensor networks 1 and 2 for distributed semantic processing of sensor data.
[0119] The wireless sensor networks 1 and 2 include multiple (two in this case) sensor device arrangements 1 and sensor fusion devices 2.
[0120] The corresponding sensor device arrangement 1 is used to perform the method 3 for operating the sensor device arrangement 1.
[0121] Method 3 includes generating sensor data X for a target variable Y. k Steps, sensor data X k Includes one or more features k .
[0122] Method 3 also includes data from sensor data X k Inferring the feature vectors of semantic processing. k The steps.
[0123] Method 3 also includes using the channel model of the uplink channel of the wireless sensor network to infer the feature vectors of semantic processing. k Inferring the feature vector U of JISC encoding 33 k The steps.
[0124] Method 3 also includes transmitting the inferred JISC-coded feature vector U via the uplink channel. k Sending 34 to sensor fusion device 2 for distributed semantic processing of accumulated sensor data in a wireless sensor network.
[0125] The sensor fusion device 2 is used to perform the method 4 for operating the sensor fusion device 2.
[0126] Method 4 includes the step of receiving the corresponding channel-distorted JISC-coded feature vector Z of the target variable Y from a plurality of sensor devices arranged 1 via the uplink channel of a wireless sensor network.
[0127] Method 4 also includes inferring an estimate of the target variable Y from the received feature vector Z based on the channel model of the uplink channel of the wireless sensor network. The steps.
[0128] In summary, wireless sensor networks 1 and 2 are proposed for distributed joint inference source-channel coding in a multiple access network of devices consisting of one or more sensor device arrangements 1 (“edge devices”) and sensor fusion devices 2 (“parent devices”), and provide the following: Distributed (joint) semantic feature extraction (SFE) At the application layer of each edge device; Use the input data as the input source and output semantic features; The aim is to extract only those features relevant to a given task; The complementarity of information between edge devices is taken into account (without explicit coordination).
[0129] Joint inference-source-channel (JISC) coding At the physical layer of each edge device; The semantic features from the corresponding SFE and (possibly) uplink channel parameters are taken as input, and the JISC code is output. It takes into account the degree of inference / relevance of a given task (distributed inference), locally observed data samples, and implicitly considered data samples observed by other devices (distributed source coding), as well as the adaptability to the channel (channel coding).
[0130] Joint inference-source-channel (JISC) decoding The received JISC encoding (which may be affected by channel distortion) and (possibly) uplink channel parameters are taken as input, and the estimated target variable is output.
[0131] The invention has been described in conjunction with various exemplary implementations. However, based on a study of the drawings, the invention, and the independent claims, those skilled in the art will be able to understand and implement other variations when practicing the claimed subject matter. In the claims and the description, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" does not exclude a plurality. A single element or other unit may fulfill the function of several entities or items described in the claims. The enumeration of certain measures in dissimilar dependent claims does not imply that combinations of these measures cannot be effectively used in advantageous implementations. Computer programs may be stored or distributed on suitable media, such as optical storage media or solid-state media provided together with or as part of other hardware, and may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunications systems.
Claims
1. A method for processing sensor data (X) in a wireless sensor network. k The sensor device arrangement for distributed semantic processing (1) is characterized by, The sensor device arrangement (1) includes Sensor assembly (11), for The sensor data (X) that generate the target variable (Y) k The sensor data (X) k ) includes one or more features ( k ); The semantic feature extraction (SFE) encoding component (12) is used for From the sensor data (X) k Inferring the feature vector of semantic processing ( k ); The joint inference-source-channel (JISC) coding component (13) is used for... Based on the channel model of the uplink channel of the wireless sensor network, the inferred semantic processing feature vector ( k Inferring the JISC-encoded feature vector (U k ); Transceiver component (14), used for The inferred JISC-coded feature vector (U) is transmitted through the uplink channel. k The data is sent to the sensor fusion device (2) for distributed semantic processing of the accumulated sensor data in the wireless sensor network.
2. The sensor device arrangement (1) according to claim 1, characterized in that, The SFE encoding component (12) includes a deep neural network (DNN) for use with Receives the sensor data (X) at its input terminal k ); From the received sensor data (X) k Inferring the feature vector of semantic processing ( k ); At its output, the inferred feature vector ( k Perform forward operations.
3. The sensor device arrangement (1) according to claim 1 or claim 2, characterized in that, The JISC coding component (13) comprises a sequence of alternating feature modules (131) and attention modules (132); Including the corresponding feature module (131) of the DNN, for From the received feature vector ( k Inferring the feature vector of semantic processing ( k ); Output the inferred feature vector ( k ); Including the corresponding attention module (132) of the DNN, for Based on the channel model of the uplink channel of the wireless sensor network, infer the feature vector indicating reception ( k The relevance weight vector of the features (w) i ); Output the inferred weight vector (w) i ) and the received feature vector ( k A linear combination of ).
4. The sensor device arrangement (1) according to claim 3, characterized in that, The inferred weight vector (w) i This includes normalized vectors.
5. The sensor device arrangement (1) according to any one of claims 1 to 4, characterized in that, The JISC encoding component (13) is also used for Channel-state information (CSI) (H, N) is received from the sensor fusion device (2) to define the channel model of the uplink channel of the wireless sensor network.
6. The sensor device arrangement (1) according to any one of claims 1 to 5, characterized in that, The SFE encoding component (12) is also used for From the sensor data (X) including labeled training data k Inferring the feature vector of semantic processing ( k ); The inferred feature vector ( k ) is sent to the sensor fusion device (2); The sensor fusion device (2) receives the error vector from the output terminal of the sensor device arrangement (1); The weights of its DNN are updated based on the received error vector.
7. The sensor device arrangement (1) according to claim 6, characterized in that, The transceiver component (14) is also used for Record all forward operations associated with the labeled training data; The sensor fusion device (2) receives the error vector from the sensor device arrangement (1) at the output end of the sensor device arrangement (1) via the downlink channel of the wireless sensor network; Based on the received error vector and the forward operation of the record, the error vector is estimated at the output of the JISC encoding component (13); The JISC encoding component (13) is also used for The weights of the DNN are updated based on the estimated error vector.
8. The sensor device arrangement (1) according to any one of claims 1 to 7, characterized in that, The transceiver component (14) includes one or more of the following: Power normalization unit (141); Quantization unit (142); Orthogonal frequency-division multiplexing (OFDM) modulation unit (143).
9. A sensor fusion device (2) for distributed semantic processing of accumulated sensor data in a wireless sensor network, characterized in that, The sensor fusion device (2) includes Transceiver component (24), used for The corresponding channel-distorted JISC-coded feature vector (Z) of the target variable (Y) is received from multiple sensor devices arranged (1) via the uplink channel of the wireless sensor network. The joint inference-source-channel (JISC) decoding component (23) is used for... Based on the channel model of the uplink channel of the wireless sensor network, an estimate of the target variable (Y) is inferred from the received feature vector (Z). ).
10. The sensor fusion device (2) according to claim 9, characterized in that, The JISC decoding component (23) is also used for Receive channel-state information (CSI) (H, N) to define the channel model of the uplink channel of the wireless sensor network.
11. The sensor fusion device (2) according to claim 9 or claim 10, characterized in that, The sensor fusion device (2) also includes an SFE decoding component (22) for use in... The feature vectors corresponding to the semantic processing of the labeled training data are received from the arrangement of the plurality of sensor devices (1). k ); From the received feature vector ( k Inferring the estimate of the target variable (Y) ); Based on the target variable (Y) and the inferred estimate of the target variable (Y) ) Calculate the error vector at the output of the SFE decoding component (22) NN ); Based on the calculated error vector ( NN Update the weights of its DNN; The corresponding error vector is sent to the corresponding sensor device arrangement (1) at the output terminal of the corresponding sensor device arrangement (1).
12. The sensor fusion device (2) according to any one of claims 9 to 11, characterized in that, The transceiver component (24) is also used for Record all forward operations associated with the labeled training data; The JISC decoding component (23) is also used for Based on the target variable (Y) and the inferred estimate of the target variable (Y) ) Calculate the error vector at the output of the JISC decoding component (23); The weights of the DNN are updated based on the calculated error vector; The transceiver component (24) is also used for Based on the calculated error vector, the recorded forward operation, and the channel model of the uplink channel, the corresponding error vector is estimated at the output of the corresponding sensor device arrangement (1); The corresponding error vector is transmitted to the corresponding sensor device arrangement (1) via the downlink channel of the wireless sensor network.
13. The sensor fusion device (2) according to any one of claims 9 to 12, characterized in that, The transceiver component (24) includes an orthogonal frequency-division multiplexing (OFDM) demodulation unit (241).
14. A wireless sensor network (1, 2) for distributed semantic processing of sensor data, characterized in that, include Arrangement of multiple sensor devices according to any one of claims 1 to 8 (1); Sensor fusion device (2) according to any one of claims 9 to 13; The sensor fusion device (2) and the corresponding sensor device arrangement (1) are in wireless network communication.
15. A method for operating a sensor device arrangement (1) to process sensor data (X) in a wireless sensor network. k The method (3) for distributed semantic processing is characterized by, The method (3) includes The sensor data (X) that generates the target variable (Y) is generated (31). k The sensor data (X) k ) includes one or more features ( k ); From the sensor data (X) k Inference (32) semantic processing feature vectors ( k ); Based on the channel model of the uplink channel of the wireless sensor network, the inferred semantic processing feature vector ( k Inferring (33) JISC-encoded feature vectors (U k ); The inferred JISC-coded feature vector (U) is transmitted through the uplink channel. k (34) is sent to the sensor fusion device (2) for distributed semantic processing of accumulated sensor data in the wireless sensor network.
16. A method (4) for operating a sensor fusion device (2) to perform distributed semantic processing on accumulated sensor data in a wireless sensor network, characterized in that, The method (4) includes The corresponding channel-distorted JISC-coded feature vector (Z) of the target variable (Y) is received (41) from multiple sensor devices arranged (1) via the uplink channel of the wireless sensor network. Based on the channel model of the uplink channel of the wireless sensor network, the estimate of the target variable (Y) is inferred (42) from the received feature vector (Z). ).
17. A computer program, characterized in that, Includes program code that, when run on a computer, performs the method according to claim 15 or claim 16.