Multi-dimensional data processing system and method based on edge calculation

Abstract

The invention relates to the technical field of edge computing, in particular to a multi-dimensional data processing system and method based on edge computing, and the method comprises the steps: monitoring the state of edge computing equipment in real time, constructing a multi-objective optimization model, and dynamically allocating data processing tasks; performing feature extraction on the received multi-dimensional data, and performing feature fusion through a modal attention mechanism; real-time anomaly identification of the fusion features is realized by using an edge side anomaly detection model; incremental training and parameter updating of the model are completed through a distillation learning method; and compression and homomorphic encryption are performed on transmission data, so that the data communication security is ensured. According to the method, efficient processing, accurate anomaly detection and secure communication of edge-side multi-dimensional data are realized, and the response speed, the intelligent level and the data security of the system are improved.

Description

Technical Field

[0001] The present invention relates to the field of edge computing technology, and specifically to a multidimensional data processing system and method based on edge computing. Background Art

[0002] With the rapid development of the Internet of Things (IoT) and edge computing, massive amounts of multi-source data need to be processed efficiently and in real time at the edge, close to the terminal. Existing technologies for processing multi-dimensional and multi-modal data streams in IoT and edge computing scenarios lack integrated and collaborative capabilities.

[0003] Traditional methods often design and implement task scheduling, feature processing, anomaly detection and data security separately, lacking systematic optimization for the entire process. This leads to low collaboration efficiency and serious information silos between processing links, making it difficult to adapt to the real-time dynamic needs of large-scale heterogeneous devices and multi-dimensional data streams in the Internet of Things environment.

[0004] Existing solutions lack a data processing system capable of achieving highly coordinated and intelligently adaptive multi-process collaboration in complex and dynamic environments, while also ensuring high performance and security. Therefore, building a multidimensional data processing system capable of full-process intelligent collaborative processing within an edge computing environment has become a complex challenge that existing technologies need to address.

[0005] In view of this, the present application proposes a multi-dimensional data processing system and method based on edge computing. Summary of the Invention

[0006] To achieve the above objectives, the present invention provides a multi-dimensional data processing system and method based on edge computing. The specific technical solutions are as follows:

[0007] Multi-dimensional data processing methods based on edge computing include:

[0008] Monitor the computing power, network status, and load status of each edge computing device in real time, build and solve a multi-objective optimization scheduling model, and dynamically allocate data processing and computing requests from IoT devices to the optimal edge computing device.

[0009] Each edge computing device performs feature extraction on the received multidimensional data and fuses the extracted multidimensional data features using the modal attention mechanism;

[0010] Based on the fused multi-dimensional data features, the anomaly detection model in the edge computing device is used to detect abnormal behaviors in the multi-dimensional data features in real time on the edge computing device.

[0011] Combined with recent data cached by edge computing devices, the anomaly detection model in the edge computing device is locally incrementally trained and parameter updated through distillation learning.

[0012] Compress the collected data and build an end-to-end compression and encryption mechanism to perform data compression and encryption on the communication between IoT devices and edge computing devices.

[0013] Preferably, a multi-objective optimization scheduling model is constructed, the edge node state vector is defined, a computing task-device mapping delay function is constructed, computing tasks are assigned to edge computing devices according to the task data size and computing requirements corresponding to the computing tasks, and the total response delay is calculated.

[0014] Preferably, a multi-objective optimization scheduling model is constructed, wherein the optimization objectives include minimizing the average computing task delay and maximizing the number of computing tasks completed per unit time;

[0015] A multi-objective optimization scheduling model is constructed based on minimizing the average computing task delay and maximizing the number of computing tasks completed per unit time, and the constraints of the multi-objective optimization scheduling model are set.

[0016] Preferably, an improved ant colony algorithm is used to solve the multi-objective optimization scheduling model in the edge computing scenario;

[0017] The steps of improving the ant colony algorithm to solve the multi-objective optimization scheduling model include: initializing the ant colony parameters and setting the heuristic factor based on the edge computing device status to guide the task-node allocation; in each iteration, the ants select the computing node for the task based on the device resources and delay to ensure that the load does not exceed the limit; after completing the scheduling, the fitness is calculated, and the minimum delay and maximum throughput are comprehensively evaluated and the pheromone is updated, and the quality of the local search optimization solution is introduced; iterate until convergence or the upper limit is reached, and output the optimal scheduling strategy.

[0018] Preferably, each edge computing device performs feature extraction on the received data;

[0019] Extract image features, including image brightness changes, segmentation of objects in the image, and motion tracking;

[0020] Extracting the time-series frequency features of the audio data. The extracted time-series frequency features include the energy distribution characteristics, spectrum change patterns, and voiceprint characteristics of the audio.

[0021] Extract environmental sensor data stream features; environmental sensor data stream features include data change trend features, data periodicity features, and data extreme value and boundary features.

[0022] Preferably, a feature fusion strategy is constructed to perform feature fusion on feature data containing the same task identifier, wherein the same task identifier includes automatically generating a unique task ID for each task when the task is created, and all data belonging to the same task are marked with the same task ID;

[0023] When a task contains only a single type of data, all feature data of the same type are fused;

[0024] When a task contains multiple types of data, vector fusion is performed on all feature data under the same task identifier. Vector fusion is performed on the data of the same type first, and then all types of feature vector data are spliced ​​and fused.

[0025] Preferably, an anomaly detection model is constructed based on a variational autoencoder (VAE) model, and a data set is constructed to train the anomaly detection model.

[0026] In the edge computing device, the decoder network of the variational autoencoder (VAE) model obtains the reconstruction vector in real time based on the acquired multidimensional feature vector and calculates the anomaly score to quantify the degree of anomaly of the current multidimensional feature vector. The anomaly score of the multidimensional feature vector is used as the confidence estimate of the sample. The higher the anomaly score, the higher the confidence that the multidimensional feature vector is anomaly.

[0027] An anomaly determination threshold is set. If the anomaly score is greater than the anomaly determination threshold, it is determined that the IoT device has abnormal behavior in the collected multi-dimensional feature vector.

[0028] Preferably, recent multimodal data and the prediction results generated by the anomaly detection model in the past reasoning process are periodically cached in the edge computing device, and combined with some manually annotated feedback labels to form a set of sample triplets for the distillation learning process;

[0029] Define a teacher model and a student model. The teacher model is the anomaly detection model originally deployed in the edge computing device, which saves the predicted probability distribution of each sample in the previous cycle. The student model is a copy of the model to be optimized in the current training cycle, and the student model parameters are updated and learned locally.

[0030] A distillation learning method is used to construct a teacher-student model. The optimization process of the distillation learning method is regulated by supervision loss and distillation loss.

[0031] Preferably, at the IoT device end, the collected data is compressed using a compressed sensing algorithm;

[0032] The compressed data is encrypted using the lightweight Paillier encryption algorithm to obtain ciphertext, which is then transmitted to the edge computing device.

[0033] After receiving the ciphertext, the edge computing device uses the private key to decrypt it, recover the compressed data, and uses the sparse reconstruction algorithm to recover the original data from the measurement vector.

[0034] A multidimensional data processing system based on edge computing, which is used to implement the multidimensional data processing method based on edge computing, includes: an edge computing device scheduling module, a feature extraction and fusion module, an anomaly detection module, an incremental update module, and a compression and encryption module;

[0035] The edge computing device scheduling module is used to monitor the computing power, network status and load status of each edge computing device in real time, build and solve a multi-objective optimization scheduling model, and dynamically allocate data processing and computing requests from IoT devices to the optimal edge computing device;

[0036] The feature extraction and fusion module is used for each edge computing device to perform feature extraction on the received multidimensional data and fuse the extracted multidimensional data features using the modal attention mechanism;

[0037] The anomaly detection module detects abnormal behaviors in the multidimensional data features in real time on the edge computing device through the anomaly detection model in the edge computing device based on the fused multidimensional data features;

[0038] The incremental update module combines the recent data cached by the edge computing device and performs local incremental training and parameter update on the anomaly detection model in the edge computing device through distillation learning;

[0039] The compression and encryption module compresses the collected data and builds an end-to-end compression and encryption mechanism to perform data compression and encryption processing on the communication between the IoT device and the edge computing device.

[0040] Beneficial effects of the invention: This application comprehensively considers the computing power, network status and load conditions of edge computing devices, establishes a multi-objective optimization scheduling model, and can achieve the optimal allocation of multi-dimensional data processing tasks of the Internet of Things, reduce resource waste, improve system response efficiency and load balancing level, and ensure the real-time and stability of edge computing.

[0041] This application uses a dedicated deep network structure to extract features from different modal data, so that all types of data features can be efficiently captured; by splicing and fusing features, the complementary information in multimodal data is retained, and the accuracy and robustness of subsequent detection tasks are improved.

[0042] This application deploys anomaly detection models on edge computing devices, which can directly perform real-time analysis on the fused multi-dimensional data features locally, reducing the demand for data to be uploaded to the cloud and improving system response speed.

[0043] This application combines recent cached data on the edge side with soft and hard label information, and performs lightweight incremental training on the detection model through knowledge distillation, avoiding full backhaul and large-scale centralized training, significantly saving communication and computing resources.

[0044] This application effectively reduces the data volume of uploaded features and results through compressed sensing compression technology; it uses the Paillier lightweight homomorphic encryption method to encrypt key data. While ensuring data security and privacy, it still supports the upper-level system's analysis and processing of ciphertext data, improves data trusted computing capabilities, and meets the needs of multi-party data collaborative analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 Flowchart of the multi-dimensional data processing method based on edge computing provided by the present invention;

[0046] Figure 2 A flowchart for constructing and solving the multi-objective optimization scheduling model provided by the present invention;

[0047] Figure 3 The multimodal data feature extraction and fusion flow chart provided by the present invention;

[0048] Figure 4 The flow chart of edge anomaly detection based on multidimensional features provided by the present invention;

[0049] Figure 5 This is a structural diagram of the multi-dimensional data processing system based on edge computing provided by the present invention. DETAILED DESCRIPTION

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0051] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can also be implemented in other ways different from those described herein. Those skilled in the art can make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0052] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0053] Example 1

[0054] Reference Figure 1 , which is the first embodiment of the present invention, provides a multi-dimensional data processing method based on edge computing.

[0055] Step 1: Monitor the computing power, network status, and load status of each edge computing device in real time, build and solve a multi-objective optimization scheduling model, and dynamically allocate data processing and computing requests from IoT devices to the optimal edge computing device. Figure 2 , which is the flow chart of building and solving the multi-objective optimization scheduling model in this step.

[0056] In edge computing scenarios, task scheduling faces challenges such as node heterogeneity, network dynamics, and limited computing resources. In order to achieve multi-objective optimization of minimizing the average task response time and maximizing the total amount of system task processing, this application constructs a multi-objective optimization scheduling model, covering task-node matching delay evaluation, resource constraint modeling, and joint objective function optimization.

[0057] Define the edge node state vector, assume there are N edge computing devices, and record all edge computing devices as a set ε={e1,e2,...,e N}, edge computing device e i Status R i Defined as: R i =(C i ,B i ,D i ,L i ); where C i Represents edge computing device e i Currently available computing power; B i Represents edge computing device e i Current available network bandwidth; D i For edge computing devices i Current network transmission delay, L i Represents edge computing device e i The current average number of queued tasks, specifically used to reflect the edge computing device load, i∈[1,N]. Edge computing device e i Status R i Used to perceive the resource and load status of edge computing devices in real time, providing a decision basis for scheduling strategies.

[0058] Construct a computing task-device mapping delay function, set M tasks to be scheduled, and record all tasks to be scheduled as a waiting task set Any computing task t j Corresponding task data size s j and computational requirements w j , j∈[1,M], the computation task t j Assigned to edge computing devices i Calculate the total response delay T i,j : Here, λ is the queue load factor, which is used to control the impact of the number of queued tasks on latency. For example, λ can be between 0.5 and 2. The constructed computing task-device mapping delay function comprehensively considers the impact of task data transmission, communication delay, and execution time, forming an evaluation function for scheduling performance.

[0059] Construct a multi-objective optimization scheduling model. The optimization objectives include minimizing the average computing task delay and maximizing the number of computing tasks completed per unit time. Construct a model that minimizes the average computing task delay: Construct a model to maximize the number of computing tasks completed per unit time: Among them, x i,j ∈{0,1},x i,j Indicates whether to assign computing tasks to edge computing devices e i ; is an indicator function, which is 1 when the condition in the brackets is met and 0 otherwise; ΔT is the preset computing task completion time threshold.

[0060] Construct a multi-objective optimization scheduling model. The objective function of the multi-objective optimization scheduling model adopts a weighted strategy to build a multi-objective optimization scheduling model based on minimizing the average computing task delay and maximizing the number of computing tasks completed per unit time:

[0061]

[0062] Among them, the weight α∈[0,1]; illustratively, the weight α value is set according to the current urgency of minimizing the average computing task delay or maximizing the number of computing tasks completed per unit time. If the urgency of the two optimization goals is equal, α can be set to 0.5. In other cases, the weight α value of the optimization goal with a higher urgency is set to be greater than the weight value of the optimization goal with a lower urgency.

[0063] The constructed multi-objective optimization scheduling model can achieve dynamic balanced scheduling of edge computing devices in terms of computing delay and data throughput.

[0064] In the constructed multi-objective optimization scheduling model, the scheduling of edge computing devices must also meet the resource constraints of the edge computing devices, including: each computing task can only be assigned to one edge computing device: And the computing load of each edge computing device cannot exceed its maximum computing capacity: where θ is the upper limit of available computation time within a scheduling period.

[0065] In order to solve the multi-objective optimization scheduling model in edge computing scenarios, an improved ant colony algorithm is adopted. First, the ant colony parameters are initialized, including pheromone concentration, volatility coefficient, and maximum number of iterations. At the same time, according to the status of edge computing devices, including computing power, bandwidth, delay, and number of queued tasks, a heuristic factor for task-node allocation is set to guide scheduling path selection.

[0066] In each round of iteration, the ants select the target edge computing devices for each task based on the current task set. The selection is based on the available resources and latency of the current device, and the matching of tasks and devices is completed under the premise that the computing load of the device does not exceed the upper limit.

[0067] After the scheduling plan is completed, the fitness of each plan is calculated, the minimum delay and maximum throughput objective functions are integrated, and the plan is evaluated using a weighted strategy.

[0068] The pheromone system updates based on the scheduling results. High-quality paths receive pheromone enhancements, while unsatisfactory paths experience pheromone attenuation. Local search strategies are also introduced to fine-tune the optimal solution and improve its quality. The pheromone system continuously iterates until convergence conditions or an iteration limit are reached, ultimately outputting an optimal scheduling strategy that balances system response time and task completion efficiency.

[0069] The improved ant colony algorithm effectively addresses the problems of node heterogeneity, resource dynamics and multi-objective conflicts in edge computing, and can efficiently solve the constructed multi-objective optimization scheduling model.

[0070] Step 2: Each edge computing device performs feature extraction on the received multidimensional data and fuses the extracted multidimensional data features using the modal attention mechanism. Figure 3 , which is the multimodal data feature extraction and fusion flow chart of this step.

[0071] Each edge computing device performs structured and categorized feature extraction on the received data (such as images, audio, and environmental sensor data streams) to obtain low-dimensional, high-representation modal feature representations.

[0072] Extract image data features, and edge computing devices receive image data transmitted by IoT devices For example, an RGB image captured by a surveillance camera; scale the image to a fixed size (for example, the image length and width are 224) and unify the image input format.

[0073] Use the MobileNet lightweight convolutional neural network deployed in edge computing devices to extract image features f img ;Extracted image features f img Including image brightness changes and object segmentation and motion tracking in the image.

[0074] The extracted image features retain the object distribution, regional differences and texture information in the image, and are suitable for edge vision scenarios such as scene recognition and video surveillance.

[0075] Perform feature extraction on audio data. The edge computing device receives audio data A(t) transmitted by the IoT device. Perform framing and windowing on the audio data A(t). For example, the framing can be set to 25ms per frame and 10ms frame shift. Obtain the spectrum of the audio data A(t) through short-time Fourier transform (STFT): S(f,t) = STFT(A(t)). Map the spectrum S(f,t) to the perceptual frequency band through the Mel filter bank: M(f′,t) = MelFilter(S(f,t)). Obtain the Mel spectrum.

[0076] A 2-layer 1D-CNN+BiGRU structure is used to extract the temporal frequency features f of audio data. aud , the temporal frequency characteristics f of the audio data aud Including audio energy distribution characteristics, spectrum change patterns and voiceprint characteristics;

[0077] The adopted 2-layer 1D-CNN+BiGRU structure can capture non-stationary signals such as alarms, impacts and abnormal sounds, which helps in sound anomaly detection and device status perception.

[0078] Extract the characteristics of environmental sensor data streams. Edge computing devices receive environmental sensor data streams transmitted by IoT devices, such as temperature, humidity, and PM2.5 concentration.

[0079] Construct a feature extraction model for environmental sensor data streams and use a three-layer stacked 1D-CNN+GAP (global average pooling) structure to extract the features f of environmental sensor data streams. sen ; The environmental sensor data stream feature f sen Including data change trend characteristics, data periodicity characteristics, and data extreme value and boundary characteristics.

[0080] The adopted three-layer stacked 1D-CNN+GAP (global average pooling) structure can effectively extract complex signal features such as mutation patterns, long-term and short-term trends in environmental sensor data, and improve sensitivity to device status fluctuations and environmental changes.

[0081] Perform feature fusion on feature data with the same task identifier. The same task identifier includes a unique task ID that is automatically generated when each task is created (such as a detection, an inspection, and an interaction). All data belonging to the task (such as image data, audio data, environmental sensor data, etc.) are marked with the same ID.

[0082] When a task contains only a single type of data (for example, a task only collects image frame sequences or only records environmental sensor data), all feature data of the same type are fused. For example, taking image data as an example: img =[f img,,1 ||f img,,2 ||...]; Vector fusion of data of the same type preserves the sequence information of the same data type at multiple time points or spatial points.

[0083] When a task contains multiple types of data, vector fusion is performed on all feature data under the same task identifier. Vector fusion is performed on the same type of data first. For example, take image data as an example: img =[f img,,1 ||f img,,2 ||...]; then concatenate and fuse all types of feature vector data: Where || represents the vector concatenation operation.

[0084] The various types of data features represented by vector fusion operations have the advantages of simple feature vector structure and high computational efficiency, making them more suitable for resource-constrained edge scenarios. At the same time, the fused multi-dimensional feature vectors can provide a rich complementary semantic information foundation for subsequent models.

[0085] Step 3: Based on the fused multidimensional data features, the anomaly detection model in the edge computing device is used to detect abnormal behaviors in the multidimensional data features in real time on the edge computing device. Figure 4 , which is the flow chart of edge anomaly detection based on multi-dimensional features in this step.

[0086] Based on the fused multi-dimensional feature vector F, the edge computing device uses the anomaly detection model in the edge computing device to identify abnormal behavior of real-time input. The anomaly detection model is constructed using a variational autoencoder (VAE), which is suitable for modeling and anomaly identification tasks of complex data distribution under unsupervised learning, and has the advantages of being lightweight and inferential.

[0087] The variational autoencoder VAE model consists of an encoder network q φ (z|F) and the decoder network p ζ (F|z), φ and ζ are parameters used to model and reconstruct the input multidimensional feature vector F.

[0088] Encoder network q φ (z|F) maps the concatenated multidimensional feature vector F to a parameter vector of a Gaussian distribution, that is, the mean μ and standard deviation σ of the output parameter vector, and then obtains the latent variable through reparameterization sampling: Where ⊙ represents the Hadamard product and ∈ is the standard normal distribution noise.

[0089] Construct a training set of prior knowledge, and manually divide the data samples collected by IoT devices into normal data samples and abnormal data samples. Each sample is accompanied by a label as the property of the sample; integrate all data and corresponding labels into a dataset, and divide the dataset into a training set, a validation set, and a test set. Use the training set to train the variational autoencoder (VAE) model, and use the validation set and test set to verify and tune the parameters of the variational autoencoder (VAE) model.

[0090] After the variational autoencoder VAE model is trained, the decoder network of the variational autoencoder VAE model in the edge computing device obtains the reconstruction vector in real time based on the obtained multi-dimensional feature vector F. And calculate the anomaly score S(F) to quantify the degree of anomaly of the current multidimensional feature vector: in, Represents the Euclidean squared distance. A larger value indicates that the multidimensional feature vector deviates further from the normal distribution in the feature space. The anomaly score of the multidimensional feature vector is used as the confidence estimate of the sample. The higher the anomaly score S(F), the higher the confidence that the multidimensional feature vector is anomaly.

[0091] Set the abnormality judgment threshold TH s , if S(F)>TH s , it is determined that the IoT device has abnormal behavior in the collected multi-dimensional feature vector and triggers the abnormal event processing logic. Exemplarily, the abnormal behavior processing logic can be set as follows: recording abnormal events and their characteristics; notifying the edge computing device management and control center; and transmitting data anomalies to the user end.

[0092] Step 4: Combined with the recent data cached by the edge computing device, the anomaly detection model in the edge computing device is locally incrementally trained and parameter updated through distillation learning.

[0093] In edge computing devices, recent multimodal data features (such as fusion vectors of images, audio, and sensor data) are periodically cached, as well as the prediction results (soft labels) generated by the anomaly detection model in the past reasoning process. These are combined with some manual annotations or feedback labels returned by the platform to form a set of sample triples. where f r is the fused multidimensional data feature, y r ∈{0,1} is the label (normal or abnormal), p r is the historical output probability distribution of the previous version of the teacher model for the sample, that is, the soft label output by the teacher model, r is the data index, and R is the data amount.

[0094] Define a teacher model and a student model. During training, the teacher model (TeacherModel) is the anomaly detection model originally deployed on the edge computing device. It stores its predicted probability distribution for each sample in the previous cycle, representing the model's "memory" of historical knowledge. The student model (StudentModel) is a copy of the model to be optimized in the current training cycle. Its parameters are updated and learned locally. The goal is to learn from the teacher model's understanding of historical knowledge while also integrating new labels to enhance its adaptability to new environmental data.

[0095] The training goal of building a teacher-student model is to allow the student model to simultaneously learn the current real-world annotation information (hard labels) and the teacher model's historical judgments (soft labels), thereby achieving the dual-objective optimization of "old knowledge retention + new knowledge absorption"; the optimization process consists of the following two loss functions, including supervision loss and distillation loss (KL divergence).

[0096] The supervised loss uses the traditional cross entropy loss function to fit the annotation label of the current sample, which is defined as follows:

[0097]

[0098] in, Represents the student model for sample f r The current predicted probability of .

[0099] In order to retain the original model’s memory of historical patterns, the distillation loss L is introduced KD , which is the KL divergence between the teacher model output and the student model's current prediction, is expressed as follows:

[0100]

[0101] in, Represents the teacher model for sample f r The current predicted probability of It is a very small constant that prevents the denominator from being zero, and is generally taken as 10 -8 .

[0102] The optimization objective of the student model is a combination of the two: L total =ρL KD +(1-ρ)L CE ; Among them, ρ∈[0,1] is the adjustment coefficient that controls the proportion of new and old knowledge. A larger ρ value (ρ>0.5) can be used in the early stage of training to strengthen knowledge learning, and the ρ value can be reduced in the later stage to retain the historical pattern.

[0103] In order to reduce the computing resource consumption of edge computing devices, a micro-batch update strategy is adopted, and only a selected subset of samples is used for parameter update. During the training process, only the rear structure of the model is adjusted, and the front feature extraction network structure is frozen, thereby reducing the computing overhead and maintaining feature consistency. After the model update is completed, samples that did not participate in the training are selected from the cached data as the validation set. If the performance indicator (such as F1 value) of the student model on the validation set is better than that of the teacher model, the student model is updated to the main model for the new round of inference, otherwise the original model state is maintained.

[0104] Through the above-mentioned model update mechanism, low-cost, highly adaptable evolutionary updates of computing models on edge computing devices are achieved; this mechanism can be widely used in anomaly detection scenarios in multi-source data environments, and is particularly suitable for edge computing architectures with limited network bandwidth or sensitive data security.

[0105] In order to achieve dynamic adaptive updating and continuous optimization of the anomaly detection model on the edge side, this step proposes a model update mechanism based on local incremental training of knowledge distillation. This mechanism uses recent observation data, historical reasoning results, and some manually labeled information cached locally on edge computing devices to build a lightweight, low-resource model update solution, enhancing the model's adaptability to environmental changes while avoiding the communication burden and computing delays caused by data uploading and full training.

[0106] Step 5: Compress the collected data and build an end-to-end compression and encryption mechanism to compress and encrypt the communication between IoT devices and edge computing devices.

[0107] On the IoT device side, the original sensor data (image, audio, environmental sensor data, etc.) is compressed using the compressed sensing algorithm; let the original data be vector Using sparse transform basis Its sparse representation is x=Ψs, where s is a sparse vector; introduce a low-dimensional measurement matrix Perform linear sampling to obtain the compression result: D = Φ·x = Φ·Ψ·s; where, This is the compressed data vector.

[0108] Compressed sensing algorithms can significantly reduce the amount of data that needs to be transmitted. They are particularly suitable for data types with high redundancy, such as videos and images. By compressing the data to be transmitted and completing the compression process before the data leaves the IoT device, it can alleviate the bandwidth pressure of the edge network to the greatest extent and reduce transmission time.

[0109] The compressed data D is encrypted using the lightweight Paillier encryption algorithm. The Paillier encryption algorithm supports homomorphic addition operations and is suitable for privacy protection analysis in subsequent edge computing. The specific encryption formula of the Paillier encryption algorithm is: E(D) = g D γ h modh 2 Where g is the public key generator, h is the product of a large prime number, and γ is a random number. All compressed data is encrypted into a ciphertext vector E(D) = {Ed1), E(d2), …} and transmitted to the edge computing device via a secure channel. The Paillier encryption algorithm is used to encrypt data collected by IoT devices, protecting data privacy from the source and effectively preventing data leakage risks at intermediate nodes.

[0110] After receiving the ciphertext vector E(D), the edge computing device uses the private key to decrypt it, recover the compressed data D, and then uses the sparse reconstruction algorithm to recover the original signal x.

[0111] Restore the original data on the edge computing device to ensure the accuracy of subsequent feature extraction and fusion analysis.

[0112] For data types that lack sparseness or require higher compression, Huffman coding can be introduced before or after compressed sensing to optimize compression efficiency. This approach uses probabilistic modeling and variable-length encoding on discrete data sequences to generate a compact bitstream. Huffman coding improves compression ratios, adapts to data diversity, and saves transmission resources.

[0113] This step starts compression from the IoT device to minimize bandwidth burden and adapt to the scenario of concurrent data reporting by large-scale devices; this step performs lightweight encryption based on compression to ensure that even if the data is intercepted, the content cannot be deciphered; the Paillier encryption constructed in this step is homomorphic, laying the foundation for subsequent privacy-preserving calculations; this step achieves a balance between data security and efficiency in the entire link of "data generation-transmission-reception-analysis".

[0114] Example 2

[0115] Reference Figure 5 , which is the second embodiment of the present invention, provides a multi-dimensional data processing system based on edge computing.

[0116] The system includes: an edge computing device scheduling module, a feature extraction and fusion module, an anomaly detection module, an incremental update module, and a compression and encryption module.

[0117] The edge computing device scheduling module is used to monitor the computing power, network status and load status of each edge computing device in real time, build and solve a multi-objective optimization scheduling model, and dynamically allocate data processing and computing requests from IoT devices to the optimal edge computing device.

[0118] The feature extraction and fusion module is used for each edge computing device to perform feature extraction on the received multidimensional data and fuse the extracted multidimensional data features using the modal attention mechanism.

[0119] The anomaly detection module detects abnormal behaviors in the multidimensional data features in real time on the edge computing device through the anomaly detection model in the edge computing device based on the fused multidimensional data features.

[0120] The incremental update module combines the recent data cached by the edge computing device and performs local incremental training and parameter updates on the anomaly detection model in the edge computing device through distillation learning.

[0121] The compression and encryption module compresses the collected data and builds an end-to-end compression and encryption mechanism to perform data compression and encryption processing on the communication between the IoT device and the edge computing device.

[0122] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0123] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific embodiments. The above-mentioned specific embodiments are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also change, modify, replace and modify the above-mentioned embodiments without departing from the scope of protection of the purpose of the present invention and the claims, and all of these are protected by the present invention.

Claims

1. A multidimensional data processing method based on edge computing, characterized in that: include: Monitor the computing power, network status, and load status of each edge computing device in real time, build and solve a multi-objective optimization scheduling model, and dynamically allocate data processing and computing requests from IoT devices to the optimal edge computing device. Each edge computing device performs feature extraction on the received multidimensional data and fuses the extracted multidimensional data features using the modal attention mechanism; Based on the fused multi-dimensional data features, the anomaly detection model in the edge computing device is used to detect abnormal behaviors in the multi-dimensional data features in real time on the edge computing device. Combined with recent data cached by edge computing devices, the anomaly detection model in the edge computing device is locally incrementally trained and parameter updated through distillation learning. Compress the collected data and build an end-to-end compression and encryption mechanism to perform data compression and encryption on the communication between IoT devices and edge computing devices.

2. The multidimensional data processing method based on edge computing according to claim 1, characterized in that: A multi-objective optimization scheduling model is constructed, the edge node state vector is defined, and a computing task-device mapping delay function is constructed. The computing tasks are assigned to edge computing devices according to the task data size and computing requirements corresponding to the computing tasks, and the total response delay is calculated.

3. The multidimensional data processing method based on edge computing according to claim 2, characterized in that: Construct a multi-objective optimization scheduling model, where the optimization objectives include minimizing the average computing task delay and maximizing the number of computing tasks completed per unit time; A multi-objective optimization scheduling model is constructed based on minimizing the average computing task delay and maximizing the number of computing tasks completed per unit time, and the constraints of the multi-objective optimization scheduling model are set.

4. The multidimensional data processing method based on edge computing according to claim 3 is characterized in that: Solve the multi-objective optimization scheduling model in edge computing scenarios using an improved ant colony algorithm; The steps of improving the ant colony algorithm to solve the multi-objective optimization scheduling model include: initializing the ant colony parameters and setting the heuristic factor based on the edge computing device status to guide the task-node allocation; In each iteration, the ants select computing nodes for the task based on device resources and latency to ensure that the load does not exceed the limit; after completing the scheduling, they calculate the fitness, evaluate and update the pheromone based on the minimum latency and maximum throughput, and introduce local search to optimize the quality of the solution; iterate until convergence or the upper limit, and output the optimal scheduling strategy.

5. The multidimensional data processing method based on edge computing according to claim 4 is characterized in that: Each edge computing device extracts features from the received data; Extract image features, including image brightness changes, segmentation of objects in the image, and motion tracking; Extracting the time-series frequency features of the audio data. The extracted time-series frequency features include the energy distribution characteristics, spectrum change patterns, and voiceprint characteristics of the audio. Extract features from environmental sensor data streams; The characteristics of environmental sensor data streams include data change trend characteristics, data periodicity characteristics, and data extreme value and boundary characteristics.

6. The multidimensional data processing method based on edge computing according to claim 5, characterized in that: Construct a feature fusion strategy to fuse feature data with the same task identifier. The same task identifier includes that each task automatically generates a unique task ID when the task is created, and all data belonging to the same task are marked with the same task ID. When a task contains only a single type of data, all feature data of the same type are fused; When a task contains multiple types of data, vector fusion is performed on all feature data under the same task identifier. Vector fusion is performed on the data of the same type first, and then all types of feature vector data are spliced ​​and fused.

7. The multidimensional data processing method based on edge computing according to claim 6, characterized in that: Build an anomaly detection model based on the variational autoencoder VAE model and construct a data set to train the anomaly detection model; In the edge computing device, the decoder network of the variational autoencoder (VAE) model obtains the reconstruction vector in real time based on the acquired multidimensional feature vector and calculates the anomaly score to quantify the degree of anomaly of the current multidimensional feature vector. The anomaly score of the multidimensional feature vector is used as the confidence estimate of the sample. The higher the anomaly score, the higher the confidence that the multidimensional feature vector is anomaly. An anomaly determination threshold is set. If the anomaly score is greater than the anomaly determination threshold, it is determined that the IoT device has abnormal behavior in the collected multi-dimensional feature vector.

8. The multidimensional data processing method based on edge computing according to claim 7, characterized in that: Recent multimodal data and the prediction results generated by the anomaly detection model during past reasoning are periodically cached in the edge computing device. Combined with some manually annotated feedback labels, a set of sample triplets is formed for the distillation learning process. Define a teacher model and a student model. The teacher model is the anomaly detection model originally deployed in the edge computing device, which saves the predicted probability distribution of each sample in the previous cycle. The student model is a copy of the model to be optimized in the current training cycle, and the student model parameters are updated and learned locally. A distillation learning method is used to construct a teacher-student model. The optimization process of the distillation learning method is regulated by supervision loss and distillation loss.

9. The multidimensional data processing method based on edge computing according to claim 8, characterized in that: On the IoT device side, the collected data is compressed using a compressed sensing algorithm; The compressed data is encrypted using the lightweight Paillier encryption algorithm to obtain ciphertext, which is then transmitted to the edge computing device. After receiving the ciphertext, the edge computing device uses the private key to decrypt it, recover the compressed data, and uses the sparse reconstruction algorithm to recover the original data from the measurement vector.

10. A multidimensional data processing system based on edge computing, which is used to implement the multidimensional data processing method based on edge computing according to any one of claims 1 to 9, characterized in that: include: Edge computing device scheduling module, feature extraction and fusion module, anomaly detection module, incremental update module, and compression and encryption module; The edge computing device scheduling module is used to monitor the computing power, network status and load status of each edge computing device in real time, build and solve a multi-objective optimization scheduling model, and dynamically allocate data processing and computing requests from IoT devices to the optimal edge computing device; The feature extraction and fusion module is used for each edge computing device to perform feature extraction on the received multidimensional data and fuse the extracted multidimensional data features using the modal attention mechanism; The anomaly detection module detects abnormal behaviors in the multidimensional data features in real time on the edge computing device through the anomaly detection model in the edge computing device based on the fused multidimensional data features; The incremental update module combines the recent data cached by the edge computing device and performs local incremental training and parameter update on the anomaly detection model in the edge computing device through distillation learning; The compression and encryption module compresses the collected data and builds an end-to-end compression and encryption mechanism to perform data compression and encryption processing on the communication between the IoT device and the edge computing device.

Images