A data processing method, device and medium based on edge computing

Abstract

This invention discloses a data processing method, device, and medium based on edge computing, relating to the field of data processing technology. The method includes comparing the value score of the data to be processed with a preset decision threshold to obtain high-value and low-value data; constructing a computational task graph based on the high-value data; constructing a decision model based on device status information; solving the computational task graph using the decision model to obtain an unloading scheme; distributing subtasks in the computational task graph to execution locations based on the unloading scheme; performing intelligent conversion calculations to obtain high-value data processing results; and performing simplified calculations locally on low-value data to obtain a low-value data summary. This invention improves the responsiveness and processing quality of edge devices for critical tasks through data processing methods and effectively avoids resource waste of low-value data through differentiated processing strategies.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data processing method, device and medium based on edge computing. Background Technology

[0002] With the deep integration of the Internet of Things (IoT), 5G communication, artificial intelligence, and big data technologies, edge computing, as a new generation of distributed computing architecture, has gained widespread attention and rapid development in both academia and industry in recent years. The core idea of ​​edge computing is to move computing, storage, and network service capabilities from centralized cloud data centers to network edge nodes closer to the data source, such as base stations, gateways, and dedicated edge servers, thereby physically shortening the data transmission path. This effectively aligns with emerging application scenarios such as intelligent transportation, industrial automation, augmented reality, and telemedicine. Compared to previous cloud computing models, edge computing not only reduces end-to-end communication latency and alleviates backbone network bandwidth pressure, but also demonstrates unique advantages in data privacy protection, fault tolerance, and energy efficiency.

[0003] Nevertheless, existing data processing methods still have room for improvement. First, most edge processing methods do not distinguish the importance of data and adopt a uniform processing strategy for high-value and low-value data. This results in a large amount of redundant and inefficient data occupying valuable edge computing power and communication bandwidth, making it difficult to meet the service quality requirements of critical tasks. Second, they do not incorporate device context information into the value assessment system, causing resource scheduling strategies to be out of touch with the actual operating environment, making it difficult to achieve a balance between energy efficiency optimization and latency control. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides a data processing method based on edge computing to solve the problem of difficulty in meeting the service quality requirements of critical tasks and the difficulty in achieving a balance between energy efficiency optimization and latency control.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: In a first aspect, the present invention provides a data processing method based on edge computing, comprising, Collect data to be processed, device status information, and device context information in the edge device, extract features from the data to be processed, and obtain the feature vector of the data to be processed. The feature vector of the data to be processed and the device context information are input into the MLP model for scoring to obtain the value score of the data to be processed. The value score of the data to be processed is compared with a preset decision threshold to obtain high-value data and low-value data. A computational task graph is constructed based on high-value data, and a decision-making model is constructed based on equipment status information. The decision model is used to solve the computational task graph to obtain an unloading scheme; Based on the unloading scheme, subtasks in the computation task graph are distributed to execution locations to perform intelligent transformation computation and obtain high-value data processing results; For low-value data, perform simplified calculations locally to obtain a summary of the low-value data.

[0007] As a preferred embodiment of the edge computing-based data processing method of the present invention, the step of collecting data to be processed, device status information, and device context information in the edge device, and extracting features from the data to be processed to obtain a feature vector of the data to be processed specifically includes: The data to be processed includes image data, video stream data, and text data; The device context information includes the remaining battery power of the edge device, available bandwidth, and task urgency. The device status information includes the current computing load of the edge device, the detection bandwidth and transmission latency between the edge device and the cloud server; The data to be processed is input into a lightweight feature extraction model for hierarchical computation to obtain a deep feature representation; The feature representations of each depth are concatenated and normalized to obtain the feature vector of the data to be processed.

[0008] As a preferred embodiment of the edge computing-based data processing method of the present invention, the step of inputting the feature vector of the data to be processed and the device context information into an MLP model for scoring to obtain a value score of the data to be processed specifically involves: The feature vector of the data to be processed and the device context information are concatenated into an input vector; The input vector is fed into the MLP model. The MLP performs nonlinear transformation and information fusion on the input vector through internal multi-layer forward computation to obtain the value score of the data to be processed.

[0009] As a preferred embodiment of the edge computing-based data processing method of the present invention, the step of comparing the value score of the data to be processed with a preset decision threshold to obtain high-value data and low-value data specifically includes: A preset decision threshold is set, and the value score of the data to be processed is compared with the decision threshold. When the value score of the data to be processed is greater than or equal to the decision threshold, the data to be processed is determined to be high-value data. When the value score of the data to be processed is less than the decision threshold, the data to be processed is determined to be low-value data.

[0010] As a preferred embodiment of the edge computing-based data processing method of the present invention, the step of constructing a computational task graph based on high-value data and constructing a decision model based on device status information specifically includes: Analyze the types and application requirements of high-value data, and determine the computational objectives of high-value data; Dynamically select the task decomposition granularity based on the value score of high-value data; Based on the selected task decomposition granularity, the computational objective is broken down into atomic computation subtasks, and the logical relationships between the atomic computation subtasks are clarified. The atomic computation subtasks and their logical relationships are formally expressed using a directed acyclic graph, resulting in a computation task graph. A mathematical optimization problem framework is predefined, and the structural information of the computation task graph and the equipment status information are used as time-varying input parameters and constraints in the mathematical optimization problem framework, respectively, to obtain a decision model.

[0011] As a preferred embodiment of the edge computing-based data processing method of the present invention, the step of using a decision model to solve the computation task graph refers to the decision model executing a dynamic programming algorithm within the framework of a mathematical optimization problem, assigning an optimal execution position to each atomic computation subtask in the computation task graph, determining the required network path, and obtaining an unloading scheme.

[0012] As a preferred embodiment of the edge computing-based data processing method of the present invention, the following steps are implemented: Based on the unloading scheme, subtasks in the computation task graph are distributed to execution locations to perform intelligent transformation computation and obtain high-value data processing results. Specifically: According to the unloading scheme, the atomic computing subtasks in the computing task graph are transmitted to the corresponding optimal execution position through the communication link. The optimal execution position performs intelligent transformation calculations on the atomic computing subtasks and obtains intermediate results. Based on the topological order of the computation task graph, intermediate results are transmitted to the optimal execution position of the subsequent atomic computation subtasks. The intelligent transformation computation and the transmission of intermediate results are executed iteratively until all atomic computation subtasks in the computation task graph are completed, and high-value data processing results are obtained.

[0013] As a preferred embodiment of the edge computing-based data processing method of the present invention, wherein: for low-value data, simplified calculations are performed locally to obtain a low-value data summary, specifically: When the low-value data is image data, perform downsampling on the image data to obtain an image data summary; When the low-value data is video stream data, perform keyframe extraction on the video stream data to obtain a video stream data summary; When the low-value data is text data, a fixed threshold filtering operation is performed on the text data to obtain a text data summary.

[0014] In a second aspect, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein the computer program, when executed by the processor, implements any step of the edge computing-based data processing method as described in the first aspect of the present invention.

[0015] Thirdly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the edge computing-based data processing method as described in the first aspect of the present invention.

[0016] The beneficial effects of this invention are as follows: By integrating data content features and device context information through the MLP dynamic value scoring mechanism, the importance of data is accurately quantified, thereby effectively distinguishing between high-value and low-value data. On this basis, for high-value data, by constructing a computational task graph and a decision model, joint modeling and global optimization of computational tasks and network resources are achieved. By solving the decision model through a dynamic programming algorithm, a joint offloading scheme including computational location and communication path is generated. This not only improves the responsiveness and processing quality of edge devices for critical tasks, but also effectively avoids resource waste of low-value data through differentiated processing strategies. While ensuring service quality, it significantly optimizes the energy efficiency and bandwidth utilization of edge devices. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a flowchart of a data processing method based on edge computing.

[0019] Figure 2 A flowchart for obtaining the computation task graph.

[0020] Figure 3 A flowchart for obtaining the uninstallation plan.

[0021] Figure 4 A flowchart for obtaining a value score for the data to be processed. Detailed Implementation

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0023] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0024] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0025] Reference Figures 1-4 This is one embodiment of the present invention, which provides a data processing method based on edge computing, including the following steps: S1. Collect the data to be processed, device status information, and device context information in the edge device, extract features from the data to be processed, and obtain the feature vector of the data to be processed.

[0026] S1.1 It should be noted that the edge device acquires image data from the image sensor interface, video stream data from the video streaming port, and text data from the application log. The acquired image data, video stream data, and text data are aggregated into data to be processed. Simultaneously, the edge device reads the battery management unit's power register to obtain the remaining power of the edge device, obtains the available bandwidth by probing the maximum available throughput of the network link, and parses the task urgency from the task configuration file. The remaining power of the edge device, available bandwidth, and task urgency are aggregated into device context information. In addition, the edge device monitors the utilization rate of the central processing unit to obtain the current computing load of the edge device, and measures the response latency and packet loss rate by sending probe data packets to the cloud server to obtain the network status information between the edge device and the cloud server. The current computing load of the edge device and the network status information between the edge device and the cloud server are aggregated into device status information.

[0027] S1.2. Collect historical unprocessed data of various types from the historical storage of edge devices. This historical unprocessed data is generated by the edge devices during historical operation and covers image data, video stream data segments, and text data samples, covering different data types, task loads, and network state scenarios. Use this historical unprocessed data as a training set, with a sample size of at least several thousand data points, to ensure that the lightweight feature extraction model can learn statistically significant data feature representations. The lightweight feature extraction model is a feedforward deep feature extraction network, including two or more sequentially connected convolutional layers. Each convolutional layer is followed by a corresponding activation function layer and a pooling layer, used to extract multi-level feature representations of the unprocessed data layer by layer. The kernel size, number of channels, and number of layers of each convolutional layer can be set according to the computing power and data type of the edge device. For example, the kernel size of a convolutional layer is 3×3, the number of channels is 16, and the number of convolutional layers can be 2.

[0028] The training set is input into the lightweight feature extraction model in batches for training. During training, the lightweight feature extraction model performs forward propagation calculation on each batch of training set to generate predicted features. The error gradient between the predicted features and the training set labels is calculated using a loss function. The error gradient is then propagated back from the output layer of the lightweight feature extraction model to the input layer layer by layer using the backpropagation algorithm. At the same time, the weights and bias parameters of each layer of the lightweight feature extraction model are updated according to the gradient. The process of data input, forward propagation, loss calculation, backpropagation and parameter update is repeated multiple times until the loss function value of the lightweight feature extraction model on the training set converges to below the boundary threshold, resulting in the trained lightweight feature extraction model.

[0029] The threshold is set based on the improvement trend of the performance index of the lightweight feature extraction model, such as "0.001 to 0.01", and is determined based on the change of the loss function value in multiple consecutive training rounds.

[0030] Image data, video stream data, and text data are read from the data to be processed and input into the trained lightweight feature extraction model. The image data, video stream data, and text data are processed through the first convolutional layer of the lightweight feature extraction model for local feature extraction. That is, each convolutional kernel of the convolutional layer performs a sliding window-style multiplication and summation operation on the spatial dimension of the input data and sums it with the convolutional layer bias term to generate a set of two-dimensional feature maps. The two-dimensional feature maps are then subjected to nonlinear transformation through the first activation function layer and then passed to the first pooling layer for downsampling to obtain the first layer feature map. The first layer feature map is used as input to the second layer of the lightweight feature extraction model and undergoes the same convolution, activation, and pooling operations to extract a more abstract and semantically stronger second layer feature map. This process propagates forward step by step in each layer of the lightweight feature extraction model, and the output feature map of each layer has a higher level of abstraction than the previous layer. When the feature map passes through the last layer of the lightweight feature extraction model, the output high-order, compact feature tensor is the deep feature representation.

[0031] The deep feature representations of the data to be processed are flattened, that is, each deep feature representation is converted from a multidimensional tensor shape into a one-dimensional vector. All flattened one-dimensional vectors are then concatenated end-to-end along the vector dimension to form a longer merged vector. The merged vector is then normalized, and its mean and standard deviation are calculated. Each element in the merged vector is subtracted from the mean and then divided by the standard deviation to convert the merged vector into a standard distribution with a mean of 0 and a standard deviation of 1. The normalized merged vector is then denoted as the feature vector of the data to be processed.

[0032] The dimension of the feature vector of the data to be processed is determined by the output layer size of the lightweight feature extraction model and remains unchanged after training; for example, 256 dimensions, to achieve a balance between feature representation capability and computational complexity of edge devices.

[0033] S2. Input the feature vector of the data to be processed and the device context information into the MLP model for scoring to obtain the value score of the data to be processed.

[0034] S2.1. Concatenate the feature vector of the data to be processed and the device context information into an input vector.

[0035] It should be noted that the device context information is vectorized and encoded. Specifically, the device context information includes the remaining power of the edge device, the available bandwidth, and the task urgency. The percentage value of the remaining power of the edge device, the megabits per second value of the available bandwidth, and the integer level of the task urgency are encoded and arranged in sequence to form a device context information vector. The feature vector of the data to be processed and the device context information vector are concatenated at the end of the feature vector of the data to be processed to form the input vector.

[0036] The integer level encoding of task urgency is obtained through the following process: The raw representation information of task urgency is read from the task configuration file, including the absolute timestamp of the task deadline; the absolute timestamp of the task deadline is compared with the current time of the edge device to calculate the remaining time; the remaining time is mapped to an integer level, for example, less than 1 minute remaining is mapped to level 5, 1-5 minutes to level 4, and so on, to obtain the integer level encoding of task urgency; the mapping is based on the negative correlation between the remaining task time and the urgency of processing. That is, the shorter the remaining time, the higher the risk of the task failing due to timeout, the stronger the urgency of processing, and therefore the higher the urgency level should be assigned.

[0037] S2.2 Feed the input vector into the MLP model. The MLP performs nonlinear transformation and information fusion on the input vector through internal multi-layer forward computation to obtain the value score of the data to be processed.

[0038] It should be noted that historical input vectors and corresponding real value scores are collected as training sets; historical input vectors consist of historical data feature vectors and historical device context information vectors; the MLP model is a feedforward artificial neural network composed of multiple layers of neurons fully connected in a hierarchical manner, and its structure includes one input layer, three hidden layers and one output layer. Initialize the MLP model by randomly generating initial weight matrices and bias parameters for all layers. Input the training set into the MLP model for training. The MLP model performs forward propagation on the training set to obtain predicted value scores. Calculate the error between the predicted value score and the true value score (loss function value) using a loss function (mean squared error function). Using the backpropagation algorithm, starting from the output layer of the MLP model, calculate the gradient of the loss function with respect to each weight matrix and bias parameter in the MLP model layer by layer. Use the Adam optimizer (e.g., "learning rate set to 0.001, batch size set to 32") to update the values ​​of all weight matrices and bias parameters in the MLP model based on the calculated gradients, thereby reducing the loss function value. Repeat the forward propagation, loss calculation, backpropagation, and parameter update process until the maximum number of iterations (e.g., 1000 times) is reached to obtain the trained MLP model.

[0039] The maximum number of iterations is set to prevent the training process from looping indefinitely, and its value is determined based on the computational resource budget and training time constraints.

[0040] The input vector is passed through the input layer to the first hidden layer of the MLP model. The first hidden layer performs a linear transformation on the input vector, which involves multiplying the input vector with the weight matrix of the hidden layer and summing the result with the bias parameters to generate the first hidden layer vector. The first hidden layer vector is then passed to the first activation function layer of the MLP model. The first activation function layer applies a linear rectified unit activation function to each element of the first hidden layer vector to generate the first nonlinearly transformed hidden layer vector. The first nonlinearly transformed hidden layer vector is then input to the second hidden layer of the MLP model. The second hidden layer performs the same linear transformation operation as the first hidden layer and outputs the second hidden layer vector. The second hidden layer vector is then processed sequentially through the second activation function layer of the MLP model. This alternating computation process of "hidden layer-activation function layer" occurs sequentially within the MLP model until it reaches the output layer. The output layer of the MLP model performs a final linear transformation on the last nonlinearly transformed hidden layer vector and outputs a real-valued scalar, which is the value score of the data to be processed.

[0041] S3. Compare the value score of the data to be processed with the preset decision threshold to obtain high-value data and low-value data.

[0042] It should be noted that the value score of the data to be processed is compared with the decision threshold. When the value score of the data to be processed is greater than or equal to the decision threshold, the data to be processed is determined to be high-value data; when the value score of the data to be processed is less than the decision threshold, the data to be processed is determined to be low-value data.

[0043] The decision threshold is set based on the critical point of the difference in benefits between high-value and low-value data in the historical data to be processed. It is determined by analyzing the distribution of historical value scores and weighing the necessary cost of processing high-value data against the information loss caused by simplifying low-value data. The value range is between 0 and 1.

[0044] S4. Construct a computational task graph based on high-value data, and construct a decision-making model based on equipment status information.

[0045] S4.1 Analyze the types and application requirements of high-value data, and determine the computational objectives of high-value data.

[0046] It should be noted that the metadata tags carried by high-value data are checked to identify the specific type of high-value data (including image data, video stream data, and text data); based on the identified high-value data types, a pre-defined application requirement configuration table is queried to obtain the corresponding application requirements (including the information categories to be extracted from the high-value data, the processing accuracy indicators to be met, and the acceptable maximum processing latency); a mapping relationship table between information categories and computational task definitions is established and maintained. The mapping relationship table records the standard computational task description, recommendation algorithm, model name, and expected input and output data format corresponding to each information category; the information categories in the application requirements are located in the mapping relationship table, which is queried using the information category as the primary key (the primary key can be generated by performing a logical numbering on the information category name to uniquely identify the corresponding information category), and constraints are applied in conjunction with the processing accuracy indicators and the maximum processing latency to obtain the corresponding computational objective; for example, "within a latency of no more than Y milliseconds, execute the Z algorithm on the high-value data to complete the extraction of specific information with an accuracy of at least P%."

[0047] The process of constructing the application requirement configuration table is as follows: Based on historical business scenarios, collect historical work orders and log data, and summarize the correspondence between typical application scenarios and requirements; structurally associate the characteristics of each application scenario (such as data type, required information category and business value goal) with specific processing requirements (including the information categories that must be extracted, the minimum acceptable processing accuracy index, and the maximum allowable processing latency), and persistently store them in the form of a data table to form the application requirement configuration table.

[0048] The mapping relationship table analyzes historical task execution records to summarize the most effective implementation method for each information category. It uses the information category name as the primary key and the corresponding technical implementation details (including the description of the computation task, the recommendation algorithm / model identifier, the preprocessing specification, and the expected output data format) as the association value to organize the data in a structured manner and store it in a data table.

[0049] S4.2 Dynamically select the task decomposition granularity based on the value score of high-value data.

[0050] It should be noted that the value score is divided into three continuous and non-overlapping numerical intervals: the first interval is defined as [0.8, 1.0], the second interval as [0.5, 0.8], and the third interval as [0, 0.5]. The first interval is bound to fine-grained task decomposition, the second interval to medium-grained task decomposition, and the third interval to coarse-grained task decomposition (different task decomposition granularities correspond to different task breakdown templates). During runtime, the value score of high-value data is compared with the boundaries of these three numerical intervals. When the value score falls within a numerical interval, the corresponding granularity of task decomposition level is adopted.

[0051] The basis for dividing the numerical intervals is as follows: Analyze historical high-value data processing tasks, observe the comprehensive benefits of processing data in different value scoring intervals with different task decomposition granularities, and compare the comprehensive benefits in three dimensions: result accuracy, task completion delay, and computational resource consumption. Select the scoring boundary point that can achieve the optimal balance of comprehensive benefits in each dimension as the dividing threshold, thereby determining [0.8, 1.0] as the high-efficiency refined processing interval, [0.5, 0.8) as the medium-efficiency balanced processing interval, and [0, 0.5) as the efficiency-sensitive fast processing interval.

[0052] S4.3 Based on the selected task decomposition granularity, the computational objective is broken down into atomic computation subtasks, and the logical relationships between the atomic computation subtasks are clarified.

[0053] It should be noted that, based on the selected task decomposition granularity, the corresponding task decomposition template is invoked. The task decomposition template takes the computation target as input and, according to the built-in decomposition rules, decomposes the computation target layer by layer into functionally independent and indivisible basic operation units. Each basic operation unit is an atomic computation subtask. During the decomposition process, the input-output dependencies between atomic computation subtasks are analyzed and recorded to form a clear execution order logic. Specifically, each atomic computation subtask is traversed to identify the input data required for the execution of the atomic computation subtask and the output data that will be generated after the execution of the atomic computation subtask. If the output data of the current atomic computation subtask is the input data of the subsequent atomic computation subtask, then there is a direct dependency relationship between the atomic computation subtasks. All atomic computation subtasks are traversed in turn to form a clear execution order logic.

[0054] S4.4. The atomic computation subtasks and logical relationships are formally expressed using a directed acyclic graph to obtain the computation task graph.

[0055] It should be noted that a directed acyclic graph (DAG) data structure is created, consisting of a set of vertices and a set of directed edges. Each vertex of the DAG is uniquely mapped to an atomic computation subtask, and the vertex attributes record the computational objective of the atomic computation subtask. According to the sequential logic between atomic computation subtasks, a directed edge is added between atomic computation subtasks with direct dependencies, and the weight attribute of the directed edge can record the amount of data to be transferred between the atomic computation subtasks. This DAG data structure, including vertices, directed edges, and attributes, is the computation task graph.

[0056] S4.5. It should be noted that an objective function should be constructed, which is a linear combination of the total task execution latency and the total power consumption of the edge devices, and the formula is: ; in, This represents the objective function value within the framework of a mathematical optimization problem. The weighting coefficient represents the total execution latency of the task. The weighting coefficient represents the total energy consumption of edge devices. Indicates the total execution delay of the task. This represents the total power consumption of edge devices.

[0057] The weighting coefficients for total task execution latency and total edge device energy consumption are set to adjust the relative importance of latency and energy consumption in optimization. The specific values ​​are determined based on historical task execution data. The standard deviations of total task execution latency and total edge device energy consumption are calculated separately, and the values ​​are determined according to the proportion of their standard deviations in the overall fluctuation. For example, the weighting coefficient for total task execution latency is 1, and the weighting coefficient for total edge device energy consumption is 0.5.

[0058] The weighting coefficients for total task execution latency and total edge device energy consumption are used to characterize the relative importance of different latency and energy consumption indicators in the overall optimization objective, and their values ​​are not limited to normalized form.

[0059] Define decision variables, which represent the execution location to which each atomic computational subtask in the computational task graph is assigned, and on which network link data is transmitted between atomic computational subtasks; establish a set of constraints, which include that each atomic computational subtask must be assigned to exactly one execution location, the computational and communication resource consumption of each execution location cannot exceed the corresponding available capacity, and the execution order determined by the data dependencies between atomic computational subtasks must be satisfied; the objective function, decision variables, and set of constraints together constitute the framework of the mathematical optimization problem.

[0060] The computation task graph is traversed, and the computational load of each atomic computation subtask recorded at each vertex is read and converted into a floating-point number for computational load parameters. The directed edge set of the computation task graph is traversed, and the data transfer volume between atomic computation subtasks recorded at each directed edge is read and converted into a floating-point number for data volume parameters. The topology of the computation task graph is analyzed, and the dependencies between atomic computation subtasks are converted into a set of partial order relations describing the execution order of atomic computation subtasks. The computational load parameters, data volume parameters, and partial order relation set together constitute the quantized input parameters.

[0061] Collect the current CPU utilization and core frequency of edge devices, nearby edge servers, and remote cloud servers, and calculate the real-time available computing power at each execution location using the following formula: ; in, This indicates the real-time available computing power at the execution location. Indicates the utilization rate of the central processing unit. This indicates the number of physical cores in the central processing unit. This indicates the operating frequency of each core in the central processing unit.

[0062] The device status information is analyzed to extract the current detection bandwidth and measurement latency of the network links between edge devices and each edge server, as well as between each edge server and the cloud. The current detection bandwidth is used as the upper limit of the communication link bandwidth capacity, and the measurement latency is used as the transmission latency parameter. The real-time available computing power, the upper limit of the communication link bandwidth capacity, and the transmission latency parameter constitute numerical constraints.

[0063] Substituting the quantified input parameters and numerical constraints into the pre-defined mathematical optimization problem framework, all variables and constants in the mathematical optimization problem framework are now realized, and the realized mathematical optimization problem is the decision model.

[0064] S5. Use a decision model to solve the computational task graph and obtain an unloading scheme.

[0065] It should be noted that, based on the topology of the computation task graph, the atomic computation subtasks are topologically sorted to obtain a linear sequence of atomic computation subtask executions. Starting from the first atomic computation subtask in the execution sequence, a dynamic programming algorithm is used in the decision model to traverse all possible execution positions of the first atomic computation subtask. Based on the quantified computational load parameters, data volume parameters, real-time available computing power, upper limit of communication link bandwidth capacity, and transmission latency in the mathematical optimization problem framework, the objective function value of the first atomic computation subtask at each possible execution position is calculated, and the corresponding decision position is recorded. For subsequent atomic computation subtasks in the execution sequence, the dynamic programming algorithm iteratively calculates the objective function value of the subsequent atomic computation subtasks at each possible execution position. After completing the calculation of the execution sequence, the dynamic programming algorithm compares the objective function values ​​of the last atomic computation subtask at all possible execution positions and selects the position with the smallest objective function value as the final target position of the computation task graph. Starting from the final target position, based on the recorded decision positions, the algorithm backtracks to the first atomic computation subtask in the execution sequence, thereby reconstructing the complete decision chain to reach the final target position, which is the optimal execution position sequence determined for each atomic computation subtask. Optimal execution locations include edge devices, nearby edge servers, and cloud servers (in the cloud).

[0066] Traverse the set of directed edges in the computation task graph, with each directed edge connecting a predecessor atomic computation subtask and a successor atomic computation subtask. Based on the optimal execution position allocation sequence, find the optimal execution position assigned to the predecessor atomic computation subtask and the optimal execution position assigned to the successor atomic computation subtask. Based on the optimal execution positions corresponding to the predecessor and successor atomic computation subtasks, query the records in the preset network topology routing table with the optimal execution position as the index key. Retrieve all reachable path information associated with the optimal execution position from the network topology routing table to obtain a candidate network path set. For each candidate network path in the candidate network path set, read the probe bandwidth and measurement delay parameters of each network link in the candidate network path. Simultaneously, combine the data transmission volume between atomic computation subtasks to calculate the end-to-end transmission delay cost and bandwidth occupancy cost of the candidate network path. The end-to-end transmission delay cost is obtained by hop-by-hop summing the transmission delay parameters of each network link in the candidate network path and then superimposing the result of the division between the data transmission volume and the probe bandwidth. The ratio of the data transmission volume to the probe bandwidth of each network link is the bandwidth occupancy cost. Under the constraint of meeting the upper limit of communication link bandwidth capacity, the candidate network path with the minimum end-to-end transmission delay cost and bandwidth occupancy cost is selected as the network path required by the predecessor atomic computation subtask and the successor atomic computation subtask. After repeatedly performing search, calculation and selection operations on each directed edge in the computation task graph, the predecessor atomic computation subtask, the successor atomic computation subtask and the corresponding network path of each directed edge are written into a table to form a path mapping table. The optimal execution position sequence and the path mapping table together constitute the offloading scheme.

[0067] The process of constructing the network topology routing table is as follows: A unique execution location identifier (logical identifier) ​​is assigned to each execution location (edge ​​device, adjacent edge server, and cloud server); starting from each execution location, network link probing operations are performed on other directly or indirectly connected execution locations to obtain the network link's probe bandwidth, transmission latency, and connectivity status information; based on the network link probing results, a reachable path (at least one) is generated between any two execution location identifiers. The reachable path consists of one or more network links, and the corresponding path identifier, link sequence, cumulative transmission latency, and available bandwidth information are recorded; using the execution location identifier as the index key, the reachable path information is written into the network topology routing table.

[0068] S6. Based on the unloading scheme, the subtasks in the computation task graph are distributed to the execution location to perform intelligent transformation computation and obtain high-value data processing results.

[0069] It should be noted that the optimal execution location and corresponding network path determined for each atomic computing subtask in the unloading scheme are read. The input data of the atomic computing subtask is transmitted according to the network path recorded in the mapping table, so that the atomic computing subtask reaches the optimal execution location specified by the unloading scheme. After receiving the atomic computing subtask and its corresponding input data, the optimal execution location reads the computing target recorded in the atomic computing subtask and performs intelligent transformation calculations on the input data according to the processing flow sequence corresponding to the computing target in the task decomposition template, namely, preprocessing operation, core computing operation, and result processing operation. Among them, the preprocessing operation is used to convert the input data into the standard data format required by the task decomposition template. The core computing operation performs calculations on the standard data format according to the recommended algorithm and parameter configuration defined in the task decomposition template. The result processing operation formats and verifies the calculation results. After the above operations, the processing result (i.e., intermediate result) of the atomic computing subtask is generated. The preprocessing operation, core computing operation, and result processing operation are all performed by the optimal execution location and cannot be separated to other execution locations.

[0070] For completed atomic computation subtasks, find their corresponding successor atomic computation subtasks with direct dependencies; use the intermediate results of the atomic computation subtasks as output data, and upload the output data to the corresponding successor atomic computation subtasks according to the path mapping table, perform intelligent transformation computation, and obtain new intermediate results. Iterate through the intelligent transformation computation and the transmission of intermediate results until all atomic computation subtasks in the computation task graph have been executed, obtaining high-value data processing results.

[0071] S7. For low-value data, perform simplified calculations locally to obtain a summary of the low-value data.

[0072] S7.1 When the low-value data is image data, perform downsampling on the image data to obtain an image data summary.

[0073] It should be noted that the original pixel matrix of the image data is read. The original pixel matrix has an initial height, initial width, and number of color channels, and a preset scaling factor (e.g., 0.25). The downsampling algorithm receives the initial height and initial width values ​​of the original pixel matrix, as well as the preset scaling factor, as input parameters. The initial height and initial width values ​​are multiplied by the scaling factor respectively to obtain the calculated height and calculated width values. The calculated height and calculated width values ​​are rounded to obtain the new height and new width values. An empty target pixel matrix with dimensions [new height value, new width value, number of color channels (consistent with the number of color channels in the original pixel matrix)] is created and initialized. Each target pixel position in the empty target pixel matrix is ​​traversed. The target pixel position is defined by the target row index and the target column index. Based on the target row index and the target column index, combined with the initial height and initial width values ​​of the original pixel matrix, the original floating-point coordinates of the target pixel position in the original pixel matrix are calculated through coordinate mapping relationships. The formula is: ; ; in, This represents the x-coordinate value of the original floating-point coordinates. Indicates the index of the target column. This indicates the new width value. This represents the initial width value. This represents the ordinate value of the original floating-point coordinates. Indicates the target row index. This indicates the new height value. This represents the initial height value.

[0074] Centered on the original floating-point coordinates, select four neighboring known pixels in the original pixel matrix (forming a 2x2 neighborhood). Calculate the difference in the horizontal coordinates between the original floating-point coordinates and the known pixels to obtain the horizontal distance; calculate the difference in the vertical coordinates between the original floating-point coordinates and the known pixels to obtain the vertical distance; based on the calculated horizontal and vertical distances, substitute them into a preset weight kernel function to calculate the weight value of the known pixels, using the following formula: ; in, This represents the weight coefficient of a known pixel. Indicates horizontal distance. Indicates vertical distance.

[0075] For each known pixel, the color value in each color channel is multiplied by its corresponding weight coefficient to obtain a set of weighted color values. All weighted color values ​​in the same color channel are summed to obtain a weighted average for that color channel. The weighted averages of all color channels together constitute the target pixel color value. The calculated target pixel color value is then filled into the corresponding position in the target pixel matrix. After traversing and filling all positions in the target pixel matrix, a downsampled pixel matrix is ​​obtained. The downsampled pixel matrix, the original pixel matrix, the downsampling timestamp, and the scaling factor are then encapsulated together to form an image data digest.

[0076] The scaling factor is set based on the basic information preservation requirements of low-value data summaries. The value is determined by analyzing the relationship between the recognizability of the downsampled image and the data compression ratio under different scaling factors.

[0077] S7.2 When the low-value data is video stream data, perform keyframe extraction on the video stream data to obtain a video stream data summary.

[0078] It should be noted that the continuous video frame sequence in the video stream data is read in chronological order, and the content difference between adjacent video frames is calculated. The content difference is obtained by comparing the histogram distance of the pixel values ​​of the two frames. The calculated content difference is compared with a preset scene change threshold. If the content difference exceeds the scene change threshold, the current video frame is marked as a keyframe. The timestamps of all marked keyframes are recorded, and the keyframes are extracted from the video stream and arranged in their original chronological order to form a keyframe set. The extracted keyframe set and the corresponding timestamps are encapsulated together to form a video stream data summary.

[0079] The scene change threshold is set to avoid generating too many invalid keyframes due to slight disturbances in the image. It is calculated by statistically analyzing the content differences between adjacent video frames and calculating the mean and standard deviation of the content differences. The linear combination of the mean and standard deviation is used as the scene change threshold, with a value ranging from 0.5 to 0.7.

[0080] S7.3 When the low-value data is text data, perform a fixed threshold filtering operation on the text data to obtain a text data summary.

[0081] The application describes a process of reading each text entry from low-value text data, performing word segmentation on each entry to obtain a sequence of terms; counting the occurrence counts of all different terms in each text entry, and calculating the probability of each different term in the text entry (the quotient of the number of occurrences of a term to the total number of terms in the text entry); and calculating the information entropy of the text entry based on the probability of occurrence of all different terms, using the following formula: ; in, The information entropy of a text entry. Indicates the number of different terms in a text entry. Indicates the first The probability of each different term appearing in a text entry.

[0082] Information entropy is used as an importance score for text entries and compared with a preset fixed threshold. If the importance score is greater than or equal to the fixed threshold, the corresponding text entry is retained; if the importance score is less than the fixed threshold, the corresponding text entry is filtered out. All retained text entries are sorted in their original chronological order to form a concise text set. The text set, filtering time, number of retained entries, and fixed threshold used are encapsulated to form a text data summary.

[0083] The fixed threshold is set to distinguish between key text information that needs to be retained and secondary text information that can be filtered. By statistically analyzing the information entropy of historical text data, 90% to 95% of the information entropy is taken as the fixed threshold. The specific value can be adjusted adaptively according to the different requirements of the actual business for the completeness and conciseness of the summary.

[0084] This embodiment also provides a computer device applicable to edge computing-based data processing methods, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the edge computing-based data processing method proposed in the above embodiment.

[0085] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0086] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the data processing method based on edge computing as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0087] In summary, this invention achieves precise quantification of data importance through an MLP dynamic value scoring mechanism that integrates data content features and device context information, thereby effectively distinguishing between high-value and low-value data. Based on this, for high-value data, it constructs a computational task graph and a decision model to achieve joint modeling and global optimization of computational tasks and network resources. By solving the decision model using a dynamic programming algorithm, a joint offloading scheme including computational location and communication path is generated. This not only improves the responsiveness and processing quality of edge devices for critical tasks but also effectively avoids resource waste of low-value data through differentiated processing strategies, significantly optimizing the energy efficiency and bandwidth utilization of edge devices while ensuring service quality.

[0088] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A data processing method based on edge computing, characterized in that: include, Collect data to be processed, device status information, and device context information in the edge device, extract features from the data to be processed, and obtain the feature vector of the data to be processed. The feature vector of the data to be processed and the device context information are input into the MLP model for scoring to obtain the value score of the data to be processed. The value score of the data to be processed is compared with a preset decision threshold to obtain high-value data and low-value data. A computational task graph is constructed based on high-value data, and a decision-making model is constructed based on equipment status information. The decision model is used to solve the computational task graph to obtain an unloading scheme; Based on the unloading scheme, subtasks in the computation task graph are distributed to execution locations to perform intelligent transformation computation and obtain high-value data processing results; For low-value data, perform simplified calculations locally to obtain a summary of the low-value data.

2. The data processing method based on edge computing as described in claim 1, characterized in that: The process of collecting data to be processed, device status information, and device context information in the edge device, and extracting features from the data to be processed to obtain a feature vector of the data to be processed, specifically involves: The data to be processed includes image data, video stream data, and text data; The device context information includes the remaining battery power of the edge device, available bandwidth, and task urgency. The device status information includes the current computing load of the edge device, the detection bandwidth and transmission latency between the edge device and the cloud server; The data to be processed is input into a lightweight feature extraction model for hierarchical computation to obtain a deep feature representation; The feature representations of each depth are concatenated and normalized to obtain the feature vector of the data to be processed.

3. The data processing method based on edge computing as described in claim 2, characterized in that: The step of inputting the feature vector of the data to be processed and the device context information into the MLP model for scoring to obtain the value score of the data to be processed is as follows: The feature vector of the data to be processed and the device context information are concatenated into an input vector; The input vector is fed into the MLP model. The MLP performs nonlinear transformation and information fusion on the input vector through internal multi-layer forward computation to obtain the value score of the data to be processed.

4. The data processing method based on edge computing as described in claim 3, characterized in that: The step of comparing the value score of the data to be processed with a preset decision threshold to obtain high-value data and low-value data specifically involves: A preset decision threshold is set, and the value score of the data to be processed is compared with the decision threshold. When the value score of the data to be processed is greater than or equal to the decision threshold, the data to be processed is determined to be high-value data. When the value score of the data to be processed is less than the decision threshold, the data to be processed is determined to be low-value data.

5. The data processing method based on edge computing as described in claim 4, characterized in that: The process of constructing a computational task graph based on high-value data and building a decision model based on equipment status information specifically involves: Analyze the types and application requirements of high-value data, and determine the computational objectives of high-value data; Dynamically select the task decomposition granularity based on the value score of high-value data; Based on the selected task decomposition granularity, the computational objective is broken down into atomic computation subtasks, and the logical relationships between the atomic computation subtasks are defined. The atomic computation subtasks and their logical relationships are formally expressed using a directed acyclic graph, resulting in a computation task graph. A mathematical optimization problem framework is predefined, and the structural information of the computation task graph and the equipment status information are used as time-varying input parameters and constraints in the mathematical optimization problem framework, respectively, to obtain a decision model.

6. The data processing method based on edge computing as described in claim 5, characterized in that: The use of a decision model to solve the computation task graph refers to the decision model executing a dynamic programming algorithm within the framework of a mathematical optimization problem, assigning an optimal execution position to each atomic computation subtask in the computation task graph, determining the required network path, and obtaining an unloading scheme.

7. The data processing method based on edge computing as described in claim 6, characterized in that: The unloading-based scheme distributes subtasks in the computation task graph to execution locations, performs intelligent transformation computation, and obtains high-value data processing results, specifically as follows: According to the unloading scheme, the atomic computing subtasks in the computing task graph are transmitted to the corresponding optimal execution position through the communication link. The optimal execution position performs intelligent transformation calculations on the atomic computing subtasks and obtains intermediate results. Based on the topological order of the computation task graph, intermediate results are transmitted to the optimal execution position of the subsequent atomic computation subtasks. The intelligent transformation computation and the transmission of intermediate results are executed iteratively until all atomic computation subtasks in the computation task graph are completed, and high-value data processing results are obtained.

8. The data processing method based on edge computing as described in claim 7, characterized in that: For low-value data, a simplified calculation is performed locally to obtain a low-value data summary, specifically as follows: When the low-value data is image data, perform downsampling on the image data to obtain an image data summary; When the low-value data is video stream data, perform keyframe extraction on the video stream data to obtain a video stream data summary; When the low-value data is text data, a fixed threshold filtering operation is performed on the text data to obtain a text data summary.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the edge computing-based data processing method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the edge computing-based data processing method according to any one of claims 1 to 8.

Images