An adaptive selection method and device for data preprocessing algorithms of edge heterogeneous devices

Abstract

The application provides an edge heterogeneous device data preprocessing algorithm adaptive selection method and device. The method provided by the application receives multiple concurrent data preprocessing requests sent by an edge heterogeneous device, identifies the dependency relationship of each request, constructs a request relationship graph and generates a request group. The data processing magnitude and urgency are fused to determine the priority of the group and form a first priority queue. The requests in the group are sorted according to the data processing causal relationship to obtain a second priority queue. The target request is traversed, the target algorithm is determined based on the agent and device running state data, the time slice is determined according to the algorithm complexity and data magnitude, the original data in the target data preprocessing request is processed using the target data preprocessing algorithm in the order of the second request queue. The edge heterogeneous device data preprocessing algorithm adaptive selection method and device provided by the application can quickly process multiple concurrent requests, effectively improve resource utilization and processing effect.

Description

Technical Field

[0001] This application relates to the field of data preprocessing technology, and in particular to an adaptive selection method and apparatus for data preprocessing algorithms for edge heterogeneous devices. Background Technology

[0002] In the digital age, edge computing has been widely used in many fields due to its advantages such as proximity to data sources and low latency. Heterogeneous edge devices, such as various sensors and smart terminals, can collect massive amounts of data in real time. However, this data often varies in format and quality, requiring effective data preprocessing to meet the needs of subsequent analysis and applications.

[0003] Currently, common preprocessing methods for edge heterogeneous devices mainly involve setting a fixed data preprocessing algorithm for each device or task, or uniformly processing all data preprocessing requests using a single fixed algorithm. Because using only a fixed data preprocessing algorithm fails to meet the task requirements of all requests when the data content and processing objectives of different preprocessing requests vary, it leads to poor data preprocessing results or excessively long processing times. Furthermore, when faced with multiple data preprocessing requests, a single data preprocessing algorithm struggles to balance reasonable resource allocation and high processing efficiency, thus impacting the overall system efficiency. Summary of the Invention

[0004] In view of this, this application provides an adaptive selection method and apparatus for data preprocessing algorithms for edge heterogeneous devices, which can quickly process multiple concurrent data preprocessing requests, improve resource utilization and processing effect.

[0005] Specifically, this application is implemented through the following technical solution:

[0006] The first aspect of this application provides an adaptive selection method for data preprocessing algorithms of edge heterogeneous devices, the method comprising:

[0007] Receive multiple concurrent data preprocessing requests from the edge heterogeneous device, where each data preprocessing request processes different data;

[0008] Identify the dependencies between each data preprocessing request and construct a request relationship graph, which is used to characterize the task correlation between each data preprocessing request;

[0009] Multiple concurrent data preprocessing requests are bound together according to the request relationship graph to generate multiple request groups, and the number of data preprocessing requests in each request group is not exactly the same.

[0010] The data processing volume and urgency are determined based on each data preprocessing request, and the data processing volume and urgency are combined to obtain the request processing priority for each request group;

[0011] Multiple request groups are sorted according to the request processing priority to obtain the first priority request queue;

[0012] The data preprocessing requests within the request group are sorted according to the causal relationship of data processing within the request group, and combined with the first priority request queue, a second priority request queue is obtained.

[0013] The data preprocessing requests are traversed in the order of the second priority request queue. For the target data preprocessing request currently being traversed, the target data preprocessing algorithm corresponding to the data preprocessing request is determined based on the operating status data of the intelligent agent and the edge heterogeneous device.

[0014] The time slice of the target data preprocessing request is determined based on the complexity of the target data preprocessing algorithm and the processing data volume of the target data preprocessing request, wherein the resources and operating status of the edge heterogeneous device within the time slice are matched with the complexity and the data volume.

[0015] The edge heterogeneous device processes the raw data in the target data preprocessing request using the target data preprocessing algorithm within the time slice corresponding to the target data preprocessing request, according to the order in the second priority request queue.

[0016] A second aspect of this application provides an adaptive selection device for data preprocessing algorithms of edge heterogeneous devices, the device comprising a processing module, a sorting module, and an execution module; wherein...

[0017] The processing module is used to receive multiple concurrent data preprocessing requests from the edge heterogeneous device, and the data processed by each data preprocessing request is not exactly the same;

[0018] The processing module is also used to identify the dependencies between each of the data preprocessing requests and construct a request relationship graph, which is used to characterize the task correlation between each data preprocessing request.

[0019] The processing module is also used to bind multiple concurrent data preprocessing requests according to the request relationship graph, and generate multiple request groups, wherein the number of data preprocessing requests in each request group is not exactly the same;

[0020] The sorting module is used to determine the data processing volume and urgency level based on each data preprocessing request, and to combine the data processing volume and urgency level to obtain the request processing priority for each request group.

[0021] The sorting module is also used to sort multiple request groups according to the request processing priority to obtain a first priority request queue;

[0022] The sorting module is further configured to sort the data preprocessing requests within the request group according to the data processing causal relationship within the request group, and combine the first priority request queue to obtain a second priority request queue.

[0023] The execution module is used to traverse the data preprocessing requests in the order of the second priority request queue, and for the currently traversed target data preprocessing request, determine the target data preprocessing algorithm corresponding to the data preprocessing request based on the operating status data of the intelligent agent and the edge heterogeneous device.

[0024] The execution module is further configured to determine the time slice of the target data preprocessing request based on the complexity of the target data preprocessing algorithm and the processing data volume of the target data preprocessing request, wherein the resources and operating status of the edge heterogeneous device within the time slice are matched with the complexity and the data volume.

[0025] The execution module is further configured to allow the edge heterogeneous device to process the raw data in the target data preprocessing request using the target data preprocessing algorithm within the time slice corresponding to the target data preprocessing request, according to the order in the second priority request queue.

[0026] The adaptive selection method and apparatus for data preprocessing algorithms for edge heterogeneous devices provided in this application accurately, adaptively, and customarily match preprocessing algorithms based on the real-time status of the edge heterogeneous devices and the needs of data preprocessing requests, achieving efficient, accurate, and resource-utilization-efficient data preprocessing. To achieve precise matching between the data preprocessing request response method and the real-time status, firstly, on the data preprocessing request side, related requests are bound for processing, improving the scientific nature of data preprocessing and providing a categorized set of requests for subsequent processing by edge heterogeneous devices; secondly, request groups are sorted between groups based on data processing volume and urgency, and also sorted within groups according to logical relationships, ensuring that the processing order of requests follows logical priority while prioritizing urgent requests, thus improving the scientific nature of multiple request responses; thirdly, after sorting multiple requests, the most suitable preprocessing algorithm is determined according to the status of the edge heterogeneous devices. After matching the algorithm with the request, the time slice corresponding to the processing request is determined according to the resource situation of the edge heterogeneous devices, i.e., time allocation corresponds to resources, ultimately achieving the most efficient processing using the most effective algorithm. Attached Figure Description

[0027] Figure 1 A flowchart of an embodiment of the adaptive selection method for data preprocessing algorithms of edge heterogeneous devices provided in this application;

[0028] Figure 2 A schematic diagram of the structure of an embodiment of the adaptive selection device for data preprocessing algorithms of edge heterogeneous devices provided in this application. Detailed Implementation

[0029] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0030] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used herein are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0031] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0032] The following specific embodiments are given to illustrate the technical solution of this application in detail.

[0033] Figure 1 The flowchart illustrates an embodiment of the adaptive selection method for data preprocessing algorithms of edge heterogeneous devices provided in this application. Please refer to... Figure 1 The method provided in this embodiment may include:

[0034] S101. Receive multiple concurrent data preprocessing requests from the edge heterogeneous device, where each data preprocessing request processes different data.

[0035] Specifically, edge heterogeneous devices refer to various devices located at the network edge, such as sensors, smart home devices, and in-vehicle terminals. It should be noted that multiple concurrent data preprocessing requests can be issued by a single edge heterogeneous device or by multiple edge heterogeneous devices; this embodiment does not limit this. The edge heterogeneous device sends a request to the system to preprocess the data it has collected. This data preprocessing request includes information such as the data source, data format, data type, processing target, deadline, and data processing task. Regarding the presentation of the data source, if the data preprocessing request contains a large amount of data, it only carries a link to the data, allowing the corresponding data to be retrieved through the link. If the data preprocessing request contains a small amount of data, it directly carries the corresponding data itself. This considers the actual needs of data processing, enabling more efficient data transmission and storage, and also facilitating the system's accurate extraction of data features. Furthermore, different devices process different data content due to differences in function and collection purpose; therefore, the data processed by each data preprocessing request may be the same or different. For example, in one possible implementation, there are two concurrent data preprocessing requests, namely request 1 and request 2. Request 1 is for detecting and screening raw material quality data, and request 2 is for optimizing production processes based on the screened data.

[0036] It should be noted that the process of receiving data preprocessing requests also includes:

[0037] (1) Establish an algorithm database;

[0038] (2) Classify the algorithms in the algorithm database according to the algorithm processing target, and encode each type of algorithm;

[0039] Specifically, an algorithm database refers to a collection that stores various data preprocessing algorithms. In the field of data preprocessing, there are many different types and functions of algorithms, such as data dimensionality reduction algorithms, data standardization algorithms, data clustering algorithms, and data discretization algorithms. Establishing an algorithm database can centrally store and manage these algorithms, making it convenient to call them according to different data preprocessing needs.

[0040] Furthermore, based on the processing objectives, multiple algorithms are categorized. For example, algorithms used for data cleaning are grouped into one category, and algorithms used for data transformation into another. This allows for the rapid identification of the set of algorithms that meet specific processing needs. After categorizing the algorithms, each category is assigned a unique code, which facilitates easier matching of algorithms with data preprocessing requests in subsequent processes, improving efficiency.

[0041] (3) For algorithms under different categories, determine the distinguishing features corresponding to all algorithms under the same category, wherein the values ​​of the distinguishing features corresponding to all algorithms under the same category are different;

[0042] Specifically, although algorithms of the same type share the same processing objective, the processes used to achieve that objective may differ. Feature extraction can be performed on algorithms within the same category, selecting distinguishing features from the corresponding features of each algorithm; that is, each algorithm has unique distinguishing features. It should be noted that a distinguishing feature can be a single feature or a combination of multiple features. The number of features in a combination is determined by the ability to uniquely distinguish other algorithms. For example, consider data imputation algorithms, all of which aim to fill in missing values. The method of imputing missing values ​​can be used as a distinguishing feature. Algorithm A uses the mean to imput missing values, Algorithm B uses the median, and Algorithm C uses interpolation. Therefore, the method of imputing missing values ​​alone can clearly distinguish these three algorithms. Another example is data transformation algorithms, where standardization of numerical data is a common operation. At this point, the combination of the two features, "suitable data type" and "mathematical method used," can serve as distinguishing features. Algorithm D is suitable for normally distributed data and uses the Z-score standardization method; Algorithm E is suitable for non-normally distributed data and uses the Min-Max standardization method; Algorithm F is suitable for data with specific interval restrictions and uses the decimal scaling standardization method. These three algorithms belong to the data transformation standardization algorithm category. When faced with user age data that conforms to a normal distribution, Algorithm D can standardize it to a standard normal distribution with a mean of 0 and a standard deviation of 1. For grade data, its distribution may not conform to a normal distribution, and it is desirable to uniformly map the scores to the [0,1] interval. The Min-Max standardization method of Algorithm E is more suitable. If the numerical range of the data is large, such as when processing monetary data, in order to avoid the standardized data being too small or too large, the decimal scaling standardization method of Algorithm F can perform standardization based on the maximum absolute value of the data. Therefore, the distinguishing features formed by the combination of the suitable data type and the mathematical method used can differentiate the algorithms.

[0043] (4) Generate a unique identifier for all algorithms in the same category based on the distinguishing features.

[0044] Specifically, based on the identified distinguishing features, a unique identifier is generated for each algorithm. This identifier represents only one corresponding algorithm. With the unique identifier, when selecting a suitable algorithm based on the data preprocessing request, the specific algorithm can be quickly and accurately located, avoiding algorithm confusion and improving the accuracy and efficiency of algorithm selection.

[0045] Furthermore, the different values ​​of each distinguishing feature are used to define and identify the algorithm. For example, when the distinguishing feature is the applicable data type, numerical algorithms are represented by "N", and character algorithms by "C". When the distinguishing feature is the core mathematical method, regression analysis algorithms are represented by "R01", and clustering algorithms by "C01". The number of values ​​for each distinguishing feature is counted, and the required number of bits for the code is determined based on the number of values. The category code and the codes of each distinguishing feature are combined in a certain order to form the unique identifier of the algorithm. For example, an algorithm belongs to the data transformation category, is applicable to numerical data types, and uses regression analysis to process large amounts of data. The category code for the data transformation category is "D02", the code for numerical data is "N" (assuming 1 bit), the code for regression analysis is "R01" (assuming 3 bits), and the code for large amounts of data is "B" (assuming 1 bit). Then its unique identifier is "D02N01B".

[0046] S102. Identify the dependencies between each of the data preprocessing requests and construct a request relationship graph, which is used to characterize the task correlation between each data preprocessing request.

[0047] Specifically, the dependency relationship between various data preprocessing requests refers to the fact that the execution of a data preprocessing request depends on the results of other requests, while the request relationship graph presents the relationship between various data preprocessing requests in a graphical form.

[0048] Optionally, constructing a request relationship graph based on dependencies includes:

[0049] (1) Treat each data preprocessing request as a node;

[0050] Specifically, a node refers to the basic unit in the request relationship graph. Each node represents a data preprocessing request. For example, if two data preprocessing requests are issued by edge heterogeneous devices, it means that there are two nodes in the constructed request relationship graph. By breaking down the complex data processing flow into identifiable individuals, a foundation can be laid for subsequent analysis of the relationships between data preprocessing requests.

[0051] (2) Determine the dependency direction between each data preprocessing request based on the data content and processing target corresponding to each data preprocessing request, and determine the connection direction of the edges between corresponding nodes based on the dependency direction;

[0052] It should be noted that the dependency direction is the direction of the connection between the edges between nodes in the request relationship graph. For example, in one embodiment, if the second request depends on the output of the first request to be processed, then the connection relationship between the first request and the second request is that the first request points to the second request.

[0053] Specifically, the dependency direction between each data preprocessing request can be determined based on the sequential relationship between the data content and the processing objectives. The overlap of data content in each preprocessing request can be determined, with preprocessing requests having a non-zero overlap being considered related requests. The dependency direction of each request within a related request can be determined based on the dependency relationship of the processing objectives. For example, if request A performs noise reduction on image data to improve image clarity, and request B performs feature extraction on the clearer image processed by request A for image recognition, then the dependency direction between request A and request B is from request A to request B. The dependency direction between each data preprocessing request is the direction of the edges connecting nodes in the request relationship graph.

[0054] (3) Calculate the degree of dependency between each data preprocessing request based on the data content and processing target corresponding to the data preprocessing request, and use the degree of dependency as the value of the edge between the corresponding nodes to construct the request relationship graph.

[0055] Specifically, firstly, the overlap of data content and the relevance of processing targets in each data preprocessing request are determined. The weighted sum of the data content overlap and processing target relevance is then used to determine the dependency degree. For data preprocessing requests with a non-zero overlap, the weight of the data content corresponding to the request is greater than the weight of the processing target. For example, if the data content overlap of request A and request B is 0.7 and the processing target relevance is 0.5, then the data content overlap weight is set to 0.6 and the processing target weight is set to 0.4. Therefore, the dependency degree is 0.6*0.7 + 0.5*0.4 = 0.62. Further, the dependency degree is defined as the edge length in the request relationship graph. Based on the determined dependency direction and dependency degree, directed edges with a length equal to the dependency degree are drawn between nodes, thus constructing the request relationship graph. The request relationship graph can visually display the dependencies between various data preprocessing requests, providing a basis for subsequent task scheduling and resource allocation. Since the edges in the request relationship graph have a directional attribute, this directional attribute indicates the direction of data flow, that is, the data preprocessing results corresponding to the upstream node will be used by the downstream node.

[0056] S103. Bind multiple concurrent data preprocessing requests according to the request relationship graph to generate multiple request groups, wherein the number of data preprocessing requests in each request group is not exactly the same.

[0057] Specifically, based on the request relationship graph, requests with close dependencies or similar processing needs are grouped together to form multiple request groups, with requests within each group being interconnected. For example, different processing requests for the same data source, such as requests for cleaning, classifying, and mining user information in a database, can be grouped into one request group. It should be noted that different request groups may contain the same or different numbers of requests.

[0058] S104. Determine the data processing volume and urgency level based on each data preprocessing request, and combine the data processing volume and urgency level to obtain the request processing priority for each request group.

[0059] Specifically, the data processing scale reflects factors such as the amount of data processed in each request and the processing complexity. The implementation steps for determining the data processing scale and urgency for each data preprocessing request include:

[0060] (1) Extract feature information from each of the data preprocessing requests;

[0061] Specifically, when processing data preprocessing requests, it is necessary to extract feature information that reflects the characteristics of each request. This feature information includes data features and processing target features, etc. Feature extraction for each data preprocessing request can be performed using a deep neural network. For the implementation steps of feature extraction for each data preprocessing request using a deep neural network, please refer to the description in related technologies; they will not be repeated here. Preferably, the processing target in the data preprocessing request is determined, and the distinguishing feature type encoded in the algorithm database of the processing target is used as a prompt information to extract feature information corresponding to the distinguishing feature type.

[0062] (2) Determine the data processing level based on the feature information;

[0063] Specifically, the data processing level is a comprehensive measure of the difficulty and resource consumption of data preprocessing requests. It involves quantifying the data features in the feature information and the features of the processing target, and then weighting and summing the quantified feature information to obtain a comprehensive processing level value. This processing level data is then determined as the data processing level of the data preprocessing request.

[0064] (3) Calculate the time difference for each of the data preprocessing requests, the time difference being the difference between the deadline of each of the data preprocessing requests and the current time;

[0065] (4) Determine the level of urgency based on the time difference.

[0066] Specifically, the time difference is calculated by taking the difference between the deadline and the current time for each data preprocessing request. This time difference is a crucial indicator of the urgency of a data preprocessing request, and its urgency can be determined based on the time difference. Furthermore, the urgency can be categorized into high, medium, and low levels based on the time difference. For example, requests with a time difference of less than 10 minutes can be defined as high urgency, those with a time difference between 10 and 60 minutes are medium urgency, and those with a time difference greater than 60 minutes are low urgency. In other words, the time difference is inversely proportional to the urgency; the smaller the time difference, the higher the urgency of the request.

[0067] Furthermore, after determining the data processing volume and urgency of each data preprocessing request, the steps to integrate the data processing volume and urgency to obtain the request processing priority for each request group include:

[0068] (1) Add the data processing level of each data preprocessing request in the request group to obtain the target data processing level of the request group;

[0069] Specifically, a request group contains multiple data preprocessing requests, each with its own data processing scale. This scale reflects the resources (such as computing resources, storage resources, etc.) required to process the request and the complexity of the processing. The sum of the data processing scales of all data preprocessing requests in the request group is the target data processing scale of the request group.

[0070] (2) Iterate through the urgency of each data preprocessing request in the request group and determine the largest urgency among the multiple urgency levels as the target urgency of the request group;

[0071] Specifically, when determining the target urgency level for a request group, all requests within the group are iterated through, and the highest urgency value is identified as the target urgency level. This ensures that urgent requests are processed promptly, avoiding delays caused by other, less urgent requests.

[0072] (3) Determine the priority score for each request group based on the target data processing volume and the target urgency;

[0073] Specifically, based on actual needs and experience, corresponding weights can be assigned to the target data processing volume and the target urgency, respectively. It should be noted that the weight of the target urgency is greater than the weight of the target data processing volume. Then, the target data processing volume and the target urgency are weighted and summed using the corresponding weights to obtain the priority score of the requesting group.

[0074] (4) Sort all request groups in descending order of priority scores, and determine the request processing priority based on the sorting results.

[0075] Specifically, after obtaining the priority scores of the requesting groups, all requesting groups are prioritized according to their scores from highest to lowest, with higher-scoring groups having higher priority. The system then processes these requesting groups sequentially in this order, thereby rationally allocating resources and task execution order, improving the efficiency and effectiveness of data processing. Furthermore, prioritization is conducted based on the target data processing volume and urgency of each requesting group, comprehensively considering the scale and time urgency of the tasks involved in each group. This avoids the problem of incomplete evaluation of the overall situation of the requesting groups due to focusing on only a single factor.

[0076] S105. Sort the multiple request groups according to the request processing priority to obtain the first priority request queue.

[0077] Specifically, request groups are sorted from highest to lowest request processing priority to obtain the first priority request queue. The system will prioritize processing the request groups ranked higher in the first priority request queue to ensure that urgent requests with large data volumes are processed in a timely manner.

[0078] S106. Sort the data preprocessing requests within the request group according to the data processing causal relationship within the request group, and combine them with the first priority request queue to obtain the second priority request queue.

[0079] Specifically, the causal relationship in data processing refers to the sequential execution order of requests within a request group. Based on the first priority request queue, requests within each request group are sorted according to their causal relationships to obtain the second priority request queue. The second priority request queue comprehensively considers both the priority of the request group and the sequential order of requests within that group, enabling a more rational arrangement of data preprocessing tasks.

[0080] S107. Traverse the data preprocessing requests in the order of the second priority request queue. For the target data preprocessing request currently being traversed, determine the target data preprocessing algorithm corresponding to the data preprocessing request based on the operating status data of the intelligent agent and the edge heterogeneous device.

[0081] Specifically, before traversing the data preprocessing requests in the order of the second priority request queue, the process includes: traversing each request group in the order of the second priority request queue; for a target request group, determining its representative urgency level, which refers to the smallest time difference (the difference between the request processing completion time and the current time) among the target request groups, i.e., the most urgent level. The queueing segment following the target request group is obtained based on the numerical relationship between the representative urgency level and the urgency levels of each data preprocessing request after the target request group.

[0082] Specifically, if a data preprocessing request is more urgent than the representative urgency level, it is identified as a candidate data preprocessing request. The candidate request group to which the candidate data preprocessing request belongs is located. A segment of the candidate request group, consisting of all preprocessing data requests from the first data preprocessing request in the candidate request group to the candidate data preprocessing request, is designated as a queueing segment. The insertion position of the queueing segment in the target request group is determined based on its urgency level, and the segment is inserted into the target request group according to this position. The urgency level of a queueing segment is determined by the smallest time difference (the difference between the request completion time and the current time) among the queueing segments, i.e., the most urgent segment. If there are multiple queueing segments, their order is also determined by the smallest time difference (the difference between the request completion time and the current time) among the queueing segments, i.e., the most urgent segment.

[0083] Furthermore, the agent is a pre-trained model capable of outputting a data preprocessing algorithm corresponding to a data preprocessing request based on the operational status data and data preprocessing requests of edge heterogeneous devices. The agent can select the most suitable preprocessing algorithm for each request based on the operational status data of edge heterogeneous devices (such as CPU utilization, memory usage, clock speed, disk I / O speed, and network bandwidth) and the data preprocessing request.

[0084] Optionally, the implementation steps for determining the target data preprocessing algorithm corresponding to the data preprocessing request based on the operational status data of intelligent agents and edge heterogeneous devices include:

[0085] (1) Determine the type of data preprocessing algorithm;

[0086] (2) Extract the features of the original data according to the algorithm category;

[0087] Specifically, after receiving a data preprocessing request, the first step is to determine which category the algorithm required for the data preprocessing request belongs to. Since the algorithms in the algorithm database have already been classified (such as data transformation, feature extraction, etc.), the corresponding algorithm category can be determined based on the processing goal of the request (such as removing noise from the data or standardizing the data).

[0088] Furthermore, after determining the algorithm category, feature extraction is performed on the raw data carried in the data preprocessing request to obtain the raw data features.

[0089] (3) Obtain the real-time status data of the edge heterogeneous device, and extract status features based on the real-time status data and the algorithm category;

[0090] It should be noted that when acquiring real-time status data of heterogeneous edge devices, the processing objectives in the data preprocessing requests are different, resulting in different resources and hardware requirements for the requests. Therefore, the status of the heterogeneous edge devices that need to be monitored when acquiring real-time status data is also different, and their real-time status is mainly determined by the processing objectives in the request.

[0091] Optionally, before extracting state features from real-time state data, the following steps are also included:

[0092] (1) Determine the type of resource to be invoked based on the algorithm type;

[0093] Specifically, different data preprocessing algorithms rely on different types of resources to complete their tasks. For example, deep learning-based image feature extraction algorithms typically require significant computing resources (such as the powerful computing capabilities of GPUs) to handle complex neural network operations; while simple data cleaning algorithms may primarily rely on CPU and memory resources to read, process, and store data. Therefore, once the type of data preprocessing algorithm is determined, it becomes clear which types of resources are needed to execute that algorithm.

[0094] (2) Determine the characteristic indicators based on the resource type;

[0095] Specifically, each resource type has certain characteristic indicators that reflect its status and performance. For CPU resources, these indicators might include CPU utilization, core frequency, and cache hit rate; for memory resources, they include memory utilization, memory bandwidth, and page swapping rate; and for GPU resources, important indicators include GPU utilization, video memory usage, and computational throughput. Once the type of resource to be accessed is determined, the specific characteristic indicators associated with it can be identified.

[0096] (3) Determine the target features based on the feature indicators, and construct a state feature vector based on the target features.

[0097] Specifically, after identifying the characteristic indicators related to resource types, it is necessary to select the most critical and representative indicators for the current algorithm execution as target features. Once the target features are determined, they are combined in a specific order to form the state feature vector. The state feature vector is a numerical vector that comprehensively describes the current state of the relevant resources. For example, if the target features are CPU utilization, memory usage, and GPU utilization, then the state feature vector might be in the form of [CPU utilization, memory usage, GPU utilization], with each element corresponding to a value of a target feature. This state feature vector serves as a crucial basis for subsequent algorithm selection and resource scheduling by the intelligent agent. The agent can use the state feature vector to determine whether the current resource state of the device is suitable for executing a specific algorithm.

[0098] Furthermore, before determining the target data preprocessing algorithm corresponding to the data preprocessing request based on the operational status data of the intelligent agent and edge heterogeneous devices, the process also includes:

[0099] 3.1 Calculate the error between the sample features and the preset features, and calculate the gradient between the error and the weights based on the chain method;

[0100] It should be noted that the sample features are the feature values ​​obtained by weighted summation of the state feature vectors initially extracted or selected from the real-time state data of edge heterogeneous devices, while the preset features are the standard values ​​that the sample features are expected to reach in advance. By calculating the difference between the sample features and the preset features (for example, by using the sum of squares of the differences to quantify the error), an error value that measures the degree to which the sample features deviate from the preset features can be obtained.

[0101] Furthermore, the chain method refers to the chain rule, a method in calculus for finding the derivative of a composite function. The chain method can be used to calculate the rate of change of the error with respect to the weights. For the implementation process of calculating the gradient based on the chain method, please refer to the description in the relevant technology, which will not be elaborated here.

[0102] 3.2 Update the weights based on gradient descent, update the sample features using the updated weights, and return to the step of calculating the error between the sample features and the preset features until the error is less than a threshold.

[0103] Specifically, after obtaining the gradient corresponding to the weights in the sample features, the weights are updated in the opposite direction of the gradient, which gradually reduces the error. Based on the calculated gradient, the weights are updated with a certain step size (learning rate), and the sample features are recalculated using the updated weights. The error is then recalculated again until the error is less than a threshold, at which point the weight update stops. It should be noted that the threshold is set according to actual needs, and is not limited in this embodiment.

[0104] 3.3 When the error is less than the threshold, the weight corresponding to each component in the real-time status data is taken as the target weight.

[0105] Specifically, after multiple iterations, once the error is less than the threshold, the weight corresponding to each component in the real-time status data (such as CPU utilization, memory usage, etc.) is the optimized and adjusted weight, which is then determined as the target weight. The target weight reflects the importance of each real-time status feature vector in accurately describing the device state or meeting subsequent processing requirements under the current device state. When extracting status features from the real-time status data later, processing the data according to the target weight yields state features that more accurately reflect the actual operating conditions of the device and match the algorithm's requirements.

[0106] Furthermore, the state feature vector is weighted and summed using target weights to obtain the state features. The state feature vector represents one dimension of the device's operating state, while the target weights reflect the relative importance of this component in comprehensively describing the device's state. Through this weighted summation operation, the resulting state features can more comprehensively and accurately reflect the actual operating state of heterogeneous edge devices. This state feature will help the agent better determine whether a device is suitable for executing a specific algorithm when matching target data preprocessing algorithms, thus making a more reasonable algorithm selection decision.

[0107] (4) The agent matches the target data preprocessing algorithm according to the original data features and the state features.

[0108] Specifically, the AI, based on the extracted raw data features and state features, searches for the most suitable target data preprocessing algorithm in the algorithm database. It can perform similarity analysis on the raw data features and state features using an established matching model or algorithm, and determine the algorithm corresponding to the raw data features with a similarity higher than a similarity threshold as the target data preprocessing algorithm. It should be noted that the similarity threshold is set according to actual needs; in this embodiment, it is not limited.

[0109] S108. Determine the time slice of the target data preprocessing request based on the complexity of the target data preprocessing algorithm and the processing data volume of the target data preprocessing request, wherein the resources and operating status of the edge heterogeneous device within the time slice are matched with the complexity and the data volume.

[0110] Specifically, time slicing refers to a segment of execution time allocated to each data preprocessing request. The execution time for each request is rationally determined based on the complexity of the target data preprocessing algorithm (such as computational load, number of iterations, etc.) and the volume of data to be processed. Requests with high complexity and large data volumes are allocated longer time slices, and vice versa. This allows for reasonable scheduling of device processing time, preventing a single request from consuming excessive resources and affecting the processing of other requests. Furthermore, to ensure that all requests can be executed at once, a backward-engineering approach is used to determine the time slice corresponding to each request. The length of the time slice is calculated based on the complexity and the volume of data to be processed. Real-time monitoring and historical data analysis of the operating status of edge heterogeneous devices are performed to predict their idle periods in the future. Combining historical processing data and real-time performance monitoring data of the edge heterogeneous devices, the actual processing efficiency of the devices within each predicted idle period is evaluated. Based on the actual processing efficiency and the time slice length, the idle periods of the edge heterogeneous devices are determined, and these idle periods are defined as time slices. For example, in one embodiment, the time slice length is determined to be 30 minutes. It is predicted that edge heterogeneous devices are typically relatively idle during the periods of 10:00-10:30 AM, 3:00-3:30 PM, and 8:00-8:30 PM. Analysis of historical device data reveals that during the 10:00-10:30 AM period, the device's CPU performance is relatively stable, resulting in high processing efficiency. However, during the 8:00-8:30 PM period, due to increased temperature after prolonged operation, CPU performance slightly decreases, leading to relatively lower processing efficiency. For the target request requiring 30 minutes of processing time, considering processing efficiency, selecting the 10:00-10:30 AM idle period as the time slice is more appropriate. This ensures that the request is executed all at once within the specified time, maximizes the utilization of device resources, and improves overall processing efficiency.

[0111] Furthermore, during the request execution process, the operating status and task progress of edge heterogeneous devices are continuously monitored. If the actual processing situation does not match the expectations, such as a sudden failure of the device during task execution leading to a decrease in processing efficiency, or a sudden increase in task data volume for some reason, causing the originally planned time slices to be unable to complete the task, it is necessary to re-evaluate the time slice length and device idle time periods, recalculate the time slice length based on the new situation, predict the device idle time periods again, and select appropriate time periods for task adjustment to ensure that the task can be completed smoothly.

[0112] S109. The edge heterogeneous device processes the raw data in the target data preprocessing request using the target data preprocessing algorithm within the time slice corresponding to the target data preprocessing request, according to the order in the second priority request queue.

[0113] Specifically, edge heterogeneous devices process raw data using selected target data preprocessing algorithms within the time slice corresponding to each request, following the order of the second priority request queue. The devices complete their processing tasks within the specified time and feed the results back to the system. Raw data refers to the unprocessed initial data collected by the edge heterogeneous devices. Within the allocated time slice, the devices use the selected target data preprocessing algorithms to process the raw data, transforming it into data that meets business requirements, thus supporting subsequent data analysis and decision-making.

[0114] Furthermore, after processing the raw data in the target data preprocessing request using the target data preprocessing algorithm, the process includes:

[0115] (1) Construct a performance metric function based on the execution time of the target preprocessing algorithm;

[0116] Specifically, the performance metric function can be represented by the following formula:

[0117]

[0118] Where T is the execution time.

[0119] Performance metrics are used to measure the execution time efficiency of target data preprocessing algorithms. The shorter the execution time of the target preprocessing algorithm, the better the performance of the algorithm.

[0120] (2) Evaluate the execution efficiency of the target preprocessing algorithm using the performance metric function.

[0121] Specifically, under a defined evaluation environment, the target preprocessing algorithm is run multiple times using prepared test data to obtain more accurate results. During each run, the execution time of the target preprocessing algorithm is recorded, and this execution time is substituted into the performance metric function to calculate the corresponding performance index. Ideally, the performance metric function should monotonically increase as the algorithm's execution time decreases, with a non-linear growth trend. The exponential function possesses this characteristic; it is monotonically increasing within its domain, and its growth rate varies with the independent variable, effectively reflecting the impact of small changes in execution time on performance evaluation. Therefore, an exponential function with the natural constant e as its base is chosen as the basic form.

[0122] The adaptive selection method for data preprocessing algorithms for edge heterogeneous devices provided in this embodiment receives multiple concurrent data preprocessing requests from edge heterogeneous devices, identifies the dependencies between the requests and constructs a request relationship graph, and generates multiple request groups based on the request relationship graph, rationally grouping complex concurrent tasks into multiple request groups. This simplifies the complex problem by clearly organizing the originally chaotic and disordered requests according to task relevance. Secondly, by determining the data processing volume and urgency of each data preprocessing request, the two are combined to obtain the request processing priority of the request group, and then the multiple request groups are sorted to form a first priority request queue. Simultaneously, considering the causal relationships of data processing within the request group, a second priority request queue is obtained by combining the first priority request queue. This dual sorting mechanism comprehensively and meticulously plans the request processing order, ensuring that urgent requests with large data processing volumes are processed first, and that requests within a group are executed in the correct logical order, effectively avoiding errors and inefficiencies caused by disordered processing order. Thirdly, the target data preprocessing algorithm is determined based on the operating status data of the intelligent agent and the edge heterogeneous devices. The intelligent agent integrates the characteristics of raw data and device status, and accurately matches the most suitable algorithm from the algorithm database. It can make full use of hardware resources, reduce unnecessary computing overhead and waiting time, and achieve faster data processing and response speed without sacrificing accuracy. From receiving a request to completing the processing, the entire process realizes automated and intelligent task scheduling, algorithm selection and resource allocation, which effectively improves the accuracy, timeliness and resource utilization of data preprocessing, thereby improving the overall performance of the system.

[0123] Corresponding to the aforementioned embodiment of an adaptive selection method for data preprocessing algorithms of edge heterogeneous devices, this application also provides an embodiment of an adaptive selection device for data preprocessing algorithms of edge heterogeneous devices.

[0124] Figure 2 This is a schematic diagram of an embodiment of the adaptive selection device for data preprocessing algorithms of edge heterogeneous devices provided in this application. Please refer to... Figure 2 The apparatus provided in this embodiment includes a processing module 210, a sorting module 220, and an execution module 230; wherein,

[0125] The processing module 210 is used to receive multiple concurrent data preprocessing requests from the edge heterogeneous device, and the data processed by each data preprocessing request is not exactly the same;

[0126] The processing module 210 is also used to identify the dependencies of each of the data preprocessing requests and construct a request relationship graph, which is used to characterize the task correlation between each data preprocessing request.

[0127] The processing module 210 is also used to bind multiple concurrent data preprocessing requests according to the request relationship graph, and generate multiple request groups, wherein the number of data preprocessing requests in each request group is not exactly the same;

[0128] The sorting module 220 is used to determine the data processing volume and urgency level according to each data preprocessing request, and to combine the data processing volume and urgency level to obtain the request processing priority of each request group.

[0129] The sorting module 220 is also used to sort multiple request groups according to the request processing priority to obtain a first priority request queue;

[0130] The sorting module 220 is further configured to sort the data preprocessing requests within the request group according to the data processing causal relationship within the request group, and combine the first priority request queue to obtain a second priority request queue.

[0131] The execution module 230 is used to traverse the data preprocessing requests in the order of the second priority request queue, and for the currently traversed target data preprocessing request, determine the target data preprocessing algorithm corresponding to the data preprocessing request based on the operating status data of the intelligent agent and the edge heterogeneous device.

[0132] The execution module 230 is further configured to determine the time slice of the target data preprocessing request based on the complexity of the target data preprocessing algorithm and the processing data volume of the target data preprocessing request, wherein the resources and operating status of the edge heterogeneous device within the time slice are matched with the complexity and the data volume.

[0133] The execution module 230 is further configured to have the edge heterogeneous device process the raw data in the target data preprocessing request using the target data preprocessing algorithm within the time slice corresponding to the target data preprocessing request, according to the order in the second priority request queue.

[0134] The apparatus of this embodiment can be used to perform... Figure 1 The steps of the method embodiment shown are similar in principle and process, and will not be repeated here.

[0135] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0136] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0137] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. An adaptive selection method for data preprocessing algorithms of edge heterogeneous devices, characterized in that, The method includes: Receive multiple concurrent data preprocessing requests from the edge heterogeneous device, where each data preprocessing request processes different data; Identify the dependencies between each data preprocessing request and construct a request relationship graph, which is used to characterize the task correlation between each data preprocessing request; Multiple concurrent data preprocessing requests are bound together according to the request relationship graph to generate multiple request groups, and the number of data preprocessing requests in each request group is not exactly the same. The data processing volume and urgency are determined based on each data preprocessing request, and the data processing volume and urgency are combined to obtain the request processing priority for each request group; Multiple request groups are sorted according to the request processing priority to obtain the first priority request queue; The data preprocessing requests within the request group are sorted according to the causal relationship of data processing within the request group, and combined with the first priority request queue, a second priority request queue is obtained. The data preprocessing requests are traversed according to the order in the second priority request queue. For the currently traversed target data preprocessing request, a target data preprocessing algorithm corresponding to the data preprocessing request is determined based on the operating status data of the intelligent agent and the edge heterogeneous device; including: Determine the type of data preprocessing algorithm; Extract features from the original data according to the algorithm category; Acquire real-time status data of the edge heterogeneous device, and extract status features based on the real-time status data and the algorithm category; The intelligent agent matches the target data preprocessing algorithm based on the original data features and the state features; The time slice of the target data preprocessing request is determined based on the complexity of the target data preprocessing algorithm and the processing data volume of the target data preprocessing request, wherein the resources and operating status of the edge heterogeneous device within the time slice are matched with the complexity and the data volume. The edge heterogeneous device processes the raw data in the target data preprocessing request using the target data preprocessing algorithm within the time slice corresponding to the target data preprocessing request, according to the order in the second priority request queue.

2. The method according to claim 1, characterized in that, The step of identifying the dependencies between the various data preprocessing requests and constructing a request relationship graph includes: Treat each data preprocessing request as a node; The dependency direction between each data preprocessing request is determined based on the data content and processing target corresponding to each data preprocessing request, and the connection direction of the edges between corresponding nodes is determined based on the dependency direction. The dependency degree between each data preprocessing request is calculated based on the data content and processing target corresponding to the data preprocessing request, and the dependency degree is used as the value of the edge between the corresponding nodes to construct the request relationship graph.

3. The method according to claim 1, characterized in that, The step of determining the data processing volume and urgency level based on each data preprocessing request includes: Extract feature information from each of the data preprocessing requests; The data processing volume level is determined based on the aforementioned feature information; Calculate the time difference for each of the data preprocessing requests, the time difference being the difference between the deadline for each data preprocessing request and the current time; The level of urgency is determined based on the time difference.

4. The method according to claim 1, characterized in that, The process of combining the data processing volume and the urgency level to obtain the request processing priority for each request group includes: The data processing volume of each data preprocessing request in the request group is added together to obtain the target data processing volume of the request group; Iterate through the urgency of each data preprocessing request in the request group, and determine the largest urgency among the multiple urgency levels as the target urgency of the request group; A priority score for each request group is determined based on the target data processing volume and the target urgency. All request groups are sorted in descending order of priority score, and the request processing priority is determined based on the sorting results.

5. The method according to claim 1, characterized in that, Before extracting state features based on the real-time state data, the method includes: The type of resource to be invoked is determined based on the algorithm type. Determine characteristic indicators based on the resource type; The target features are determined based on the feature indicators, and a state feature vector is constructed based on the target features.

6. The method according to claim 1, characterized in that, The step of extracting state features based on the real-time state data includes: Before extracting state features based on the real-time state data, the method further includes: Calculate the error between sample features and preset features, and calculate the gradient between the error and the weights based on the chain method; The weights are updated based on gradient descent, and the sample features are updated using the updated weights. The process continues until the error between the sample features and the preset features is less than a threshold. When the error is less than the threshold, the weight corresponding to each component in the real-time status data is taken as the target weight. The step of extracting state features based on the real-time state data includes: The state feature is obtained by weighting and summing the state feature vector using the target weights.

7. The method according to claim 1, characterized in that, Before receiving the data preprocessing request, the method includes: Establish an algorithm database; The algorithms in the algorithm database are classified according to the algorithm processing objectives, and each category of algorithm is coded. For algorithms under different categories, determine the distinguishing features corresponding to all algorithms under the same category, wherein the values ​​of the distinguishing features corresponding to all algorithms under the same category are different; Generate unique identifiers for all algorithms within the same category based on the distinguishing features.

8. The method according to claim 1, characterized in that, After processing the raw data in the target data preprocessing request using the target data preprocessing algorithm, the method includes: Construct a performance metric function based on the execution time of the target data preprocessing algorithm; The execution efficiency of the target data preprocessing algorithm is evaluated using the performance metric function.

9. An adaptive selection device for data preprocessing algorithms of edge heterogeneous devices, characterized in that, The device includes a processing module, a sorting module, and an execution module; The processing module is used to receive multiple concurrent data preprocessing requests from the edge heterogeneous device, and the data processed by each data preprocessing request is not exactly the same; The processing module is also used to identify the dependencies between each of the data preprocessing requests and construct a request relationship graph, which is used to characterize the task correlation between each data preprocessing request. The processing module is also used to bind multiple concurrent data preprocessing requests according to the request relationship graph, and generate multiple request groups, wherein the number of data preprocessing requests in each request group is not exactly the same; The sorting module is used to determine the data processing volume and urgency level based on each data preprocessing request, and to combine the data processing volume and urgency level to obtain the request processing priority for each request group. The sorting module is also used to sort multiple request groups according to the request processing priority to obtain a first priority request queue; The sorting module is further configured to sort the data preprocessing requests within the request group according to the data processing causal relationship within the request group, and combine the first priority request queue to obtain a second priority request queue. The execution module is configured to traverse data preprocessing requests according to the order in the second priority request queue, and for the currently traversed target data preprocessing request, determine the target data preprocessing algorithm corresponding to the data preprocessing request based on the operating status data of the intelligent agent and the edge heterogeneous device; including: Determine the type of data preprocessing algorithm; Extract features from the original data according to the algorithm category; Acquire real-time status data of the edge heterogeneous device, and extract status features based on the real-time status data and the algorithm category; The intelligent agent matches the target data preprocessing algorithm based on the original data features and the state features; The execution module is further configured to determine the time slice of the target data preprocessing request based on the complexity of the target data preprocessing algorithm and the processing data volume of the target data preprocessing request, wherein the resources and operating status of the edge heterogeneous device within the time slice are matched with the complexity and the data volume. The execution module is further configured to allow the edge heterogeneous device to process the raw data in the target data preprocessing request using the target data preprocessing algorithm within the time slice corresponding to the target data preprocessing request, according to the order in the second priority request queue.

Images