Adaptive selection method and device for data preprocessing algorithm of edge heterogeneous equipment

Abstract

The invention provides a data preprocessing algorithm self-adaptive selection method and device for edge heterogeneous equipment. According to the method provided by the invention, a plurality of concurrent data preprocessing requests sent by edge heterogeneous equipment are received, the dependency relationship of each request is identified, a request relationship graph is constructed, and a request group is generated. Fusing the data processing magnitude and the emergency degree, determining group priorities, and forming a first priority queue; sorting the requests in the group according to the data processing causal relationship to obtain a second priority queue; and traversing the target request, determining a target algorithm based on the intelligent agent and equipment operation state data, determining a time slice according to algorithm complexity and data magnitude, and processing original data in the target data preprocessing request by using a target data preprocessing algorithm according to the sequence of the second request queue. According to the adaptive selection method and device for the data preprocessing algorithm of the edge heterogeneous equipment, multiple concurrent requests can be rapidly processed, and the resource utilization rate and the processing effect are effectively improved.

Description

Technical Field

[0001] The present application relates to the field of data preprocessing technology, and in particular to a method and device for adaptively selecting a data preprocessing algorithm for edge heterogeneous devices. Background Art

[0002] In the digital age, edge computing has been widely used in many fields due to its advantages such as proximity to data sources and low latency. Heterogeneous edge devices, such as various sensors and smart terminals, can collect massive amounts of data in real time. However, this data often comes in various formats and has varying quality, necessitating effective data preprocessing to meet the needs of subsequent analysis and application.

[0003] Currently, common preprocessing methods for heterogeneous edge devices primarily rely on setting a fixed data preprocessing algorithm for each device or task, or uniformly processing all data preprocessing requests according to a fixed algorithm. Because only a fixed data preprocessing algorithm is used, when different data preprocessing requests involve different data content and processing objectives, it is impossible to address the task requirements of all requests, resulting in poor data preprocessing results or excessive processing time. Furthermore, when faced with multiple data preprocessing requests, a single data preprocessing algorithm struggles to balance reasonable resource allocation and efficient processing, thus impacting the overall system efficiency. Summary of the Invention

[0004] In view of this, the present application provides a method and apparatus for adaptively selecting a data preprocessing algorithm for edge heterogeneous devices, so as to quickly process multiple concurrent data preprocessing requests and improve resource utilization and processing effects.

[0005] Specifically, this application is implemented through the following technical solutions:

[0006] A first aspect of the present application provides a method for adaptively selecting a data preprocessing algorithm for an edge heterogeneous device, the method comprising:

[0007] receiving a plurality of concurrent data preprocessing requests issued by the edge heterogeneous device, wherein the data processed by each data preprocessing request is not exactly the same;

[0008] Identifying dependencies between the data preprocessing requests and constructing a request relationship graph, wherein the request relationship graph is used to characterize the task relevance between the data preprocessing requests;

[0009] Binding multiple concurrent data preprocessing requests according to the request relationship graph to generate multiple request groups, wherein the number of data preprocessing requests in each request group is not exactly the same;

[0010] Determining a data processing level and an urgency according to each of the data preprocessing requests, and fusing the data processing level and the urgency to obtain a request processing priority for each request group;

[0011] sorting the plurality of request groups according to the request processing priority to obtain a first priority request queue;

[0012] sorting the data pre-processing requests in the request group according to the data processing causal relationship in the request group, and combining the first priority request queue to obtain a second priority request queue;

[0013] Traversing the data preprocessing requests in the order of the second priority request queue, and determining, for the currently traversed target data preprocessing request, a target data preprocessing algorithm corresponding to the data preprocessing request based on the operating status data of the intelligent agent and the edge heterogeneous device;

[0014] Determining a time slice of the target data preprocessing request based on the complexity of the target data preprocessing algorithm and the processing data level of the target data preprocessing request, wherein the resources and operating status of the edge heterogeneous device within the time slice match the complexity and the data level;

[0015] The edge heterogeneous device processes the original data in the target data preprocessing request using a target data preprocessing algorithm within a time slice corresponding to the target data preprocessing request in the order in the second priority request queue.

[0016] The second aspect of the present application provides a device for adaptively selecting a data preprocessing algorithm for an edge heterogeneous device, the device comprising a processing module, a sorting module and an execution module; wherein,

[0017] The processing module is configured to receive multiple concurrent data pre-processing requests issued by the edge heterogeneous device, where the data processed by each data pre-processing request is not exactly the same;

[0018] The processing module is further configured to identify dependencies between the data preprocessing requests and construct a request relationship graph, wherein the request relationship graph is configured to characterize the task relevance between the data preprocessing requests;

[0019] The processing module is further configured to bind multiple concurrent data preprocessing requests according to the request relationship graph to generate multiple request groups, wherein the number of data preprocessing requests in each request group is not exactly the same;

[0020] The sorting module is configured to determine a data processing level and an urgency according to each data preprocessing request, and to combine the data processing level and the urgency to obtain a request processing priority for each request group;

[0021] The sorting module is further configured to sort multiple request groups according to the request processing priorities, to obtain a first priority request queue;

[0022] The sorting module is further configured to sort data preprocessing requests in the request group according to data processing causality in the request group, and combine the first priority request queue to obtain a second priority request queue;

[0023] The execution module is configured to traverse data preprocessing requests in the second priority request queue in sequence, and determine a target data preprocessing algorithm corresponding to a target data preprocessing request currently being traversed based on an agent and running state data of the edge heterogeneous device;

[0024] The execution module is further configured to determine a time slice of the target data preprocessing request based on complexity of the target data preprocessing algorithm and a data magnitude of the target data preprocessing request, wherein resources and running states of the edge heterogeneous device in the time slice match the complexity and the data magnitude;

[0025] The execution module is further configured to cause the edge heterogeneous device to process original data in the target data preprocessing request using the target data preprocessing algorithm in a time slice corresponding to the target data preprocessing request in the second priority request queue.

[0026] The data preprocessing algorithm adaptive selection method and device of the edge heterogeneous device provided in the application can accurately, adaptively and customizably match preprocessing algorithms according to real-time state conditions of the edge heterogeneous device and requirements of data preprocessing requests, and realize efficient, high-accuracy and high-resource-utilization data preprocessing. In order to accurately match the data preprocessing request response mode with the real-time state conditions, in the first aspect, the associated requests are bound and processed on the data preprocessing request side, the scientificity of data preprocessing is improved, and a classified request set is provided for subsequent processing of the edge heterogeneous device; in the second aspect, the request groups are sorted between groups according to data magnitudes and urgency levels, and the request groups are sorted within groups according to logical relationships, so that the processing order of the requests is ensured to be processed according to the logical order, and the urgent requests are ensured to be responded to preferentially, thereby improving the scientificity of multiple request responses; in the third aspect, after the multiple requests are sorted, the most suitable preprocessing algorithm is determined according to the state of the edge heterogeneous device, the algorithm is matched with the request, and the time slice corresponding to the processing request is determined according to the resource condition of the edge heterogeneous device, that is, the time allocation and the resources are matched, and finally, the most efficient processing is realized using the most efficient algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 Flowchart of Example 1 of the method for adaptively selecting a data preprocessing algorithm for edge heterogeneous devices provided by this application;

[0028] Figure 2 This is a structural diagram of embodiment 1 of the data preprocessing algorithm adaptive selection device for edge heterogeneous devices provided by this application. DETAILED DESCRIPTION

[0029] Exemplary embodiments are described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different drawings represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with this application.

[0030] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a," "the," and "the" used in this application are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0031] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0032] Specific embodiments are given below to introduce the technical solutions of the present application in detail.

[0033] Figure 1 This is a flowchart of the first embodiment of the method for adaptively selecting a data preprocessing algorithm for edge heterogeneous devices provided by this application. Figure 1 The method provided in this embodiment may include:

[0034] S101: Receive multiple concurrent data preprocessing requests issued by the edge heterogeneous device, where data processed by each data preprocessing request is not exactly the same.

[0035] Specifically, the edge heterogeneous device refers to various devices at the edge of the network, such as sensors, smart home devices, and vehicle terminals. It should be noted that multiple concurrent data preprocessing requests can be issued by one edge heterogeneous device or multiple edge heterogeneous devices. In this embodiment, it is not limited. The edge heterogeneous device sends a request to the system to preprocess the data collected by itself. The data preprocessing request contains data source, data format, data type, processing target, deadline, and data processing task related information. It should be noted that in the presentation form of the data source, if the data amount contained in the data preprocessing request is large, the data preprocessing request only carries the link of the data, and the corresponding data can be obtained through the link carried in the request; if the data amount contained in the data preprocessing request is small, the data preprocessing request will directly carry the corresponding data itself. In this way, the actual needs of data processing are considered, data transmission and storage can be more efficient, and it is also more convenient for the system to accurately extract data features. Further, different devices have different functions and collection purposes, and the data content processed is also different. Therefore, the data processed by each data preprocessing request can be the same or different. For example, in one possible implementation, there are two concurrent data preprocessing requests, request 1 and request 2. Request 1 is to detect and screen raw material quality data, and request 2 is to analyze production process optimization based on the screened data.

[0036] It should be noted that before receiving the data preprocessing request, it also includes:

[0037] (1) Establish an algorithm database;

[0038] (2) Classify the algorithms in the algorithm database according to the processing target of the algorithm, and encode each type of algorithm;

[0039] Specifically, the algorithm database refers to a collection of various data preprocessing algorithms. In the field of data preprocessing, there are numerous algorithms of different types and functions, such as data dimensionality reduction algorithms, data standardization algorithms, data clustering algorithms, and data discretization algorithms. Establishing an algorithm database can centrally store and manage these algorithms, making it easier to call them according to different data preprocessing needs in the future.

[0040] Further, according to the processing target of the algorithm, multiple algorithms are classified, for example, algorithms for data cleaning are classified into one category, algorithms for data conversion are classified into another category, etc. In this way, the algorithm set that meets the specific processing needs can be quickly located. After classifying the algorithms, a unique code is assigned to each type of algorithm, which can more conveniently match the algorithm with the data preprocessing request in the subsequent process, improving efficiency.

[0041] (3) for algorithms in different categories, determining the distinguishing features corresponding to all algorithms in the same category, wherein the values ​​corresponding to the distinguishing features of all algorithms in the same category are different;

[0042] Specifically, although the processing objectives of algorithms in the same category are the same, the process of processing the objectives of each algorithm may be different. Feature extraction can be performed on the algorithms in the same category, and the distinguishing features of each algorithm can be screened from the features corresponding to the algorithms. That is, the distinguishing features of each algorithm are different. It should be noted that the distinguishing feature can be a single feature or a combination of multiple features. The number of features in the multiple feature combinations is determined based on the ability to uniquely distinguish other algorithms. For example, taking the data filling algorithm as an example, its processing goal is to fill in the data. The method of filling missing values ​​can be used as a distinguishing feature. Algorithm A uses the mean to fill missing values, algorithm B uses the median to fill missing values, and algorithm C uses interpolation to fill missing values. Then, the three algorithms can be clearly distinguished by the method of filling missing values ​​alone. For another example, in data conversion algorithms, standardizing numerical data is a common operation. At this time, the combination of the two features "suitable data type" and "adopted mathematical method" can be used as a distinguishing feature. Algorithm D is suitable for normally distributed data and uses the Z-score standardization method; Algorithm E is suitable for non-normally distributed data and uses the Min-Max standardization method; Algorithm F is suitable for data with specific interval restrictions and uses the decimal scaling standardization method. These three algorithms belong to the data conversion category of standardization algorithms. When faced with a user age data that conforms to the normal distribution, Algorithm D can standardize it to a standard normal distribution with a mean of 0 and a standard deviation of 1; for grade data, its distribution may not conform to the normal distribution, and it is hoped that the scores will be uniformly mapped to the [0,1] interval. The Min-Max standardization method of Algorithm E is more suitable; if the numerical range of the data spans a large range, such as processing amount data, to avoid the data being too small or too large after standardization, the decimal scaling standardization method of Algorithm F can perform standardization according to the maximum absolute value of the data. Then, the distinguishing feature composed of the combination of the two features of suitable data type and adopted mathematical method can be used to distinguish the algorithms.

[0043] (4) Generate unique identifiers for all algorithms in the same category based on the distinguishing features.

[0044] Specifically, based on the determined distinguishing features, a unique identifier is generated for each algorithm. The identifier only represents the corresponding algorithm. Through the unique identifier, when selecting the appropriate algorithm based on the data preprocessing request, the specific algorithm can be quickly and accurately located, avoiding algorithm confusion and improving the accuracy and efficiency of algorithm selection.

[0045] Furthermore, algorithms are defined and identified for each distinct value of each distinguishing feature. For example, when the distinguishing feature is an applicable data type, numerical algorithms are denoted by "N" and character algorithms by "C." When the distinguishing feature is a core mathematical method, regression analysis algorithms are denoted by "R01" and clustering algorithms by "C01." The number of values ​​for each distinguishing feature is counted, and the number of bits required for encoding is determined based on the number of values. The category code and the encoding of each distinguishing feature are combined in a certain order to form a unique identifier for the algorithm. For example, an algorithm belongs to the data conversion category, applies to numerical data types, and uses regression analysis to process large amounts of data. The category code for the data conversion category is "D02," the encoding for numerical data is "N" (assuming it is represented by 1 bit), the encoding for regression analysis is "R01" (assuming it is represented by 3 bits), and the encoding for large amounts of data is "B" (assuming it is represented by 1 bit). Thus, its unique identifier is "D02N01B."

[0046] S102: Identify the dependency relationship between each of the data preprocessing requests and construct a request relationship graph, where the request relationship graph is used to characterize the task relevance between each of the data preprocessing requests.

[0047] Specifically, the dependency relationship of each data preprocessing request means that the execution of a data preprocessing request needs to rely on the results of other requests, and the request relationship graph presents the relationship between each data preprocessing request in a graphical form.

[0048] Optionally, building a request relationship graph based on dependencies includes:

[0049] (1) Treat each data preprocessing request as a node;

[0050] Specifically, a node is the basic unit in the request relationship graph. Each node represents a data preprocessing request. For example, if a heterogeneous edge device issues two data preprocessing requests, then there are two nodes in the constructed request relationship graph. By breaking down complex data processing flows into identifiable entities, we can lay the foundation for subsequent analysis of the relationships between data preprocessing requests.

[0051] (2) determining the dependency direction between the data preprocessing requests based on the data content and processing target corresponding to each of the data preprocessing requests, and determining the connection direction of the edges between the corresponding nodes based on the dependency direction;

[0052] It should be noted that the dependency direction is the connection direction of the edges between nodes in the request relationship graph. For example, in one embodiment, the second request depends on the output of the first request to be processed, then the connection relationship between the first request and the second request is that the first request points to the second request.

[0053] Specifically, the dependency direction between each data preprocessing request can be determined based on the order of data content and processing targets, and the degree of overlap of data content in each data preprocessing request can be determined. Data preprocessing requests with a non-zero degree of overlap are associated requests. The dependency direction of each request in the associated request can be determined based on the dependency relationship of the processing targets. For example, request A performs noise reduction processing on image data to improve image clarity; request B performs feature extraction based on the clear image processed by request A for image recognition. In this case, the dependency direction between request A and request B is from request A to request B. The dependency direction between each data preprocessing request is the connection direction of the edge between nodes in the request relationship graph.

[0054] (3) Calculate the degree of dependency between the data preprocessing requests based on the data content and processing targets corresponding to the data preprocessing requests, use the degree of dependency as the value of the edge between the corresponding nodes, and construct the request relationship graph.

[0055] Specifically, the degree of data content overlap and the degree of relevance of the processing targets in each data preprocessing request are first determined. The weighted sum of the data content overlap and the processing target relevance is then used to determine the degree of dependency. Data preprocessing requests with a non-zero overlap have a data content weight greater than the processing target weight. For example, if the data content overlap of request A and request B is 0.7 and the processing target relevance is 0.5, then the data content overlap weight is set to 0.6 and the processing target weight is set to 0.4, resulting in a degree of dependency of 0.6*0.7+0.5*0.4=0.62. The degree of dependency is further determined as the length of the edge in the request relationship graph. Based on the determined dependency direction and degree, directed edges with a length equal to the degree of dependency are drawn between nodes to form a request relationship graph. The request relationship graph can intuitively display the dependencies between data preprocessing requests, providing a basis for subsequent task scheduling and resource allocation. Since the edges in the request relationship graph have a direction attribute, the direction attribute indicates the direction of data flow, that is, the data preprocessing results corresponding to the upstream node will be used by the downstream node.

[0056] S103: Bind multiple concurrent data preprocessing requests according to the request relationship graph to generate multiple request groups, where the number of data preprocessing requests in each request group is not exactly the same.

[0057] Specifically, based on the request relationship graph, requests with close dependencies or similar processing requirements are grouped together to form multiple request groups. Requests within each request group are interconnected. For example, different processing requests for the same data source, such as requests for cleaning, classifying, and mining user information in a database, can be grouped together. It should be noted that the number of requests contained in different request groups may or may not be the same.

[0058] S104: Determine a data processing level and an urgency according to each data preprocessing request, and combine the data processing level and the urgency to obtain a request processing priority for each request group.

[0059] Specifically, the data processing level reflects factors such as the amount of data processed by each request and the processing complexity. The steps for determining the data processing level and urgency based on each data preprocessing request include:

[0060] (1) extracting feature information from each of the data preprocessing requests;

[0061] Specifically, when processing data preprocessing requests, it is necessary to extract feature information that can reflect its characteristics from each request. The feature information includes data features and processing target features, etc. Feature extraction can be performed on each data preprocessing request through a deep neural network to obtain feature information. For the implementation steps of using a deep neural network to extract features from each data preprocessing request, please refer to the description in the relevant technology, which will not be repeated here. Preferably, the processing target in the data preprocessing request is determined, and the distinguishing feature type encoded by the processing target in the algorithm database is used as prompt information to extract the feature information corresponding to the distinguishing feature type.

[0062] (2) determining the data processing level according to the characteristic information;

[0063] Specifically, the data processing level is a comprehensive measure of the processing difficulty and resource consumption of the data preprocessing request. The data features and processing target features in the feature information are quantified, and the quantified feature information is weighted and summed to obtain a comprehensive processing level value. The processing level data is determined as the data processing level of the data preprocessing request.

[0064] (3) calculating a time difference for each of the data preprocessing requests, where the time difference is the difference between the deadline of each of the data preprocessing requests and the current time;

[0065] (4) Determine the degree of urgency based on the time difference.

[0066] Specifically, the time difference is calculated by calculating the difference between the deadline and the current time of each data preprocessing request. The time difference is an important indicator for measuring the urgency of data preprocessing requests, and the urgency can be determined based on the time difference. Furthermore, the urgency can be graded based on the time difference, with the urgency being divided into three levels: high, medium, and low. For example, a request with a time difference of less than 10 minutes can be defined as high urgency, a request with a time difference between 10 and 60 minutes can be defined as medium urgency, and a request with a time difference greater than 60 minutes can be defined as low urgency. In other words, the time difference is inversely proportional to the urgency; a smaller time difference indicates a higher urgency of the request.

[0067] Furthermore, after determining the data processing level and urgency of each data preprocessing request, the steps for integrating the data processing level and urgency to obtain the request processing priority of each request group include:

[0068] (1) adding the data processing levels of each data preprocessing request in the request group to obtain a target data processing level for the request group;

[0069] Specifically, a request group includes multiple data preprocessing requests, and each request has its own data processing level. This level reflects the resources required to process the request (such as computing resources, storage resources, etc.) and the complexity of the processing. The sum of the data processing levels of all data preprocessing requests in the request group is the target data processing level of the request group.

[0070] (2) traversing the urgency of each data pre-processing request in the request group, and determining the maximum urgency among the multiple urgency levels as the target urgency of the request group;

[0071] Specifically, when determining the target urgency of a request group, all requests in the group are traversed and the largest urgency value is found as the target urgency. This ensures that urgent requests can be processed in a timely manner and avoid delays due to other relatively less urgent requests.

[0072] (3) determining a priority score for each of the request groups based on the target data processing level and the target urgency;

[0073] Specifically, corresponding weights can be set for the target data processing level and target urgency based on actual needs and experience. It should be noted that the weight of the target urgency is greater than the weight of the target data processing level. The target data processing level and target urgency are then weighted and summed using the corresponding weights to obtain the priority score of the requesting group.

[0074] (4) Sort all request groups in descending order of the priority scores, and determine the request processing priority based on the sorting results.

[0075] Specifically, after obtaining the priority scores of the request groups, all request groups are prioritized from high to low based on their scores, with higher-scoring request groups receiving higher priorities. The system then processes these request groups sequentially in this order, effectively arranging resources and task execution, improving data processing efficiency and effectiveness. Furthermore, priority assessment is conducted based on the target data processing volume and target urgency of the request groups, comprehensively considering the scale and time urgency of the tasks contained in each request group. This avoids the problem of focusing on a single factor and resulting in an incomplete assessment of the overall situation of the request groups.

[0076] S105 . Sort the multiple request groups according to the request processing priorities to obtain a first-priority request queue.

[0077] Specifically, request groups are sorted from highest to lowest priority to form a first-priority request queue. The system prioritizes request groups ranked higher in the first-priority request queue, ensuring that urgent requests with large data processing volumes are processed promptly.

[0078] S106 , sorting the data pre-processing requests in the request group according to the data processing causal relationship in the request group, and combining the first priority request queue to obtain a second priority request queue.

[0079] Specifically, data processing causality refers to the order in which requests within a request group are executed. Based on the first-priority request queue, requests within each request group are sorted according to causality to create a second-priority request queue. This second-priority request queue comprehensively considers the priority of the request group and the order of requests within the request group, enabling a more efficient scheduling of data preprocessing tasks.

[0080] S107. Traverse the data preprocessing requests in the order of the second priority request queue, and for the currently traversed target data preprocessing request, determine the target data preprocessing algorithm corresponding to the data preprocessing request based on the operating status data of the intelligent agent and the edge heterogeneous device.

[0081] Specifically, before traversing the data preprocessing requests in the order of the second priority request queue, the method further includes: traversing each request group in the order of the second priority request queue, and determining a representative urgency for the target request group. The representative urgency refers to the target request group with the smallest time difference (the difference between the request processing completion time and the current time), that is, the most urgent level. The queue-jumping segment after the target request group is obtained based on the numerical relationship between the representative urgency and the urgency of each data preprocessing request after the target request group.

[0082] Specifically, if there is a data preprocessing request that is more urgent than the representative urgency, it is determined as a candidate data preprocessing request, the candidate request group where the candidate data preprocessing request is located is located, and a fragment of the candidate request group composed of all preprocessing data requests between the first data preprocessing request of the candidate request group and the candidate data preprocessing request is used as a queue-jumping segment. The insertion position in the target request group is determined according to the urgency of the queue-jumping segment, and the insertion segment is inserted into the target request group according to the insertion position. The urgency of the queue-jumping segment is determined by the time difference in the queue-jumping segment (the difference between the request processing completion time and the current time) being the smallest, that is, the most urgent degree. If there are multiple queue-jumping segments, the order of each queue-jumping segment is also determined by the time difference in the queue-jumping segment (the difference between the request processing completion time and the current time) being the smallest, that is, the most urgent degree.

[0083] Furthermore, the intelligent agent is a pre-trained model that can output the corresponding data preprocessing algorithm based on the operating status data of heterogeneous edge devices and data preprocessing requests. The intelligent agent can select the most appropriate preprocessing algorithm for each request based on the operating status data of heterogeneous edge devices (such as CPU usage, memory utilization, main frequency, disk I / O speed, and network bandwidth) and data preprocessing requests.

[0084] Optionally, the steps of implementing the target data preprocessing algorithm corresponding to the data preprocessing request based on the operating status data of the intelligent agent and the edge heterogeneous device include:

[0085] (1) Determine the type of data preprocessing algorithm;

[0086] (2) extracting raw data features according to the algorithm category;

[0087] Specifically, after receiving a data preprocessing request, it is first necessary to determine which category the algorithm required for the data preprocessing request belongs to. Since the algorithms in the algorithm database have been classified before (such as data conversion, feature extraction, etc.), the corresponding algorithm category can be judged according to the processing goal of the request (for example, whether to remove noise in the data or standardize the data, etc.).

[0088] Furthermore, after the algorithm category is determined, feature extraction is performed on the original data carried in the data preprocessing request to obtain original data features.

[0089] (3) obtaining real-time status data of the edge heterogeneous device and extracting status features based on the real-time status data and the algorithm category;

[0090] It should be noted that when obtaining real-time status data of edge heterogeneous devices, due to the different processing targets in the data preprocessing request, the resources and hardware required for the request are different. Therefore, when obtaining real-time status data, the status of edge heterogeneous devices that need to be paid attention to is also different, and their real-time status is mainly determined by the processing targets in the request.

[0091] Optionally, before extracting status features based on real-time status data, the following steps are further included:

[0092] (1) Determine the resource type to be called according to the algorithm type;

[0093] Specifically, different data preprocessing algorithms rely on different types of resources to complete their tasks. For example, deep learning-based image feature extraction algorithms typically require significant computing resources (such as the powerful computing power of GPUs) to handle complex neural network operations. Simple data cleaning algorithms, on the other hand, may rely primarily on CPU and memory resources to read, process, and store data. Therefore, once the type of data preprocessing algorithm is determined, it's clear what types of resources are needed to execute it.

[0094] (2) determining characteristic indicators according to the resource type;

[0095] Specifically, each resource type has characteristic metrics that reflect its status and performance. For CPU resources, these metrics might include CPU utilization, core frequency, and cache hit rate. For memory resources, these metrics include memory occupancy, memory bandwidth, and page swap rate. For GPU resources, important metrics include GPU utilization, video memory usage, and computational throughput. Once the resource type to be invoked is determined, the specific characteristic metrics associated with it can be determined.

[0096] (3) Determine the target feature according to the feature index, and construct a state feature vector according to the target feature.

[0097] Specifically, after determining the characteristic indicators related to the resource type, it is necessary to select the most critical and representative indicators for the current algorithm execution as target features. After determining the target features, these target features are combined in a certain order to form a state feature vector. The state feature vector is a numerical vector that comprehensively describes the current state of the relevant resources. For example, if the target features are CPU utilization, memory utilization, and GPU utilization, then the state feature vector may be in the form of [CPU utilization, memory utilization, GPU utilization], where each element corresponds to the value of a target feature. This state feature vector can serve as an important basis for subsequent intelligent agents to select algorithms and schedule resources. The intelligent agent can use the state feature vector to determine whether the current resource state of the device is suitable for executing a specific algorithm.

[0098] Furthermore, before determining the target data preprocessing algorithm corresponding to the data preprocessing request based on the operating status data of the intelligent agent and the edge heterogeneous device, the method further includes:

[0099] 3.1. Calculate the error between the sample feature and the preset feature, and calculate the gradient between the error and the weight based on the chain method;

[0100] It should be noted that the sample feature is the feature value obtained by weighted summation of the state feature vectors initially extracted or selected from the real-time state data of edge heterogeneous devices, and the preset feature is a standard value that is set in advance and expected to be achieved by the sample feature. By calculating the difference between the sample feature and the preset feature (for example, using the sum of the squares of the difference to quantify the error), an error value can be obtained to measure the degree to which the sample feature deviates from the preset feature.

[0101] Furthermore, the chain method refers to the chain rule, which is a method for finding the derivative of a composite function in calculus. The chain method can be used to calculate the rate of change of the error with respect to the weight. For the implementation process of calculating the gradient based on the chain method, please refer to the description in the relevant technology, which will not be repeated here.

[0102] 3.2. Updating the weights based on a gradient descent method, updating the sample features using the updated weights, and returning to the step of calculating the error between the sample features and the preset features until the error is less than a threshold;

[0103] Specifically, after obtaining the gradient corresponding to the weight of the sample feature, the weight is updated according to the opposite direction of the gradient, so that the error can be gradually reduced. According to the calculated gradient, the weight is updated according to a certain step (learning rate), and then the updated weight is used to recalculate the sample feature, and the error is further recalculated until the error is less than the threshold value, and the update of the weight is stopped. It should be noted that the threshold value is set according to actual needs, and in this embodiment, it is not limited.

[0104] 3.3, when the error is less than the threshold value, the weight corresponding to each component in the real-time state data is taken as the target weight.

[0105] Specifically, when the error is finally less than the threshold value after multiple iterations, the weight corresponding to each component (such as CPU usage, memory occupancy, etc.) in the real-time state data at this time is the appropriate weight after optimization and adjustment, which is determined as the target weight. The target weight reflects the importance of each real-time state feature vector in accurately describing the device state or meeting the subsequent processing needs under the current device state. When extracting state features according to real-time state data in the future, the real-time state data is processed according to the target weight, which can obtain state features that can more accurately reflect the actual running condition of the device and match the algorithm requirements.

[0106] Further, the state feature is obtained by using the target weight to weight and sum the state feature vector. The state feature vector represents one dimension of the device running state, and the target weight reflects the relative importance of the component in comprehensively describing the device state. Through such a weighted sum operation, the obtained state feature can more comprehensively and accurately reflect the actual running state of the edge heterogeneous device. This state feature can help the agent to better judge whether the device is suitable for executing a specific algorithm when the agent matches the target data preprocessing algorithm in the future, so as to make more reasonable algorithm selection decisions.

[0107] (4) The agent matches the target data preprocessing algorithm according to the original data feature and the state feature.

[0108] Specifically, the agent will find the most suitable target data preprocessing algorithm in the algorithm database according to the extracted original data feature and state feature. The similarity of the original data feature and the state feature can be analyzed through the established matching model or algorithm, and the algorithm corresponding to the original data feature with a similarity higher than the similarity threshold is determined as the target data preprocessing algorithm. It should be noted that the similarity threshold is set according to actual needs, and in this embodiment, it is not limited.

[0109] S108, determining a time slice of the target data preprocessing request based on a complexity of the target data preprocessing algorithm and a data quantity level of the target data preprocessing request, wherein a resource and a running state of the edge heterogeneous device in the time slice match the complexity and the data quantity level.

[0110] Specifically, the time slice refers to an execution time allocated to each data preprocessing request. According to the complexity of the target data preprocessing algorithm (such as the calculation amount of the algorithm, the number of iterations, etc.) and the data quantity level, the execution time of each request is reasonably determined. The request with high complexity and large data quantity level is allocated with a longer time slice, and vice versa. In this way, the processing time of the device can be reasonably arranged, and the occupation of too many resources by a certain request can be avoided, so as to affect the processing of other requests. Further, in order to ensure that all requests can be executed at one time, a backtracking method is used to determine the time slice corresponding to each request, the length of the time slice is calculated according to the complexity and the data quantity level, the running state of the edge heterogeneous device is monitored in real time and historical data is analyzed, the idle time period in the future is predicted, the actual processing efficiency of the device in each predicted idle time period is evaluated combined with historical processing data and real-time performance monitoring data of the edge heterogeneous device, and the idle time period of the edge heterogeneous device is determined according to the actual processing efficiency combined with the length of the time slice. The idle time period is determined as the time slice. For example, in an embodiment, the length of the time slice is 30 minutes, and it is predicted that the edge heterogeneous device is usually in a relatively idle state from 10:00 to 10:30 in the morning, from 3:00 to 3:30 in the afternoon, and from 8:00 to 8:30 in the evening. Through analysis of historical data of the device, it is found that the CPU performance of the device is relatively stable and the processing efficiency is high during the period from 10:00 to 10:30 in the morning. However, during the period from 8:00 to 8:30 in the evening, the CPU performance of the device decreases slightly due to the increase of temperature after long-time running, and the processing efficiency is relatively low. For the target request which needs 30 minutes of processing time, considering the processing efficiency, it is more appropriate to select the idle time period from 10:00 to 10:30 in the morning as the time slice. In this way, the request can be executed at one time within the specified time, and the device resources can be utilized to the maximum extent, thereby improving the overall processing efficiency.

[0111] Further, during the execution of the request, the running state and the task progress of the edge heterogeneous device are continuously monitored. When it is found that the actual processing situation does not match the expectation, for example, the processing efficiency decreases due to a sudden failure of the device during the execution of the task, or the data quantity of the task suddenly increases due to some reasons, so that the originally planned time slice cannot complete the task, it is necessary to reevaluate the length of the time slice and the idle time period of the device, recalculate the length of the time slice according to the new situation, predict the idle time period of the device again, and select a suitable time period for task adjustment, so as to ensure the smooth completion of the task.

[0112] S109, the edge heterogeneous device processes the original data in the target data preprocessing request using a target data preprocessing algorithm in the time slice corresponding to the target data preprocessing request in the order of the second priority request queue.

[0113] Specifically, the edge heterogeneous device processes the original data using the selected target data preprocessing algorithm in the time slice corresponding to each request in the order of the second priority request queue. The device completes the processing task within the specified time and feeds back the processing result to the system. The original data refers to the initial data collected by the edge heterogeneous device without processing. Within the allocated time slice, the device uses the selected target data preprocessing algorithm to process the original data and converts it into data that meets the business requirements, providing support for subsequent data analysis and decision-making.

[0114] Further, after processing the original data in the target data preprocessing request using the target data preprocessing algorithm, the method comprises:

[0115] (1) constructing a performance measurement function according to the execution time of the target preprocessing algorithm;

[0116] Specifically, the performance measurement function can be represented by the following formula:

[0117]

[0118] where T is the execution time.

[0119] The performance measurement function is used to measure the execution time efficiency of the target data preprocessing algorithm. The smaller the execution time of the target preprocessing algorithm, the better the performance of the algorithm.

[0120] (2) evaluating the execution efficiency of the target preprocessing algorithm using the performance measurement function

[0121] Specifically, under the determined evaluation environment, the target preprocessing algorithm is run using the prepared test data, and is run multiple times to obtain more accurate results. Each time the execution time of the target preprocessing algorithm is recorded and substituted into the performance measurement function to calculate the corresponding performance index. It is usually desired that the performance measurement function monotonically increases with the decrease of the algorithm execution time, and the growth trend is nonlinear. The exponential function has such characteristics, it monotonically increases within the domain, and the growth rate changes with the change of the independent variable, which can well reflect the influence of the slight change of the execution time on the performance evaluation. Therefore, the exponential function with natural constant e as the base is selected as the basic form.

[0122] This embodiment provides an adaptive data preprocessing algorithm selection method for edge heterogeneous devices. After receiving multiple concurrent data preprocessing requests from edge heterogeneous devices, it identifies the dependencies between the requests and constructs a request relationship graph. Based on the request relationship graph, it generates multiple request groups, rationally grouping complex concurrent tasks into multiple request groups. The previously chaotic and disorganized multiple requests are clearly organized based on task relevance, simplifying the complex problem. Secondly, by determining the data processing volume and urgency of each data preprocessing request, the two are integrated to obtain the request processing priority of the request group. The multiple request groups are then sorted to form a first-priority request queue. Simultaneously, considering the causal relationships between data processing within the request group, a second-priority request queue is derived from the first-priority request queue. Using a dual sorting mechanism, the request processing order is comprehensively and meticulously planned, ensuring that urgent and high-volume requests are processed first. Requests within a group are also executed in the correct logical order, effectively avoiding errors and inefficiencies caused by a chaotic processing order. Thirdly, the target data preprocessing algorithm is determined based on the operating status data of the intelligent agent and the edge heterogeneous devices. The intelligent agent integrates the characteristics of the original data and the status of the device, and accurately matches the most appropriate algorithm from the algorithm database. It can fully utilize hardware resources, reduce unnecessary computing overhead and waiting time, and achieve faster data processing and response speed without sacrificing accuracy. From receiving the request to completing the processing, the entire process realizes automated and intelligent task scheduling, algorithm selection and resource allocation, effectively improving the accuracy, timeliness and resource utilization of data preprocessing, thereby improving the overall performance of the system.

[0123] Corresponding to the aforementioned embodiment of a method for adaptively selecting a data preprocessing algorithm for an edge heterogeneous device, the present application also provides an embodiment of a device for adaptively selecting a data preprocessing algorithm for an edge heterogeneous device.

[0124] Figure 2 This is a structural diagram of the first embodiment of the data preprocessing algorithm adaptive selection device for edge heterogeneous devices provided by this application. Figure 2 The device provided in this embodiment includes a processing module 210, a sorting module 220 and an execution module 230; wherein,

[0125] The processing module 210 is configured to receive multiple concurrent data pre-processing requests issued by the edge heterogeneous device, where the data processed by each data pre-processing request is not exactly the same;

[0126] The processing module 210 is further configured to identify dependencies between the data preprocessing requests and construct a request relationship graph, wherein the request relationship graph is configured to characterize the task relevance between the data preprocessing requests.

[0127] The processing module 210 is further configured to bind multiple concurrent data pre-processing requests according to the request relationship graph to generate multiple request groups, wherein the number of data pre-processing requests in each request group is not exactly the same;

[0128] The sorting module 220 is configured to determine a data processing level and an urgency level according to each data pre-processing request, and to combine the data processing level and the urgency level to obtain a request processing priority for each request group;

[0129] The sorting module 220 is further configured to sort the plurality of request groups according to the request processing priority to obtain a first priority request queue;

[0130] The sorting module 220 is further configured to sort the data pre-processing requests within the request group according to the data processing causal relationship within the request group, and combine the first priority request queue to obtain a second priority request queue;

[0131] The execution module 230 is configured to traverse the data preprocessing requests in the order of the second priority request queue, and determine, for the currently traversed target data preprocessing request, a target data preprocessing algorithm corresponding to the data preprocessing request based on the operating status data of the intelligent agent and the edge heterogeneous device;

[0132] The execution module 230 is further configured to determine a time slice of the target data preprocessing request based on the complexity of the target data preprocessing algorithm and the processing data level of the target data preprocessing request, wherein the resources and operating status of the edge heterogeneous device within the time slice match the complexity and the data level;

[0133] The execution module 230 is also used for the edge heterogeneous device to process the original data in the target data preprocessing request using the target data preprocessing algorithm within the time slice corresponding to the target data preprocessing request in the order in the second priority request queue.

[0134] The device of this embodiment can be used to perform Figure 1 The steps, specific implementation principles and implementation processes of the method embodiment shown are similar and will not be repeated here.

[0135] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0136] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0137] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for adaptively selecting a data preprocessing algorithm for edge heterogeneous devices, characterized in that: The method comprises: receiving a plurality of concurrent data preprocessing requests issued by the edge heterogeneous device, wherein the data processed by each data preprocessing request is not exactly the same; Identifying dependencies between the data preprocessing requests and constructing a request relationship graph, wherein the request relationship graph is used to characterize the task relevance between the data preprocessing requests; Binding multiple concurrent data preprocessing requests according to the request relationship graph to generate multiple request groups, wherein the number of data preprocessing requests in each request group is not exactly the same; Determining a data processing level and an urgency according to each of the data preprocessing requests, and fusing the data processing level and the urgency to obtain a request processing priority for each request group; sorting the plurality of request groups according to the request processing priority to obtain a first priority request queue; sorting the data pre-processing requests in the request group according to the data processing causal relationship in the request group, and combining the first priority request queue to obtain a second priority request queue; Traversing the data preprocessing requests in the order of the second priority request queue, and determining, for the currently traversed target data preprocessing request, a target data preprocessing algorithm corresponding to the data preprocessing request based on the operating status data of the intelligent agent and the edge heterogeneous device; Determining a time slice of the target data preprocessing request based on the complexity of the target data preprocessing algorithm and the processing data level of the target data preprocessing request, wherein the resources and operating status of the edge heterogeneous device within the time slice match the complexity and the data level; The edge heterogeneous device processes the original data in the target data preprocessing request using a target data preprocessing algorithm within a time slice corresponding to the target data preprocessing request in the order in the second priority request queue.

2. The method according to claim 1, characterized in that The step of identifying the dependency relationships between the data preprocessing requests and constructing a request relationship graph includes: Treat each data preprocessing request as a node; Determining dependency directions between the data preprocessing requests according to the data content and processing target corresponding to each of the data preprocessing requests, and determining connection directions of edges between corresponding nodes according to the dependency directions; The dependency levels between the data preprocessing requests are calculated based on the data content and processing targets corresponding to the data preprocessing requests, and the dependency levels are used as the values ​​of the edges between the corresponding nodes to construct the request relationship graph.

3. The method according to claim 1, characterized in that Determining the data processing level and urgency according to each data pre-processing request includes: Extracting feature information from each of the data preprocessing requests; determining the data processing level according to the characteristic information; Calculating a time difference for each of the data preprocessing requests, where the time difference is a difference between a deadline of each of the data preprocessing requests and a current time; The urgency level is determined according to the time difference.

4. The method according to claim 1, wherein The step of fusing the data processing level and the urgency to obtain a request processing priority for each request group includes: Adding the data processing levels of each data preprocessing request in the request group to obtain a target data processing level for the request group; Traversing the urgency of each data pre-processing request in the request group, and determining the greatest urgency among the multiple urgency levels as the target urgency of the request group; Determining a priority score for each of the request groups according to the target data processing level and the target urgency; All request groups are sorted in descending order of the priority scores, and the request processing priority is determined based on the sorting results.

5. The method according to claim 1, wherein The method of determining a target data preprocessing algorithm corresponding to a data preprocessing request based on the operating status data of the intelligent agent and the edge heterogeneous device comprises: Determine the category of data preprocessing algorithms; Extracting raw data features according to the algorithm category; Acquire real-time status data of the edge heterogeneous device, and extract status features according to the real-time status data and the algorithm category; The intelligent agent matches the target data preprocessing algorithm according to the original data characteristics and the state characteristics.

6. The method according to claim 5, characterized in that Before extracting status features based on the real-time status data, the method includes: Determining the type of resource to be called according to the algorithm type; Determining characteristic indicators according to the resource type; A target feature is determined according to the feature index, and a state feature vector is constructed according to the target feature.

7. The method according to claim 5, characterized in that The extracting of state features according to the real-time state data comprises: Before extracting the state feature according to the real-time state data, the method further includes: Calculate the error between the sample feature and the preset feature, and calculate the gradient between the error and the weight based on the chain method; Updating the weights based on a gradient descent method, updating the sample features using the updated weights, and returning to the step of calculating the error between the sample features and the preset features until the error is less than a threshold; When the error is less than the threshold, the weight corresponding to each component in the real-time status data is used as the target weight; The extracting of state features according to the real-time state data comprises: The state feature vectors are weighted summed using the target weight to obtain the state feature.

8. The method according to claim 1, characterized in that Before receiving the data preprocessing request, the method includes: Establish an algorithm database; Classifying the algorithms in the algorithm database according to algorithm processing objectives, and performing category coding on each type of algorithm; For algorithms in different categories, determining a distinguishing feature corresponding to all algorithms in the same category, wherein the values ​​corresponding to the distinguishing feature for all algorithms in the same category are different; Unique identifiers are generated for all algorithms in the same category based on the distinguishing features.

9. The method according to claim 1, characterized in that After processing the original data in the target data preprocessing request using the target data preprocessing algorithm, the method includes: Constructing a performance measurement function based on the execution time of the target data preprocessing algorithm; The performance metric function is used to evaluate the execution efficiency of the target data preprocessing algorithm.

10. A device for adaptively selecting a data preprocessing algorithm for edge heterogeneous devices, characterized in that: The device includes a processing module, a sorting module and an execution module; wherein, The processing module is configured to receive multiple concurrent data pre-processing requests issued by the edge heterogeneous device, where the data processed by each data pre-processing request is not exactly the same; The processing module is further configured to identify dependencies between the data preprocessing requests and construct a request relationship graph, wherein the request relationship graph is configured to characterize the task relevance between the data preprocessing requests; The processing module is further configured to bind multiple concurrent data preprocessing requests according to the request relationship graph to generate multiple request groups, wherein the number of data preprocessing requests in each request group is not exactly the same; The sorting module is configured to determine a data processing level and an urgency according to each data preprocessing request, and to combine the data processing level and the urgency to obtain a request processing priority for each request group; The sorting module is further configured to sort the plurality of request groups according to the request processing priority to obtain a first priority request queue; The sorting module is further configured to sort the data pre-processing requests within the request group according to the data processing causal relationship within the request group, and obtain a second priority request queue by combining the first priority request queue; The execution module is configured to traverse the data preprocessing requests in the order of the second priority request queue, and for a currently traversed target data preprocessing request, determine a target data preprocessing algorithm corresponding to the data preprocessing request based on the operating status data of the intelligent agent and the edge heterogeneous device; The execution module is further configured to determine a time slice of the target data preprocessing request based on the complexity of the target data preprocessing algorithm and the processing data level of the target data preprocessing request, wherein the resources and operating status of the edge heterogeneous device within the time slice match the complexity and the data level; The execution module is also used for the edge heterogeneous device to process the original data in the target data preprocessing request using the target data preprocessing algorithm within the time slice corresponding to the target data preprocessing request in accordance with the order in the second priority request queue.

Images