Data distribution method and system

Abstract

The invention discloses a data distribution method and system, and the method comprises the steps: dividing to-be-processed data into a plurality of data subsets, determining tasks and service components corresponding to the data subsets, determining the dependency relationship between the tasks, and the operation resource demands of the service components, and carrying out the processing of the to-be-processed data. Determining a calling relationship between the service components and a communication resource demand of the calling relationship between the components; on the premise that operation resource requirements of service components on a server are less than available computing resources of the server, and communication resource requirements of a calling relationship between the components are less than or equal to available communication resources between the servers where the components are located, data processing benefit maximization is taken as a target; and the tasks, the service components and the data subsets are distributed to the servers, so that reasonable distribution of the data is realized, and subsequent data processing benefits can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data distribution method and system. Background Art

[0002] In the power industry, collected power-related data is typically centralized on a single server for processing. However, with the development of smart grids, the increasing complexity of power systems, more refined management, and advancements in data collection technology, the amount of data in the power industry is growing rapidly, and traditional centralized data processing methods can no longer guarantee data processing efficiency.

[0003] Cloud computing, as an emerging computing model, has garnered significant attention from the power industry. The industry has proposed introducing cloud computing to the power sector, using distributed computing to improve data processing efficiency. However, implementing this concept currently faces a key challenge: the lack of a comprehensive and practical solution for distributing power-related data to cloud computing centers. Summary of the Invention

[0004] Based on this, the present invention provides a data distribution method and system to solve the defect of the existing technology that there is a lack of a specific solution for reasonably distributing power-related data to a cloud computing center.

[0005] To achieve the above objectives, an embodiment of the present invention provides a data distribution method, comprising: Acquire power-related data to be processed, and divide the data to be processed into several data subsets; Determining tasks corresponding to the data subset, service components for executing the tasks, and task dependencies; Determine an inter-component call relationship based on the task, the service component for executing the task, and the task dependency, and obtain an operating resource requirement of the service component and a communication resource requirement of the inter-component call relationship; Under resource constraints, with the goal of maximizing data processing efficiency, the task, the service component used to execute the task, and the data subset associated with the task are allocated to each server in the cloud computing center; wherein the resource constraints include: the operating resource requirements of the service component on the server are less than or equal to the available computing resources of the server, and the communication resource requirements of the calling relationship between the components are less than or equal to the available communication resources between the servers where the components are located.

[0006] To achieve the above objectives, an embodiment of the present invention further provides a data distribution system, comprising: A data partitioning module is used to obtain power-related data to be processed and partition the data to be processed to obtain several data subsets; a relationship determination module, configured to determine a task corresponding to the data subset, a service component for executing the task, and a task dependency relationship; An information acquisition module is used to determine the calling relationship between components based on the task, the service component for executing the task and the task dependency, and obtain the running resource requirements of the service component and the communication resource requirements of the calling relationship between components; The data allocation module is used to allocate the tasks, service components used to execute the tasks, and data subsets associated with the tasks to various servers in the cloud computing center under resource constraints and with the goal of maximizing data processing efficiency; wherein the resource constraints include: the operating resource requirements of the service components on the servers are less than or equal to the available computing resources of the servers, and the communication resource requirements of the calling relationships between the components are less than or equal to the available communication resources between the servers where the components are located.

[0007] Compared with the prior art, the data distribution method and system disclosed in the embodiment of the present invention first obtain power-related data to be processed, and obtain several data subsets by dividing the data to be processed; then determine the tasks corresponding to the data subsets, the service components for executing the tasks, and the task dependencies; then, determine the inter-component calling relationships based on the tasks, the service components for executing the tasks, and the task dependencies, and obtain the operating resource requirements of the service components and the communication resource requirements of the inter-component calling relationships; finally, under resource constraints, with the goal of maximizing data processing efficiency, distribute the tasks, the service components for executing the tasks, and the data subsets associated with the tasks to the various servers of the cloud computing center; wherein the resource constraints include: the operating resource requirements of the service components on the servers are less than or equal to the available computing resources of the servers, and the communication resource requirements of the inter-component calling relationships are less than or equal to the available communication resources between the servers where the components are located. It can be seen from this that the embodiment of the present invention divides the data to be processed into multiple data subsets, and determines the tasks and service components corresponding to the data subsets, determines the dependencies between the tasks, the operating resource requirements of the service components, determines the calling relationships between the service components, and the communication resource requirements of the calling relationships between components; on the premise of ensuring that the operating resource requirements of the service components on the server are less than the available computing resources of the server, and that the communication resource requirements of the calling relationships between components are less than or equal to the available communication resources between the servers where the components are located, with the goal of maximizing data processing efficiency, tasks, service components and data subsets are allocated to each server, thereby achieving reasonable data distribution and helping to improve subsequent data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for use in the implementation. Obviously, the drawings described below are only some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0009] Figure 1 This is a flow chart of a data distribution method provided by one embodiment of the present invention; Figure 2 It is a structural diagram of a data distribution system provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0010] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0011] As a first preferred embodiment, see Figure 1 , is a flow chart of a data distribution method provided by an embodiment of the present invention. Specifically, the data distribution method includes steps S1 to S4: S1. Obtaining power-related data to be processed, and dividing the data to be processed to obtain several data subsets; S2. Determine tasks corresponding to the data subset, service components for executing the tasks, and task dependencies; S3. Determine an inter-component call relationship based on the task, the service component for executing the task, and the task dependency, and obtain an operating resource requirement of the service component and a communication resource requirement of the inter-component call relationship; S4. Under resource constraints, with the goal of maximizing data processing efficiency, the task, the service component used to execute the task, and the data subset associated with the task are allocated to the various servers of the cloud computing center; wherein the resource constraints include: the operating resource requirements of the service components on the server are less than or equal to the available computing resources of the server, and the communication resource requirements of the calling relationship between the components are less than or equal to the available communication resources between the servers where the components are located.

[0012] It is worth noting that in the task processing architecture, a job is defined as a collection of multiple tasks. There may be dependencies between tasks, for example, the execution of one task depends on the data processing results of another task. Each task must be executed by a specific service component. A task can be understood as a processing request for a data subset, and a service component is a functional unit module that implements the processing request. If there is a dependency between tasks, there is also a calling relationship between the corresponding service components. For example, if you want to process a certain data subset, you must call the corresponding service component, and the component will perform the relevant data processing operations. This operation process constitutes a task, and multiple "service components" need to work together to complete the processing of a "job". The operational resource requirements of service components to perform tasks encompass a wide range of server resource elements, such as the processing power of the Central Processing Unit (CPU), the computing power of the Graphics Processing Unit (GPU), random access memory (RAM), and cache. Service components on different servers with a call relationship also require communication resources. The stable and efficient operation of service components is highly dependent on sufficient and appropriate computing and communication resources. Furthermore, due to the varying nature of the tasks performed by different service components, their requirements for computing and communication resources vary significantly. For example, some service components focused on complex computing tasks have substantial and continuous demands for CPU and GPU resources during task execution. Service components primarily responsible for data transmission, on the other hand, have high requirements for the bandwidth and transmission stability of communication link resources (such as wired links).

[0013] Specifically, for steps S1 to S4, after obtaining the power-related data to be processed, in order to perform distributed processing on the data, the data to be processed is divided to obtain multiple data subsets. One data subset corresponds to a task, and each task needs to call the corresponding service component to execute. There may be dependencies between the tasks, so that there is a calling relationship between the service components. Therefore, after completing the data division, the tasks corresponding to each data subset, the service components used to execute the tasks, and the task dependencies are determined, and then the calling relationship between the components is determined, and the operating resource requirements of the service components and the communication resource requirements of the calling relationship between the components are obtained. After determining the above information, based on the available computing resources of each server in the cloud computing center and the available communication resources between servers, and ensuring that the operational resource requirements of the service components assigned to the servers are less than or equal to the available computing resources of the servers, and that the communication resource requirements of the inter-component call relationships are less than or equal to the available communication resources between the servers where the components are located, with the goal of maximizing data processing efficiency, tasks, service components, and data subsets are allocated to the servers in the cloud computing center. For example, Task A, the service components used to execute Task A, and the data subset corresponding to Task A are allocated to one server (the available computing resources of this server must meet the operational resource requirements of the service components used to execute Task A), and Task B, the service components used to execute Task B, and the data subset corresponding to Task B are allocated to another server (the available computing resources of this server must meet the operational resource requirements of the service components used to execute Task B). If Task B depends on Task A, the available communication resources between the two servers must meet the communication resource requirements of the corresponding inter-component call relationships. Optionally, the data to be processed can be the raw data obtained through collection, or it can be data obtained after cleaning and / or attribute reduction of the raw data.

[0014] It is worth noting that maximizing data processing benefits can be maximizing data processing efficiency, maximizing benefits, or maximizing the comprehensive benefits of data processing efficiency and benefits. Data processing benefits can be set according to actual needs to achieve load balancing, optimal resource utilization, or minimum cost.

[0015] Compared with the prior art, the embodiment of the present invention divides the data to be processed into multiple data subsets, determines the tasks and service components corresponding to the data subsets, determines the dependencies between the tasks, the operating resource requirements of the service components, determines the calling relationships between the service components, and the communication resource requirements of the calling relationships between components; on the premise of ensuring that the operating resource requirements of the service components on the server are less than the available computing resources of the server, and that the communication resource requirements of the calling relationships between components are less than or equal to the available communication resources between the servers where the components are located, with the goal of maximizing data processing benefits, tasks, service components and data subsets are allocated to each server, thereby achieving reasonable data distribution. After completing the data distribution, the cloud computing center performs data processing operations, thereby improving data processing benefits including data processing efficiency.

[0016] As a second preferred implementation, on the basis of steps S1 to S4, when a calling relationship exists between the first service component and the second service component and the first service component and the second service component are not on the same server, the Dijkstra algorithm is used to find the shortest path that meets the multi-hop allocation requirement; wherein, the first service component and the second service component are any two of all the service components, and the multi-hop allocation requirement means that the available communication resources between all adjacent servers in the path are greater than or equal to the communication resource requirements between the first service component and the second service component, and the shortest path that meets the multi-hop allocation requirement is used to provide a calling path for the call between the first service component and the second service component.

[0017] Specifically, the communication resource requirements resulting from the call relationship between service components must not exceed the available communication resources between the servers where the corresponding service components are deployed. This requirement can be further interpreted as follows: When a call relationship between components indicates that a first service component requires a call from a second service component, the following two situations exist: 1. If the first server hosting the first service component and the second server hosting the second service component are capable of direct communication, the available communication resources from the second server to the first server must be greater than or equal to the communication resource requirements resulting from the call relationship between the first and second service components. 2. If direct communication between the first and second servers is not possible, a shortest path from the second server to the first server must be searched using methods such as the Dijkstra algorithm. Furthermore, the available communication resources of each path segment within this shortest path must be greater than or equal to the communication resource requirements resulting from the call relationship between the first and second service components to ensure smooth inter-service call processing and reliable communication. The Dijkstra algorithm is the Dijkstra algorithm. Optionally, for searching the shortest path, an algorithm improved based on the Dijkstra algorithm or having a similar concept to the Dijkstra algorithm may be used.

[0018] As a third preferred implementation, based on steps S1 to S4, the step S1 includes obtaining power-related data to be processed and dividing the data to be processed to obtain several data subsets, including: Acquire raw data related to electricity, perform sampling analysis on the raw data, and identify data quality issues in the raw data; Select a data cleaning algorithm corresponding to the mined data quality issues, perform parallel cleaning operations on the original data, and obtain data to be processed; Slicing and discretizing the data to be processed into equal widths to obtain a number of data sets, and performing attribute simplification on the data sets; The data set after attribute reduction is divided into several data subsets.

[0019] Specifically, the quality of raw data is susceptible to interference from a variety of factors, primarily including acquisition errors, channel status, and parameter settings (such as sampling frequency). Raw data may also contain data redundancy, so preprocessing is necessary. First, sample and analyze the raw data to identify the primary or all data quality issues. Then, use appropriate programs to repair different types of data quality issues in parallel, generating the data to be processed. Finally, the data to be processed is sharded and discretized to obtain multiple data sets, each of which is then subjected to attribute reduction.

[0020] Optionally, the data quality problem includes at least one of a data mutation outlier problem, a data missing problem, and a data zero value problem. The data cleaning algorithm corresponding to the data mutation outlier problem is a data correction algorithm based on horizontal and vertical similarity, and the data cleaning algorithm corresponding to the data missing problem and the data zero value problem is a weighted moving average repair algorithm. Exemplarily, first, a sampling analysis is performed on the original data to explore the data quality problems existing in the data to be processed. When it is identified that the data to be processed has a data mutation outlier problem, the original data is corrected based on the data correction algorithm based on horizontal and vertical similarity; when it is identified that the data to be processed has a data missing problem and a data zero value problem, the original data is repaired for zero value data and missing data based on the weighted moving average repair algorithm.

[0021] Specifically, a data correction algorithm based on horizontal and vertical similarity is designed for sudden outlier data. Horizontal similarity refers to the similarity of changes in raw data within adjacent days; vertical similarity refers to the fact that the data to be processed at adjacent moments generally do not mutate. The basic idea of ​​this algorithm is to use the mean to penalize sudden changes in the raw data. The specific correction formula is: ; in, is the corrected data. The data of mutation occurred; is the average value of the original data in the time period of the mutation moment; is the average value of the original data in the adjacent days at the time of mutation; and is the weight factor; is the penalty factor, and its value range is between 0 and 1. It is worth noting that according to the vertical similarity, , according to the horizontal similarity, the correction formula is set .

[0022] A weighted moving average repair algorithm is designed for zero-value data and missing data. A set of adjacent actual data values ​​is used to repair missing data. The specific repair formula is: ; ; in, It's time After repair, the data is the moving period; For the The original data of the period, It is The closer the data to be repaired is, the closer the monitoring values ​​of the two are, so the greater the weight is.

[0023] In this embodiment, by analyzing the data quality problems existing in the original data and selecting the corresponding cleaning means, there is no need to use all data cleaning means to clean the original data, which shortens the data cleaning time and improves the data cleaning efficiency.

[0024] As a fourth preferred embodiment, based on any of the above embodiments, the data to be processed is sliced ​​and discretized with equal width to obtain a number of data sets, and then attribute reduction is performed on the data sets, dividing the attribute-reduced data sets into multiple data subsets. It is worth noting that, in general, discrete attribute values ​​are closer to the representation of knowledge level than continuous attribute values, and discretization can improve the efficiency of attribute reduction algorithms. Therefore, the data to be processed is discretized, for example, the equal width discretization method can be selected based on the characteristics of the data to be processed to discretize the data to be processed.

[0025] Preferably, when sharding the data to be processed, ensure that the data types in the same data set are the same. Furthermore, attribute reduction can be performed on each data set in parallel on different computer nodes to improve the efficiency of attribute reduction.

[0026] Specifically, the data set is attribute-reduced in the following manner: Taking the data set as a domain, determining a condition attribute set and a decision attribute set based on the data set; Dividing the data set into a plurality of equivalence classes according to the conditional attribute set; When there are inconsistent objects in the equivalence class, the probability of the equivalence class to which the inconsistent objects belong on each decision attribute value in the decision attribute set is calculated using a preset decision attribute distribution function; wherein the inconsistent objects refer to data in the equivalence class with inconsistent decision attribute values; When the maximum probability is greater than a set probability threshold, the inconsistent object is classified into the decision attribute value corresponding to the maximum probability; When the maximum probability is less than the set probability threshold, the inconsistent objects are comprehensively assigned to each of the decision attribute values ​​according to the probabilities, or the decision attribute values ​​are adjusted according to the probabilities to update the decision attribute set; Dividing the domain based on the condition attribute set and the decision attribute set to obtain a first positive region corresponding to each decision attribute value in the decision attribute set; Constructing a reduced set; calculating the importance of each condition attribute in the condition attribute set to the first positive region; performing multiple rounds of adjustments on the reduced set; In each round of adjustment of the reduced set, the conditional attribute with the greatest importance is selected from all conditional attributes to be analyzed and added to the reduced set; wherein the conditional attribute to be analyzed is a conditional attribute not in the reduced set; Dividing the domain according to the reduced set and the decision attribute set to obtain second positive regions corresponding to the respective decision attribute values; When the first positive region and the second positive region are consistent, the adjustment of the reduced set is ended; when the first positive region and the second positive region are inconsistent, the next round of adjustment of the reduced set is entered.

[0027] In this embodiment, in order to address the redundancy problem of the data to be processed, an efficient decision table conversion algorithm is designed to handle inconsistent objects. A heuristic attribute importance calculation formula is used to reduce the search space, and the correlation between the conditional attributes and the decision attribute types is introduced when calculating the importance, which further improves the calculation efficiency and can quickly find the conditional attribute combination that is most valuable for the decision (i.e., the final reduced set).

[0028] Specifically, attribute reduction refers to removing redundant information from the data to be processed and retaining information that is useful for decision-making, thereby improving the efficiency of data analysis. The following is a brief introduction to various terms related to attribute reduction and the reduction methods.

[0029] 1. Overview of Overall Attribute Simplification The following will introduce the relevant content of implementing overall attribute simplification by treating the data to be processed as the data to be processed.

[0030] 1. Terminology The decision table for data to be processed is , It is a non-empty finite set of data objects that need to be processed; is a non-empty finite set of attributes, where is the set of conditional attributes of the data to be processed, is the set of decision attributes, , is the empty set, is an attribute The value range of is an information function that converts the object Map to A value in the object for Any data in .

[0031] For decision tables that require data processing ,definition Conditional attribute set A subset of The equivalence relation can be defined as: ; Will Divide into several equivalence classes, and we get: ; in, Representation object In the subset The equivalence class on For the convenience of representation, Simplified to , It is a non-empty finite set of data objects that need to be processed.

[0032] According to the decision attribute set Domain Divide and get , It is a different subset divided; define the conditional attribute set About decision attribute sets The positive region and boundary region of are: ; ; in, Represents a conditional attribute set About decision attribute sets The positive area, Represents a conditional attribute set About decision attribute sets The boundary domain, yes Any subset of for The lower approximation of Representation object In the conditional attribute set The equivalence class on Indicates that all of , take the lower approximation to merge.

[0033] 2. Basic simplification methods for attribute reduction In attribute reduction, the positive region reduction method is used. The principle of the positive region reduction method is as follows: Assume Is a conditional attribute set A subset of , then the conditional attribute is necessary, otherwise it indicates for is unnecessary. If the following two conditions are met, it means Is the decision table for processing data A simplification of : (1) ; (2) ; in, Representation subset About decision attribute sets The positive area, Indicates that there is no conditional attribute Subset of About decision attribute sets positive area.

[0034] 2. Overview of Parallel Attribute Reduction It is understandable that the attribute reduction process involves a large number of equivalence class calculations, so it is considered whether the equivalence classes can be calculated in parallel.

[0035] For decision tables that require data processing , if you will Split into , For multiple data sets obtained by segmentation, , , for Data shards (i.e. data collections), Shard the data The domain of , is a non-empty finite set of attributes, is a set of conditional attributes, is the set of decision attributes, express and The union of .

[0036] Assumptions ,in, ,Right now Based on The equivalence class of the partition, is greater than or equal to 1 and less than or equal to An integer. right Perform equivalence class division: ; ; in, Is a union expression, which means arrive All to carry out a merger; The above content shows that the equivalence classes calculated on multiple data shards and finally merged are the same as the equivalence classes calculated directly on the entire data. Therefore, the calculation of equivalence classes can be parallelized. That is, large-scale data to be processed can be divided into several data sets and distributed to different computer nodes to calculate equivalence classes in parallel, thereby improving the efficiency of attribute reduction.

[0037] 3. Overview of Decision Table Consistency Conversion It is worth noting that if the conditional attribute set If the decision attribute values ​​of all objects in the divided equivalence class are the same, the data decision table needs to be processed is consistent, in which case and Otherwise, the data decision table needs to be processed Inconsistent.

[0038] In practice, due to factors such as data collection errors, power outages, unstable energy supply, and server failures, the data to be processed may be inconsistent to a certain extent, potentially interfering with decision-making accuracy. Common attribute reduction algorithms are generally unable to directly handle such inconsistent decision tables. Therefore, further transformation of the decision table is necessary.

[0039] In the initial decision table of the data to be processed, if If inconsistent objects are generated on In the conditional attribute set The inconsistent object produced above can be defined as: ; ; in, Representation object and objects The value of the conditional attribute is the same as Representation object and objects The decision attribute values ​​are inconsistent.

[0040] By defining and analyzing inconsistent objects, we can find that there are two main types of distributions: (1) Due to data collection errors or equipment failures, some decision attribute values ​​deviate from the normal range, resulting in data inconsistency. The distribution of these inconsistent objects is usually irregular, and their proportion is very small.

[0041] (2) Due to incomplete data collection, some key conditional attributes are missing, resulting in different decision attribute values ​​appearing under the same conditional attributes. The distribution of these inconsistent objects is usually regular, and they account for a large proportion.

[0042] In order to make the decision table of the data to be processed consistent, for any ,definition The decision attribute distribution function of the equivalence class is: ; in, , , express exist The probability of for Belong to the equivalence class, It is a non-empty finite set of data objects that need to be processed; According to the distribution type of inconsistent objects and their decision attribute distribution function, the inconsistent decision table of data to be processed is converted into a consistent decision table of data to be processed. Its formal description is shown in the following formula: ; Among them, the function Used to find the maximum value; Indicates the set probability threshold, which is generally set to 0.7. It is worth noting that the threshold can also be other values, such as 0.73, 0.75 or 0.78, and is generally between 0.5 and 1; Indicates the union of decision attribute values ​​according to probability, and then updates the decision table of the data to be processed .

[0043] 4. Adjustment of Reduced Sets In order to reduce the search space and the time complexity of attribute reduction, this embodiment adopts the following method to determine the reduced set: Set up a decision table for data that needs to be processed after consistency processing , , ;for , whose domain is , the conditional attribute set is , yes The decision attribute set is , the information function is ; yes and The union of yes Any of these equivalence classes, Relative to Positive area The calculation formula is as follows: ; in, Represents equivalence class All data in have the same decision attribute value. about The positive region of is composed of the union of equivalence classes with the same decision attribute value. That is, if the decision attribute value of the equivalence class is not unique, the positive region does not contain the equivalence class. Indicates satisfaction with All The union of .

[0044] make is a subset of the conditional attributes, and ,property Conditional attribute set The importance of can be defined as: ; for Relative to The dependency of is defined as: ; make , and The correlation between them is defined as: ; in, represents the entropy function; express and The conditional entropy of .

[0045] Combined with the above content, set the initial reduction set to an empty set and calculate The importance of all remaining attributes in the set is calculated, and then the attribute with the greatest importance is selected and added to the reduced set. If more than one attribute has the greatest importance, the correlation between the condition attribute and the decision attribute value is calculated based on the inconsistent objects, and the condition attribute with the greatest correlation is selected.

[0046] It is worth noting that attribute reduction of the data to be processed does not necessarily require the execution of a decision table consistency conversion operation.

[0047] As a fifth preferred implementation, based on the fourth implementation, the domain is divided according to the reduced set and the decision attribute set to obtain second positive regions corresponding to the respective decision attribute values, including: Subtract the second positive area obtained from the previous round of adjustment from the domain to obtain the remaining data; Dividing the remaining data according to the reduced set and the decision attribute set to obtain candidate positive regions corresponding to each decision attribute value; The candidate positive regions and the second positive regions with the same decision attribute value are merged to obtain the second positive regions corresponding to the respective decision attribute values ​​in this round of adjustment.

[0048] Specifically, in order to simplify the attribute reduction algorithm, the second positive region is calculated in the following way: Set the domain corresponding to the data to be processed to , Is a conditional attribute set A subset of , satisfying the following formula: ; In actual calculation, for the domain , the second positive area obtained in the previous round of adjustment It has been determined, so in this round of adjustment, we only need to calculate the Identify candidate positive regions , and then the second positive area obtained by the previous round of adjustment and candidate positive regions You can get the second positive area obtained in this round of adjustment , express and Compared with traditional calculation methods, this method can greatly reduce the time complexity and space complexity of calculating the second positive region.

[0049] As a sixth preferred implementation manner, based on the third implementation manner, acquiring raw data related to electricity, performing sampling analysis on the raw data, and discovering data quality issues existing in the raw data include: Obtaining raw data related to electricity, calculating a standard deviation of the raw data, a Z statistic under a preset confidence interval, and a marginal error, and calculating a sampling size based on the standard deviation, the Z statistic, and the marginal error; When the sampling size is larger than a set percentage of the overall size of the original data, multiplying the product of the sampling size and the overall size by the sum of the sampling size and the overall size to obtain a new sampling size; A sampling analysis is performed on the original data based on the sampling scale to identify data quality issues existing in the original data.

[0050] Specifically, the sample size = (Z statistic squared + standard deviation squared) / marginal error squared, where the specific value of the Z statistic is determined by the confidence interval. For example, the Z statistic for a 95% confidence interval is 1.96, and the Z statistic for a 90% confidence interval is 1.64. In addition, if the sample size is larger than a set percentage of the overall size (such as 5%), the sample size should be adjusted by dividing the product of the sample size and the overall size by the sum of the sample size and the overall size to obtain the new sample size.

[0051] As a seventh preferred implementation, based on any of the above implementations, the step S4, under resource constraints and with the goal of maximizing data processing efficiency, allocating the task, the service component for executing the task, and the data subset associated with the task to each server in the cloud computing center, includes: Construct a comprehensive model: Represent the servers of the cloud computing center, the available computing resources of the servers, and the available communication resources between different servers in the form of vectors; represent the operating resource requirements of the service components and the communication resource requirements of the call relationships between the components in the form of vectors, use the service components as vertices of a directed graph, and establish edges of the directed graph based on the call relationships between the components; construct an objective function with the goal of maximizing data processing efficiency; and establish resource constraints; Inputting the data subset, the task corresponding to the data subset, the service component for executing the task, the calling relationship between the components, the running resource requirements of the service component and the communication resource requirements of the calling relationship between the components into the comprehensive model, and outputting a data allocation plan; Based on the data distribution plan, the task, the service component for executing the task, and the data subset associated with the task are distributed to the servers in the cloud computing center.

[0052] Specifically, the following introduces the server and resource constraints involved in the comprehensive model: For a cloud computing center consisting of several servers, all servers together constitute a set And the total number of servers is Each server can provide Computing resources, such as CPU processing power, GPU computing power, memory and cache, etc. The available computational resources are defined as a vector as follows: ; in It is a server Provided The available computing resources provided by all servers together form the available computing resource matrix of the cloud computing center: ; Defining Servers arrive The wired link between , the available communication resources it provides can be represented as a vector ,in, The total number of resource types that a wired link can provide.

[0053] In the server architecture, application services with inter-component calling relationships can be represented by a directed graph. Represents that all service components constitute a vertex set , and the number of service components is , all the calling relationships between components together constitute a directed edge set , and the calling relationship has For service components , the required operating resource requirements can be expressed as the following vector: ; Among them, the service component need server resources (i.e. computing resources), It is for The server resource requirements of all service components of the application service (i.e., the running resource requirements) together constitute the resource requirement matrix: ; And the edge The corresponding service component arrive The call relationship, the wired link resources required by the call relationship (i.e., communication resource requirements) are expressed as the following vector: ; in, For the required The communication resource requirements of all inter-component call relationships constitute a set Combining the above content, the corresponding server directed graph is represented as: ; For all the servers in the cloud computing center that are assigned data, a server sequence is formed according to the order in which the data arrives. It is understandable that the server needs to run continuously for a period of time when processing data, and will continue to occupy the corresponding server and wired link resources during this period of time. After the server runs, the occupied resources will be released. In the server allocation process, it is necessary to consider the relationship between the various resource requirements of all service components and the calling relationships between components and the various available resources of the current cloud computing center. When the server The available resources provided satisfy the service components The service component can be allocated to the corresponding server only when the demand is met.

[0054] For example, if the server communicates via a wired link, arrive Assigned to servers arrive When the corresponding components are called Requires continuous server usage arrive Therefore, as shown in the following formula, the available communication resources provided by the wired link need to be greater than the communication resource requirements of the calling relationship: ; in, Representation Server arrive The first wired link Available communication resources.

[0055] If and only if all resource requirements of all service components and the calling relationships between components are met at the same time, the server is considered to be allocated successfully. At this time, after the server is successfully allocated, the cloud computing center status needs to be updated. For example, the service component arrive Has a call relationship, service component arrive Assigned to servers arrive Then the update is as follows: ; ; ; in, It is a server Updated available computing resources, It is a server Available computing resources before the update, Is a service component operational resource requirements; It is a server Updated available computing resources, It is a server Available computing resources before the update, Is a service component operational resource requirements; Is the updated server arrive The available communication resources of the wired link, Is the server before the update arrive The available communication resources of the wired link, It's about service components and The communication resource requirements of the calling relationship.

[0056] In addition, the following two situations need to be considered: Local allocation of service components: Service components arrive Since components are allocated on the same server, the inter-component call relationship can be directly implemented using the server's shared memory without occupying wired link resources. In this case, the communication resource requirements for inter-component call relationships can be ignored.

[0057] Multi-hop distribution of service components: Service components arrive Assigned to two servers that are not directly connected, it is necessary to use an algorithm similar to the Dijkstra algorithm to find the shortest path to meet the communication resource requirements of the calling relationship between components. It can be understood that the shortest path includes at least two servers, and the available communication resources between adjacent servers need to meet the requirements of the service components. arrive The shortest path not only meets the communication resource requirements of the calling relationship between components, but also saves resources, reduces resource waste, and thus reduces costs.

[0058] Specifically, the following introduces the content related to the objective function: Assume that the data to be processed is divided into the first type of data set and the second data set These two data sets are then divided into two groups to obtain Corresponding multiple data subsets and Corresponding multiple data subsets; assuming that the data processing efficiency is the benefit. Let Indicates the resource usage fee paid by the user to the service provider. Indicates the cost per unit time, represents the unit resource cost, represents the time span of task execution, that is, the longest execution completion time among all tasks, then the profit function It can be expressed as the following formula: ; in, is the first type of data set, For the allocation A collection of servers, for Assigned to The data type on the server is The number of executable data, For data types In the Estimated execution energy consumption on each server; For the second type of data set, For the allocation A collection of servers, for Assigned to The data type on the server is The amount of data actionable, For data types In the The estimated execution energy consumption on each server is calculated, and the objective function for maximizing the benefit is a nonlinear integer programming as shown below: ; st ; ; ; ; ; in, represents the set of positive numbers, yes Medium data type The total number of yes Medium data type The total number of is with The estimated execution time associated with is with The associated estimated execution time.

[0059] In an eighth preferred implementation, based on the seventh implementation, solving the objective function under the resource constraint condition to obtain the data allocation solution includes: Converting the objective function from a nonlinear integer program to a linear program; Solving the linear programming to obtain a fractional optimal solution under the constraints; and determining a relaxed optimal solution of the nonlinear integer programming based on the fractional optimal solution; constructing a bipartite graph about the data subset and the server based on the relaxed optimal solution; Find the N-matching BM of the bipartite graph based on resource constraints; According to the N matching BM, a rounding algorithm is performed to obtain an integer solution as the final data allocation solution.

[0060] Specifically, since solving nonlinear integer programming is an NP-hard problem, we replace the variables involved in the nonlinear integer programming with parameters and find the optimal solution of its relaxed form (linear programming). Then, we use the bipartite graphs B-matching rounding (BGBR) algorithm to obtain an approximate integer optimal solution to the objective function formula in the seventh embodiment. The variables are replaced as follows: ; Accordingly, the profit function can be expressed as: ; Then the nonlinear integer programming is transformed into a linear programming: ; st ; ; ; ; ; ; This linear program can be solved in polynomial time using the simplex method to obtain a fractional optimal solution. Obviously, the optimal solution of nonlinear integer programming is Corresponding is a feasible solution to the above equation, so The objective function value is not less than the objective function value of the optimal solution .

[0061] Score optimal solution The objective function value of satisfy: ; Among them, OPT represents the optimal solution, for the optimal solution of the score , the corresponding relaxed optimal solution of nonlinear integer programming can be obtained by the following formula .

[0062] ; At this point, a common rounding algorithm can be used to obtain a feasible solution, that is, ,make , the unassigned tasks are scheduled using the greedy method.

[0063] However, this rounding method will not only increase the time span, but also increase resource usage, thereby reducing the unit time benefit. Therefore, a feasible solution to the original problem is constructed by solving an N matching of the bipartite graph, that is, relaxing the optimal solution. Convert to a feasible integer solution .

[0064] Assume that the data to be processed is divided into the first type of data set and the second data set These two data sets are then divided according to data type to obtain Corresponding multiple data subsets and The following are the specific steps to construct a bipartite graph. For example, is a set of weights, for a given solution ,make ,and .make Represents the first type of data set, represents a server set for processing a first type of data set, wherein, nodes For servers that process the first type of data set .

[0065] Bipartite Graph The edge sets in correspond to data types, and , its edge set The construction method is as follows: for each , according to the estimated execution time To sort in non-increasing order, without loss of generality, assume ,in, Is a positive value, if ,Right now ,mean Contains only one corresponding to Vertex , in this case, for every Data type ,make Belongs to the edge set , express Otherwise, find the one that satisfies Minimum subscript , for every satisfaction Data type ,make belong For each , find satisfaction The minimum subscript , for every satisfaction Data type ,make belong .right ,like , belong ; Finally, for every satisfaction Data type ,make belong .

[0066] Bipartite Graph The edge weights in The construction method is as follows: for each edge , whose weight is defined as Indicates setting parameters. Every point , whose capacity is defined as .

[0067] It can be seen that for At any point in , we have: ; in, An integer.

[0068] Next, for the constructed bipartite graph , find an N matching BM, where N represents a matching subset. , BM represents the same as the above for processing the first type of data set A collection of servers , and the first type of data set , where each edge in BM is Just related times. For each edge , assign a data type of Tasks to the server , and obtain a new integer solution , that is, , then: ; Similarly, construct a bipartite graph according to the above method , and obtain an integer solution according to the bipartite graph N matching rounding algorithm , that is, , then: ; In summary, construct integer solutions The idea of ​​the N matching rounding algorithm is as follows: Step 1: According to the relaxed optimal solution Construct bipartite graphs separately and Step 2: Find the bipartite graph and N matches BM.

[0069] Step 3: Match BM according to N, perform rounding algorithm, and get integer solution Compared with the prior art, in an embodiment of the present invention, first, power-related data to be processed is obtained, and the data to be processed is divided to obtain several data subsets; then, the task corresponding to the data subset, the service component for executing the task, and the task dependency are determined; then, the inter-component calling relationship is determined based on the task, the service component for executing the task and the task dependency, and the operating resource requirements of the service component and the communication resource requirements of the inter-component calling relationship are obtained; finally, under resource constraints, with the goal of maximizing data processing efficiency, the task, the service component for executing the task and the data subset associated with the task are allocated to each server of the cloud computing center; wherein, the resource constraint conditions include: the operating resource requirements of the service component on the server are less than or equal to the available computing resources of the server, and the communication resource requirements of the inter-component calling relationship are less than or equal to the available communication resources between the servers where the components are located. It can be seen from this that the embodiment of the present invention divides the data to be processed into multiple data subsets, and determines the tasks and service components corresponding to the data subsets, determines the dependencies between the tasks, the operating resource requirements of the service components, determines the calling relationships between the service components, and the communication resource requirements of the calling relationships between components; on the premise of ensuring that the operating resource requirements of the service components on the server are less than the available computing resources of the server, and that the communication resource requirements of the calling relationships between components are less than or equal to the available communication resources between the servers where the components are located, with the goal of maximizing data processing efficiency, tasks, service components and data subsets are allocated to each server, thereby achieving reasonable data distribution and helping to improve subsequent data processing efficiency.

[0070] See also Figure 2 , Figure 2 FIG. 2 is a schematic diagram of a data distribution system according to an embodiment of the present invention. The data distribution system 20 includes: The data partitioning module 21 is used to obtain power-related data to be processed and partition the data to be processed to obtain a number of data subsets; a relationship determination module 22 for determining tasks corresponding to the data subset, service components for executing the tasks, and task dependencies; An information acquisition module 23 is configured to determine an inter-component call relationship based on the task, the service component for executing the task, and the task dependency relationship, and to acquire an operating resource requirement of the service component and a communication resource requirement of the inter-component call relationship; The data allocation module 24 is used to allocate the tasks, service components used to execute the tasks, and data subsets associated with the tasks to various servers in the cloud computing center under resource constraints and with the goal of maximizing data processing efficiency; wherein the resource constraints include: the operating resource requirements of the service components on the server are less than or equal to the available computing resources of the server, and the communication resource requirements of the calling relationship between the components are less than or equal to the available communication resources between the servers where the components are located.

[0071] In a preferred embodiment, the data allocation module is also used to: when a calling relationship exists between the first service component and the second service component and the first service component and the second service component are not on the same server, use the Dijkstra algorithm to find the shortest path that meets the multi-hop allocation requirements; wherein the first service component and the second service component are any two of all the service components, and the multi-hop allocation requirement means that the available communication resources between all adjacent servers in the path are greater than or equal to the communication resource requirements between the first service component and the second service component, and the shortest path that meets the multi-hop allocation requirements is used to provide a calling path for the call between the first service component and the second service component.

[0072] It is worth noting that the specific working process of the data distribution system can refer to the working process of the data distribution method described in the above embodiment, and will not be repeated here.

[0073] Compared with the prior art, the data distribution system disclosed in the embodiment of the present invention divides the data to be processed into multiple data subsets, determines the tasks and service components corresponding to the data subsets, determines the dependencies between the tasks, the operating resource requirements of the service components, determines the calling relationships between the service components, and the communication resource requirements of the calling relationships between components; on the premise of ensuring that the operating resource requirements of the service components on the server are less than the available computing resources of the server, and that the communication resource requirements of the calling relationships between components are less than or equal to the available communication resources between the servers where the components are located, with the goal of maximizing data processing efficiency, tasks, service components and data subsets are distributed to each server, thereby achieving reasonable data distribution and helping to improve subsequent data processing efficiency.

[0074] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A data distribution method, characterized in that: include: Acquire power-related data to be processed, and divide the data to be processed into several data subsets; Determining tasks corresponding to the data subset, service components for executing the tasks, and task dependencies; Determine an inter-component call relationship based on the task, the service component for executing the task, and the task dependency, and obtain an operating resource requirement of the service component and a communication resource requirement of the inter-component call relationship; Under resource constraints, with the goal of maximizing data processing efficiency, the task, the service component used to execute the task, and the data subset associated with the task are allocated to each server in the cloud computing center; wherein the resource constraints include: the operating resource requirements of the service component on the server are less than or equal to the available computing resources of the server, and the communication resource requirements of the calling relationship between the components are less than or equal to the available communication resources between the servers where the components are located.

2. The data distribution method according to claim 1, wherein: When a calling relationship exists between a first service component and a second service component and the first service component and the second service component are not on the same server, the Dijkstra algorithm is used to find the shortest path that meets the multi-hop allocation requirement; wherein, the first service component and the second service component are any two of all the service components, and the multi-hop allocation requirement means that the available communication resources between all adjacent servers in the path are greater than or equal to the communication resource requirements between the first service component and the second service component, and the shortest path that meets the multi-hop allocation requirement is used to provide a calling path for the call between the first service component and the second service component.

3. The data distribution method according to claim 1, wherein: The step of obtaining power-related data to be processed and dividing the data to be processed to obtain several data subsets includes: Acquire raw data related to electricity, perform sampling analysis on the raw data, and identify data quality issues in the raw data; Select a data cleaning algorithm corresponding to the mined data quality issues, perform parallel cleaning operations on the original data, and obtain data to be processed; Slicing and discretizing the data to be processed to obtain a number of data sets, and performing attribute simplification on the data sets; The data set after attribute reduction is divided into several data subsets.

4. The data distribution method according to claim 3, wherein: The data set is attribute-reduced in the following way: Taking the data set as a domain, determining a condition attribute set and a decision attribute set based on the data set; Dividing the domain based on the condition attribute set and the decision attribute set to obtain a first positive region corresponding to each decision attribute value in the decision attribute set; Construct a reduced set; Calculating the importance of each condition attribute in the condition attribute set to the first positive region; performing multiple rounds of adjustments on the reduced set; In each round of adjustment of the reduced set, the conditional attribute with the greatest importance is selected from all conditional attributes to be analyzed and added to the reduced set; wherein the conditional attribute to be analyzed is a conditional attribute not in the reduced set; Dividing the domain according to the reduced set and the decision attribute set to obtain second positive regions corresponding to the respective decision attribute values; When the first positive region and the second positive region are consistent, the adjustment of the reduced set is ended; when the first positive region and the second positive region are inconsistent, the next round of adjustment of the reduced set is entered.

5. The data distribution method according to claim 4, wherein: After determining a condition attribute set and a decision attribute set based on the data set using the data set as the domain, and before dividing the domain based on the condition attribute set and the decision attribute set to obtain a first positive region corresponding to each decision attribute value in the decision attribute set, the method further includes: Dividing the data set into a plurality of equivalence classes according to the conditional attribute set; When there are inconsistent objects in the equivalence class, the probability of the equivalence class to which the inconsistent objects belong on each decision attribute value in the decision attribute set is calculated using a preset decision attribute distribution function; wherein the inconsistent objects refer to data in the equivalence class with inconsistent decision attribute values; When the maximum probability is greater than a set probability threshold, the inconsistent object is classified into the decision attribute value corresponding to the maximum probability; When the maximum probability is less than the set probability threshold, the inconsistent objects are comprehensively allocated to each of the decision attribute values ​​according to the probabilities, or the decision attribute values ​​are adjusted according to the probabilities to update the decision attribute set.

6. The data distribution method according to claim 4, wherein: The dividing the domain according to the reduced set and the decision attribute set to obtain second positive regions corresponding to the respective decision attribute values ​​includes: Subtract the second positive area obtained from the previous round of adjustment from the domain to obtain the remaining data; Dividing the remaining data according to the reduced set and the decision attribute set to obtain candidate positive regions corresponding to each decision attribute value; The candidate positive regions and the second positive regions with the same decision attribute value are merged to obtain the second positive regions corresponding to the respective decision attribute values ​​in this round of adjustment.

7. The data distribution method according to claim 3, wherein: The acquiring of raw power-related data, and performing sampling analysis on the raw data to identify data quality issues in the raw data include: Obtaining raw data related to electricity, calculating a standard deviation of the raw data, a Z statistic under a preset confidence interval, and a marginal error, and calculating a sampling size based on the standard deviation, the Z statistic, and the marginal error; When the sampling size is larger than a set percentage of the overall size of the original data, multiplying the product of the sampling size and the overall size by the sum of the sampling size and the overall size to obtain a new sampling size; A sampling analysis is performed on the original data based on the sampling scale to identify data quality issues existing in the original data.

8. The data distribution method according to claim 1, wherein: The method of allocating the tasks, service components for executing the tasks, and data subsets associated with the tasks to various servers in a cloud computing center under resource constraints and with the goal of maximizing data processing benefits includes: Construct a comprehensive model: Represent the servers of the cloud computing center, the available computing resources of the servers, and the available communication resources between different servers in the form of vectors; represent the operating resource requirements of the service components and the communication resource requirements of the call relationships between the components in the form of vectors, use the service components as vertices of a directed graph, and establish edges of the directed graph based on the call relationships between the components; construct an objective function with the goal of maximizing data processing efficiency; and establish resource constraints; Inputting the data subset, the task corresponding to the data subset, the service component for executing the task, the calling relationship between the components, the operation resource requirements of the service component, and the communication resource requirements of the calling relationship between the components into the comprehensive model; Under the resource constraint conditions, solving the objective function to obtain a data allocation solution; Based on the data distribution plan, the task, the service component for executing the task, and the data subset associated with the task are distributed to the servers in the cloud computing center.

9. The data distribution method according to claim 8, wherein: Solving the objective function under the resource constraint conditions to obtain a data allocation solution includes: Converting the objective function from a nonlinear integer program to a linear program; Solving the linear programming under the constraints to obtain a fractional optimal solution; and determining a relaxed optimal solution of the nonlinear integer programming based on the fractional optimal solution; constructing a bipartite graph about the data subset and the server based on the relaxed optimal solution; Based on resource constraints, find the N-matching BM of the bipartite graph; According to the N matching BM, a rounding algorithm is performed to obtain an integer solution as the final data allocation solution.

10. A data distribution system, characterized in that: include: A data partitioning module is used to obtain power-related data to be processed and partition the data to be processed to obtain several data subsets; a relationship determination module, configured to determine a task corresponding to the data subset, a service component for executing the task, and a task dependency relationship; An information acquisition module is used to determine the calling relationship between components based on the task, the service component for executing the task and the task dependency, and obtain the running resource requirements of the service component and the communication resource requirements of the calling relationship between components; The data allocation module is used to allocate the tasks, service components used to execute the tasks, and data subsets associated with the tasks to various servers in the cloud computing center under resource constraints and with the goal of maximizing data processing efficiency; wherein the resource constraints include: the operating resource requirements of the service components on the servers are less than or equal to the available computing resources of the servers, and the communication resource requirements of the calling relationships between the components are less than or equal to the available communication resources between the servers where the components are located.

Images