A power data sharing device and method for high-concurrency data publishing requests

Abstract

The present invention belongs to the field of power information processing, and provides a power data sharing device and method for high-concurrency data publishing requests. The device includes an application module, a service module and an interface module; the application module is used to obtain different business application requests for data to be published by the power consumption information collection system from the peripheral system, decompose the required data into each minimum data object, extract and transform the data accordingly, and store them in advance in the interface database of the interface module; the service module includes a computing cluster, which is used to configure data according to the data publishing request, organize the minimum data objects in the interface database into corresponding data packets, pass them to the service cluster, and publish the data based on the size of the data packet; the application module is also used to estimate the download time interval of each data packet available to the client, obtain the Gaussian confidence determined by the optimal publishing strategy, and then publish the download time interval of each data packet available to the client and store it in the interface database.

Description

Technical Field

[0001] The present invention belongs to the field of power information processing, and in particular relates to a power data sharing device and method for high-concurrency data publishing requests. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] As the basic data platform of the marketing domain of the full-service data center, the electricity consumption information collection system, on the one hand, continuously shortens the data collection cycle, supports the access of tens of millions of devices, and realizes data integration with the marketing domain of the full-service data center; on the other hand, it provides unified business interaction and data services to the outside world, such as issuing fee control instructions and uploading meter codes.

[0004] However, the inventors found that the current method of sharing power business data is limited by the goals and carrying capacity at the beginning of the design. The timeliness of business applications based on data sharing services cannot be met, which slows down the overall data collection and data landing time, and cannot cope with the current situation of growing business computing needs of various peripheral systems and increasing amount of collected data sharing. Summary of the invention

[0005] In order to solve the technical problems existing in the above-mentioned background technology, the present invention provides a power data sharing device and method for high-concurrency data publishing requests, which can improve the interactivity and stability of data sharing.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A first aspect of the present invention provides a power data sharing device for high-concurrency data publishing requests, which includes an application module, a service module and an interface module;

[0008] The application module is used to obtain different business application requests for the data to be released by the power consumption information collection system from the peripheral system, decompose the required data into various minimum data objects, extract and transform the data accordingly, and store them in the interface database of the interface module in advance;

[0009] The service module includes a computing cluster, which is used to configure data according to the data publishing request, organize the minimum data objects in the interface database into corresponding data packets according to different requests, and transmit them to the service cluster, which then publishes the data based on the size of the data packet and determines the data publishing time;

[0010] The application module is also used to: estimate the download time interval of each data packet available to the client based on the data release time, obtain the Gaussian confidence determined by the optimal release strategy, and then release the download time interval of each data packet available to the client and store it in the interface database.

[0011] As an implementation method, the application module includes a demand management module, a configuration management module and a monitoring management module;

[0012] The demand management module is used to decompose the data required by the application according to the application request;

[0013] The configuration management module is used to provide data object configuration services based on data decomposition;

[0014] The monitoring and management module is used to perform online monitoring and control on the shared service information.

[0015] As an implementation method, the computing cluster includes a static data publishing computing cluster and a real-time data publishing computing cluster. The static data publishing computing cluster is used to publish historical offline data, and the real-time data publishing computing cluster is used to obtain real-time data and push its subscribed data to the external system in real time.

[0016] As an implementation method, the service cluster includes an interface service cluster and a file service cluster. The interface service cluster is responsible for providing a unified interface service to the outside in a distributed load balancing manner, and the file service cluster provides a big data file download service to the outside.

[0017] As an implementation method, when the service cluster publishes data, if the amount of data published is less than the set publishing quantity threshold, the data is directly returned through the interface service cluster; otherwise, the data packet is placed on the file service node through the file service cluster, and then the download time interval for each data packet available to the client is estimated based on the client download record data.

[0018] As an implementation mode, the file service cluster estimates the download time interval of each data packet available for the client based on the client download record data and the Gaussian regression prediction model.

[0019] As an implementation method, the application module is also used to: judge the Gaussian confidence determined by the optimal publishing strategy, and determine whether the Gaussian confidence mean difference is less than a set threshold. If so, directly publish the estimated download time interval for each data packet available to the client; otherwise, adjust the download time interval for each data packet available to the client.

[0020] As an implementation mode, the application module is also used to use a Markov decision process to predict the environmental state of the next time period of the interface database before obtaining the Gaussian confidence determined by the optimal release strategy, so as to map the environmental state to the download time interval of the client.

[0021] A second aspect of the present invention provides a power data sharing method for high-concurrency data publishing requests, comprising:

[0022] The application module obtains different business application requests for the data to be released by the power consumption information collection system from the peripheral system, decomposes the required data into each minimum data object, extracts and transforms the data accordingly, and stores them in the interface database of the interface module in advance;

[0023] The computing cluster performs data configuration according to the data publishing request, organizes the minimum data objects in the interface database into corresponding data packets according to different requests, and transmits them to the service cluster, which then publishes the data based on the size of the data packet and determines the data publishing time;

[0024] The application module estimates the download time interval of each data packet available to the client based on the data release time, obtains the Gaussian confidence determined by the optimal release strategy, and then releases the download time interval of each data packet available to the client and stores it in the interface database.

[0025] As an implementation manner, the power data sharing method for high-concurrency data publishing requests further includes:

[0026] The Gaussian confidence determined by the optimal publishing strategy is judged to determine whether the Gaussian confidence mean difference is less than the set threshold. If so, the estimated download time interval of each data packet available to the client is directly published; otherwise, the download time interval of each data packet available to the client is adjusted.

[0027] As an implementation mode, before obtaining the Gaussian confidence determined by the optimal release strategy, a Markov decision process is used to predict the environment state of the interface database in the next time period, so as to map the environment state to the download time interval of the client.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] (1) The present invention provides a method and device for sharing power data for high-concurrency data publishing requests. On the one hand, data sharing layers are used to accelerate the timeliness of data from collection to application, thereby improving data sharing efficiency. On the other hand, real-time control and management of the overall resources of the cluster are enhanced, and based on service performance and resource consumption prediction, the interactivity and stability of data sharing are improved. This lays a data foundation for building a new power system with new energy as the main supply and a strong smart grid as the hub platform, and realizing the dynamic characteristics of "safe and controllable, flexible and efficient, intelligent and friendly, and open and interactive".

[0030] (2) In order to solve the problem of low timeliness of data sharing in the electricity consumption information collection system, the present invention constructs a data sharing module, provides data caching services through an interface database, reduces resource consumption caused by frequent access of peripheral system applications to the original production database, and adopts the idea of ​​data sharing hierarchy. On the one hand, data is classified and released based on priority to avoid non-core business affecting the processing timeliness of core business; on the other hand, through the feedback of monitoring information, the overall resources of the data sharing module are controlled and managed in real time, thus avoiding idle resources and improving the utilization rate of cluster sorting resources.

[0031] (3) The present invention is aimed at non-real-time, large-data-volume data packet download services. According to the client download records, the Gaussian regression prediction model is used to predict the download time zones of various types of data packets. On the one hand, the differences and tendencies of the client for the download time zone are evaluated based on the multi-dimensional download characteristics, thereby ensuring that the download time zone provided to the peripheral system client meets its download effective time zone requirements; on the other hand, Gaussian regression prediction has the advantages of strong generalization, easy implementation, and probabilistic output. When the network bandwidth resource environment changes or the interface database input and output are limited, the download time zone can be adjusted to avoid occupying core business resources.

[0032] (4) In order to achieve real-time control and management of the overall resources of the data sharing module, the present invention adopts a Markov decision process to predict the environmental status of the next time period (including the storage status of the interface database and the network resource status), and makes feedback adjustments based on the prediction results, so that when resources are idle or tight, strategy adjustments can be made in advance to avoid idle resources and accelerate the efficiency of shared services.

[0033] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0035] Figure 1 A schematic diagram of the structure of a power data sharing device for high-concurrency data publishing requests provided by an embodiment of the present invention;

[0036] Figure 2 A flow chart of a power data sharing method for high-concurrency data publishing requests provided by an embodiment of the present invention;

[0037] Figure 3 A time interval prediction flow chart based on a Gaussian regression prediction model provided in an embodiment of the present invention;

[0038] Figure 4 A flow chart of an environmental state prediction model based on a Markov decision process provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0040] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0041] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0042] Embodiment 1

[0043] Reference Figure 1 and Figure 2 , this embodiment provides a structural diagram of a power data sharing device for high-concurrency data publishing requests, which specifically includes an application module, a service module and an interface module.

[0044] In the specific implementation process, the application module is used to: obtain different business application requests for data to be released by the electricity consumption information collection system from the peripheral system, decompose the required data into various minimum data objects, extract and transform the data accordingly, and store them in the interface database of the interface module in advance.

[0045] exist Figure 1 In, the application module includes a demand management module, a configuration management module and a monitoring management module;

[0046] The demand management module is used to decompose the data required by the application according to the application request; for example, to standardize the process of application, approval, and execution of the sharing request for the data required for the data sharing of the power consumption information collection system;

[0047] The configuration management module is used to provide data object configuration services based on data decomposition; for example, to standardize and control shared service access rights, data sharing module operating parameters, and configuration data publishing models;

[0048] The monitoring and management module is used to perform online monitoring and control of shared service information; for example, online monitoring and control of information such as the running status, call status, interactive traffic, shared logs, and database storage resources of shared services.

[0049] Among them, the demand management module obtains different business application requests for the data to be released by the electricity consumption information collection system from the peripheral system, and decomposes the required data into various minimum data objects according to the application requests.

[0050] Specifically, the physical architecture of data sharing in the electricity consumption information collection system relies on the virtualization resources of Alibaba Cloud, and is divided into computing node container groups, service node container groups, and file storage service groups according to their functions. Through the firewall devices and network devices uniformly deployed by the electricity consumption information collection system, data sharing services are provided to peripheral systems such as marketing service systems. Based on the production library of the electricity consumption information collection system, more than 100,000 business application requests and corresponding request data download records from January 2021 to September 2021 were obtained as example samples, as shown in Table 1, which are examples of some business requests:

[0051] Table 1 Examples of some business requests and the data involved in collection

[0052]

[0053] Decompose 52 types of business application requests to obtain each minimum data object. The data object configuration service of the configuration management module extracts and transforms the required data from the production database of the electricity consumption information collection system, and stores it in the interface database of the interface module in advance to treat the unified storage and management of the published data.

[0054] During the specific implementation process, the service module includes a computing cluster, which is used to configure data according to data publishing requests, organize the minimum data objects in the interface database into corresponding data packets according to different requests, and pass them to the service cluster, which publishes data based on the data packet size and determines the data publishing time.

[0055] Among them, the computing cluster includes a static data publishing computing cluster and a real-time data publishing computing cluster. The static data publishing computing cluster is used to publish historical offline data (including publishing time calculation, publishing data object collection, etc.), and the real-time data publishing computing cluster is used to obtain real-time data and push its subscribed data to the external system in real time.

[0056] For example: The real-time data publishing computing cluster obtains real-time data through the power consumption information collection system front-end machine's dual-receive and independent application capture technologies, and pushes the subscribed data to the external system in real time through the client.

[0057] The service cluster includes an interface service cluster and a file service cluster. The interface service cluster is responsible for providing a unified interface service to the outside in a distributed load balancing manner, and the file service cluster provides a big data file download service to the outside.

[0058] Specifically, the service module obtains different data publishing requests and assigns them to the computing cluster. The computing cluster configures data according to the data scope, data frequency, data content combination and other dimensions of the data publishing request, organizes the minimum data units in the interface database into the required data packets according to different requests, and transmits them to the service cluster.

[0059] In some embodiments, when the service cluster publishes data, if the amount of data published is less than the set publishing quantity threshold (for example: 1TB), the data is directly returned through the interface service cluster; otherwise, the data packet is placed on the file service node through the file service cluster, and then the download time interval of each data packet available to the client is estimated based on the client download record data.

[0060] In some embodiments, the file service cluster estimates the download time interval of each data packet available to the client based on the client download record data and the Gaussian regression prediction model. For example: if the amount of data released is greater than or equal to 1TB, the data packet is placed on the file service node through the file service cluster, and the client download record data is obtained based on the historical records in the log file of the monitoring management module, and the Gaussian regression prediction model is used to estimate the download time interval of each data packet available to the client.

[0061] Specifically, Figure 3 As shown, the Gaussian regression prediction model is used to calculate the earliest download time and the latest download time of the data packet, which is specifically divided into the following steps:

[0062] Step 1.1. Obtain client download record data based on the historical records in the log file, including d (e.g., d=6) dimensional feature vectors, which are respectively the size of the data packet, the weight value of the data packet, the historical download duration of the data packet, the historical download time of the data packet, the historical download times of the data packet, and the network transmission rate of the download process, forming a historical download record sample set and sample feature data;

[0063] Step 1.2. Use the K-means clustering method to cluster the historical download record samples. Based on the download time and the amount of downloaded data, select a number of samples (e.g., 26) at random and relatively uniformly as the initial cluster center points, calculate the sum of the feature distances between the samples, and use the greedy strategy to divide the current cluster in turn and iteratively update the mean vector of each cluster until the clustering result remains unchanged after the iterative update. Output the clustering result of the historical download record sample set, and divide each cluster into a training sample set and a test sample set according to the proportion;

[0064] Step 1.3. According to the clustering results, the data of the Gaussian regression model is defined based on the clustered sample set;

[0065] The training sample set of a certain cluster is defined as

[0066]

[0067] Where X = [x1, x2, …, x d ] is the d=5-dimensional feature vector of the historical download record sample, then the input feature vector of the i-th sample is X i =[x i1 ,x i2 ,…,x id ],Y=[y1,y2,…,y r ] is the actual output value of the sample, and its corresponding output set that satisfies Gaussian process regression is {f(x1),f(x2),…,f(x r )}, the relationship between the two is as follows:

[0068]

[0069] Among them, ε is an independent Gaussian white noise, which also conforms to the Gaussian distribution, with a mean of 0 and a variance of σ 2 , denoted as ε~N(0,σ 2 );

[0070] Step 1.4. Construct a kernel function based on the historical sample feature data of the clustered sample set, and calculate and obtain the kernel matrix of the clustered sample set;

[0071] The linear combination of two independent Gaussian distributions still satisfies the Gaussian distribution, so here Y satisfies the Gaussian distribution, that is,

[0072] Y~GP(m(X),C(X,X)) (3)

[0073] Where m(X) = E(f(X)), the mean is calculated based on the historical sample data of the clustered sample set, and C(X,X) = K(X,X) + σ 2 I, I is the r×r identity matrix, C(X,X) is the r×r covariance matrix,

[0074] K(X,X) is the r×r kernel function matrix:

[0075]

[0076] Among them, the element K ij = k(x i ,x j ), k(x i ,x j )=E[(f(x i )-m(x i ))(f(x j )-m(x j ))] are calculated based on the historical sample data of the clustered sample set.

[0077] The kernel function is a square exponential covariance, that is,

[0078]

[0079] Among them, θ0 is a hyperparameter, which represents the average distance between the mean function of each sample and the mean function of the clustered sample set. θ1 represents the length of the function's "swing", and the process of determining this parameter is as follows:

[0080] Set the objective function to logp(Y|X)=logN(μ,K y ), and find the optimal value by gradient descent method, that is:

[0081]

[0082] When the gradient descent value is less than the set threshold, it is determined that under the condition of Y~GP(m(X),C(X,X)), p(Y|X) is the largest, and the parameter θ1 is fixed at this time;

[0083] Step 1.5. Use the kernel matrix as the covariance matrix of the joint Gaussian distribution, and jointly calculate the sample data to be tested and the historical sample data of the clustered sample set to obtain the joint prior distribution matrix of the sample to be tested and the historical sample set;

[0084] The test data {X * ,y *} is classified into the Gaussian process with the same distribution as the training data {X,Y}, and the joint Gaussian density distribution is obtained as follows:

[0085]

[0086] Among them, K is the covariance matrix;

[0087] Step 1.6. According to the probability distribution of the clustered historical sample set and the posterior probability of the sample to be tested, the Gaussian regression prediction of the sample set to be tested is obtained using the Bayesian formula;

[0088] Calculate the output vector y according to the Bayesian formula * The conditional probability

[0089] y * |y~N(μ * +K * K -1 (y-μ),K ** -K * K -1 K *T ) (8)

[0090] y * The predicted value is

[0091]

[0092] y * The variance of the prediction uncertainty is

[0093] v(y * )=Var[y * |X,y,X * ]=K(X * ,X * )-K(X * ,X)(K(X , X)+σ 2 I) -1 K(X , X * ) (10)

[0094] By training the mapping relationship f(·) between the input dataset X and the output dataset Y: d →R, so as to achieve the test input data x * When , we can predict the mean and variance of the sample data to satisfy the Gaussian distribution, and output f(x * );

[0095] Step 1.7. Set the Gaussian regression model confidence φ, and determine the time confidence interval of the data packet download based on the Gaussian regression prediction of the sample to be tested, that is, determine the earliest download time and the latest download time of the data packet.

[0096] In a specific implementation, the application module is also used to: estimate the download time interval of each data packet available to the client based on the data release time, obtain the Gaussian confidence determined by the optimal release strategy, and then publish the download time interval of each data packet available to the client and store it in the interface database.

[0097] Specifically, the application module is further used to use a Markov decision process to predict the environment state of the next time period of the interface database before obtaining the Gaussian confidence determined by the optimal release strategy, so as to map the environment state to the download time interval of the client.

[0098] For example: Based on the monitoring and management module, the request data volume, request priority, resource storage status, network resource status and service cluster estimated download time interval of each data packet available to the client for more than 100,000 historical download records from January 2021 to September 2021 are obtained, and the Markov decision process is used to predict the environmental status of the interface database in the next time period to obtain the Gaussian confidence determined by the optimal release strategy.

[0099] Among them, Figure 4 As shown in Figure 2, the specific process of obtaining the Gaussian confidence determined by the optimal release strategy is:

[0100] Step 2.1. Define the state decision quaternary M = (S, A, P sa ,R,γ);

[0101] Where S represents the state set, S=(s1,s2,…s i ,…),s i Indicates the resource status of the interface database at step i, expressed as a percentage; A represents a set of actions, A = (a1, a2, ... a i ,…), a i represents the action of step i, representing the three states of data storage, download and deletion in the interface database; P represents the state transition probability, P sa Indicates that in the current state s∈S, after taking action a∈A, the data environment state changes from state s i Transition to next state s i+1 The probability that in state s i Next, perform action a i , transfer to the next state s i+1 The probability is expressed as p(s i+1 |s i ,a i ), is the division result based on time interval, the earliest download time represents the storage of data, the time interval is for download, and the latest download time represents data deletion, and the state transition probability is 1; R represents the reward function, which is If a group (s i ,a i ) moves to the next state s i+1 , then the reward function is recorded as r(s i+1 |s i ,a i ), the r value is determined by the interface database setting the remaining capacity of 40% and the client's satisfaction with the download processing result ∈[0,100%], where the optimal remaining capacity is calculated using the Euclidean distance, and the customer satisfaction is the Euclidean distance index; γ∈[0,1] is the conversion factor;

[0102] Step 2.2. Initialize the value function V(s) = 0, determine the download time by setting the confidence φ, and then map the actions based on time into loading, downloadable, and deleting (i.e., behavior strategy), recorded as strategy π:S→A, to form strategy π(s);

[0103] Define the state value function V π (s) represents the value function of state s under strategy π:

[0104]

[0105] Among them, r i represents the immediate reward of the future i-th step, s′ represents the next state, V π (s) represents the long-term impact of adopting strategy π starting from state s0, V π (s′) is the long-term impact of adopting strategy π in the next state, r(s′|s,a) is the immediate reward from taking action a in state s to the next state s′;

[0106] Given a strategy π and an initial state s, the initial action a is determined by the strategy π and the state s, that is, action a = π(s). At the next moment, the next state s' will be transferred with probability p(s'|s,a). The above formula is expressed based on the transition probability as:

[0107]

[0108] In the current state s and the current action a, if we follow the strategy π in the future, we will move to the next state s′ with probability p(s′|s,a), and define the action value function Q π (s,a) is as follows:

[0109]

[0110] Based on V π (s) and Q π The Bellman equation for the above expression of (s,a) is as follows:

[0111]

[0112]

[0113] Thus, in dynamic programming, the relationship between the value function of the current state and the value function of the next state is shown.

[0114] Then the optimization objective π* is expressed as:

[0115]

[0116] Let the optimal strategy π be * The corresponding state value function and behavior value function are V * (s) and Q * (s,a), the relationship between the two is as follows:

[0117]

[0118] The state value function and behavior value function satisfy the following Bellman optimality equations respectively:

[0119]

[0120]

[0121] Step 2.3. Calculate the state value function V based on any policy π π (s), that is, to perform strategy estimation and calculate V(s) until convergence;

[0122] Based on formula (14), under a certain strategy π, there are three possible actions a corresponding to π(s), each of which is recorded as π(a|s), and the state value function is extended as follows:

[0123]

[0124] First, set all V π The initial value of (s) is set to 0, and then the value function of all states s is updated based on formula (21):

[0125]

[0126] In the formula, k represents the number of iterations. Only one array is used to store each state value function. Whenever a new value is obtained, the old value is overwritten, as shown in [V k+1 (s1),V k+1 (s2),V k (s3),…], indicating that states s1 and s2 are updated, but state s3 is not updated, so that the new value can be used in time during the iteration process to achieve faster convergence;

[0127] Set a constant threshold θ to judge each state s, |Vk (s)-V k+1 (s)|<θ, if so, then determine V k+1 (s) is the state value function V of the current strategy π π (s);

[0128] Step 2.4. Obtain the state value function V of strategy π based on the strategy estimation in step 2.3 π (s), in order to find a better strategy, the action a taken in state s is changed by changing the confidence φ, and the strategy is improved based on the greedy idea maxq(s,a);

[0129] Traverse all states and all possible actions a, and adopt a greedy strategy to obtain a new strategy π', that is, change the confidence φ, thereby changing the action a taken in state s, generating a new strategy π', and calculating the behavior value function Q based on formula (15) π′ (s,a);

[0130] Judge Q π′ Is (s,a) greater than V? π (s), if Q π′ (s,a)>V π (s), the new strategy π′ (taking action a only in state s and following the old strategy π in other states) has a higher function value than the old strategy (following strategy π in all states), so the strategy is updated, that is, the following update strategy is adopted:

[0131]

[0132] Obtain a new strategy by adopting a greedy strategy with a value function;

[0133] Step 2.5. If there is a new strategy π′, return strategy π′ to step E3; if there is no new strategy π′, output the current strategy π as the optimal strategy π * ;

[0134] Step 2.6. Optimal strategy π * The corresponding action transfer map generated based on the environment state is the client download time interval, and the Gaussian confidence φ mapped in step 1.7 is output based on the highest proportion symmetric interval.

[0135] In some embodiments, the application module is also used to: judge the Gaussian confidence determined by the optimal publishing strategy, and determine whether the Gaussian confidence mean difference is less than a set threshold ε. If so, directly publish the estimated download time interval for each data packet available to the client; otherwise, adjust the download time interval for each data packet available to the client.

[0136] Based on the estimated download time intervals of each data packet available to the client, 50 and 500 data sharing request data were simulated and tested based on the method of this embodiment on October 15 and 18, 2021, respectively, to inform each client of the earliest download time and the latest download time of the data packet downloaded based on the method of this embodiment. The experimental comparison results are as follows:

[0137]

[0138]

[0139] As can be seen from the above table, the fixed client download time of the traditional process is difficult to take into account the real-time evolution of environmental resources and the preferences of each client for downloading data. Therefore, when the daily shared business request volume increases, on the one hand, there is a risk of saturation of database storage resources and network transmission resources, and on the other hand, some data sharing services are not completed, including poor timeliness of core business and incomplete download of data by the client. The data sharing process described in this patent can allocate download time based on the client's download time preference, and estimate the resource occupancy of the next time period based on the environmental status, so it can ensure that the data sharing service adopts an optimized execution strategy, and can still provide high-quality sharing services when the amount of shared data increases suddenly;

[0140] Finally, the peripheral system takes away the downloaded data through the client within a certain time interval, and the interface database clears the expired data.

[0141] Embodiment 2

[0142] like Figure 2 As shown, this embodiment also provides a power data sharing method for high-concurrency data publishing requests, which includes:

[0143] The application module obtains different business application requests for the data to be released by the power consumption information collection system from the peripheral system, decomposes the required data into each minimum data object, extracts and transforms the data accordingly, and stores them in the interface database of the interface module in advance;

[0144] The computing cluster performs data configuration according to the data publishing request, organizes the minimum data objects in the interface database into corresponding data packets according to different requests, and transmits them to the service cluster, which then publishes the data based on the size of the data packet and determines the data publishing time;

[0145] The application module estimates the download time interval of each data packet available to the client based on the data release time, obtains the Gaussian confidence determined by the optimal release strategy, and then releases the download time interval of each data packet available to the client and stores it in the interface database.

[0146] The power data sharing method for high-concurrency data publishing requests further includes:

[0147] The Gaussian confidence determined by the optimal publishing strategy is judged to determine whether the Gaussian confidence mean difference is less than the set threshold. If so, the estimated download time interval of each data packet available to the client is directly published; otherwise, the download time interval of each data packet available to the client is adjusted.

[0148] In the specific implementation process, the Gaussian regression prediction model is used to estimate the download time interval of each data packet available for the client. The specific steps are as shown in Example 1. Figure 3 As shown, no further description is given here.

[0149] Before obtaining the Gaussian confidence determined by the optimal release strategy, a Markov decision process is used to predict the environmental state of the interface database in the next time period to map the environmental state to the client's download time interval.

[0150] Specifically, the monitoring management module can obtain the requested data volume, request priority, resource storage status, network resource status and the estimated download time interval of each data packet available for the client based on the historical download record, and use the Markov decision process to predict the environmental status of the interface database in the next time period to obtain the Gaussian confidence determined by the optimal release strategy. The specific process of obtaining the Gaussian confidence determined by the optimal release strategy is as shown in Example 1. Figure 4 As shown, no further description is given here.

[0151] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A power data sharing device for high concurrent data publishing requests, characterized in that: Includes application module, service module and interface module; The application module is used to obtain different business application requests for the data to be released by the power consumption information collection system from the peripheral system, decompose the required data into various minimum data objects, extract and transform the data accordingly, and store them in the interface database of the interface module in advance; The service module includes a computing cluster, which is used to configure data according to the data publishing request, organize the minimum data objects in the interface database into corresponding data packets according to different requests, and transmit them to the service cluster, which then publishes the data based on the size of the data packet and determines the data publishing time; The application module is also used to: estimate the download time interval of each data packet available to the client based on the data release time, obtain the Gaussian confidence determined by the optimal release strategy, and then release the download time interval of each data packet available to the client and store it in the interface database; The computing cluster includes a static data publishing computing cluster and a real-time data publishing computing cluster. The static data publishing computing cluster is used to publish historical offline data, and the real-time data publishing computing cluster is used to obtain real-time data and push its subscribed data to the external system in real time; The service cluster includes an interface service cluster and a file service cluster. The interface service cluster is responsible for providing a unified interface service to the outside in a distributed load balancing manner, and the file service cluster provides a large data file download service to the outside. The application module is also used to: determine the Gaussian confidence determined by the optimal publishing strategy, determine whether the Gaussian confidence mean difference is less than a set threshold, and if so, directly publish the estimated download time interval of each data packet available to the client; otherwise, adjust the download time interval of each data packet available to the client; The application module is also used to use a Markov decision process to predict the environmental state of the next time period of the interface database before obtaining the Gaussian confidence determined by the optimal release strategy, so as to map the environmental state to the download time interval of the client; Among them, the specific process of obtaining the Gaussian confidence determined by the optimal release strategy is: Step a. Define the state decision quad ; Among them, S represents the state set, , Indicates i The resource status of the interface database of the step is expressed as a percentage; A represents a group of actions, , Indicates i The action of the step represents the three states of data storage, download and deletion in the interface database; 𝑃 represents the state transition probability, Indicates that at present In this state, take action After that, the data environment status is changed from status Transfer to next state The probability that in the state Next action , transfer to the next state The probability of , is the division result based on time interval, the earliest download time represents the storage of data, the time interval is for download, and the latest download time represents data deletion, and the state transition probability is 1; R represents the reward function; is the conversion factor; Step b. Initialize the value function , by setting the confidence Determine the download time, and then map the actions to loading, downloadable, and deleting based on time, that is, the behavior strategy, recorded as strategy , forming a strategy ; Step c. Calculate the state value function based on any strategy π , that is, to estimate the strategy and calculate to convergence; Step d: Get the state value function of strategy π based on the strategy estimation in step c , in order to find a better strategy, by changing the confidence To change the action a taken in state s, and based on the greedy idea Make strategic improvements; Step e. If there is a new strategy , then the strategy Return to step c. If there is no new policy , then output the current strategy π as the optimal strategy ; Step f. Optimal strategy The corresponding action transfer map generated based on the environment state is the client download time interval, and the Gaussian confidence of the mapping is output based on the highest proportion of symmetric intervals .

2. The power data sharing device for high concurrent data publishing requests according to claim 1, characterized in that: The application module includes a demand management module, a configuration management module and a monitoring management module; The demand management module is used to decompose the data required by the application according to the application request; The configuration management module is used to provide data object configuration services based on data decomposition; The monitoring and management module is used to perform online monitoring and control on the shared service information.

3. The power data sharing device for high concurrent data publishing requests according to claim 1, characterized in that: When the service cluster publishes data, if the amount of data published is less than the set publishing quantity threshold, the data is directly returned through the interface service cluster; otherwise, the data packet is placed on the file service node through the file service cluster, and then the download time interval of each data packet available to the client is estimated based on the client download record data.

4. The power data sharing device for high concurrent data publishing requests according to claim 3, characterized in that: The file service cluster estimates the download time interval of each data packet available for the client based on the client download record data and the Gaussian regression prediction model.

5. A power data sharing method for high concurrent data publishing requests, characterized in that: include: The application module obtains different business application requests for the data to be released by the power consumption information collection system from the peripheral system, decomposes the required data into each minimum data object, extracts and transforms the data accordingly, and stores them in the interface database of the interface module in advance; The computing cluster performs data configuration according to the data publishing request, organizes the minimum data objects in the interface database into corresponding data packets according to different requests, and transmits them to the service cluster, which then publishes the data based on the size of the data packet and determines the data publishing time; The application module estimates the download time interval of each data packet available to the client based on the data release time, obtains the Gaussian confidence determined by the optimal release strategy, and then releases the download time interval of each data packet available to the client and stores it in the interface database; The computing cluster includes a static data publishing computing cluster and a real-time data publishing computing cluster. The static data publishing computing cluster is used to publish historical offline data, and the real-time data publishing computing cluster is used to obtain real-time data and push the subscribed data to the external system in real time. The service cluster includes an interface service cluster and a file service cluster. The interface service cluster is responsible for providing a unified interface service to the outside in a distributed load balancing manner, and the file service cluster provides a big data file download service to the outside. The power data sharing method for high-concurrency data publishing requests also includes: The Gaussian confidence determined by the optimal release strategy is judged to determine whether the mean difference of the Gaussian confidence is less than the set threshold. If so, the estimated download time interval of each data packet available to the client is directly released; otherwise, the download time interval of each data packet available to the client is adjusted; Before obtaining the Gaussian confidence determined by the optimal release strategy, a Markov decision process is used to predict the environmental state of the interface database in the next time period to map the environmental state to the client's download time interval; Among them, the specific process of obtaining the Gaussian confidence determined by the optimal release strategy is: Step a. Define the state decision quad ; Among them, S represents the state set, , Indicates i The resource status of the interface database of the step is expressed as a percentage; A represents a group of actions, , Indicates i The action of the step represents the three states of data storage, download and deletion in the interface database; 𝑃 represents the state transition probability, Indicates that at present In this state, take action After that, the data environment status is changed from status Transfer to next state The probability that in the state Next action , transfer to the next state The probability of , is the division result based on time interval, the earliest download time represents the storage of data, the time interval is for download, and the latest download time represents data deletion, and the state transition probability is 1; R represents the reward function; is the conversion factor; Step b. Initialize the value function , by setting the confidence Determine the download time, and then map the actions to loading, downloadable, and deleting based on the time, that is, the behavior strategy, recorded as strategy , forming a strategy ; Step c. Calculate the state value function based on any strategy π , that is, to estimate the strategy and calculate to convergence; Step d: Get the state value function of strategy π based on the strategy estimation in step c , in order to find a better strategy, by changing the confidence To change the action a taken in state s, and based on the greedy idea Make strategic improvements; Step e. If there is a new strategy , then the strategy Return to step c. If there is no new policy , then output the current strategy π as the optimal strategy ; Step f. Optimal strategy The corresponding action transfer map generated based on the environment state is the client download time interval, and the Gaussian confidence of the mapping is output based on the highest proportion of symmetric intervals .

Images