A method and system for processing massive data of a power grid master station

Abstract

The application relates to the technical field of data processing, and discloses a power grid master station mass data processing method and system. The method is characterized in that: the power grid master station data is segmented to obtain multiple unit data groups; the multiple unit data groups are subjected to clustering processing to obtain multiple unit data groups under each cluster family; the multiple unit data groups under each cluster family are combined respectively to obtain multiple target data groups; the feature identifier of the corresponding target data group is determined according to the data quantity of the target data group and the unit data group; the feature identifier is taken as the key value of the corresponding target data group and unit data group, and is stored in the memory; and the key value is taken as the unique identifier corresponding to the target data group and the unit data group respectively, so that the interception time and interception frequency of the power grid master station data are greatly reduced, the situation that the previous data is expired before being completely stored and analyzed is avoided, and the processing efficiency of the power grid data is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and system for processing massive amounts of data from a power grid master station. Background Technology

[0002] In the daily operation and maintenance of the power grid, the power grid master station usually needs to send interactive data to the server to realize the storage, processing, or distribution of power grid data. The existing interactive data processing method is relatively simple, which only stores data by extracting data one by one. That is, the extracted data is stored sequentially into memory, and then the next data is extracted and stored sequentially into memory again, until the end of the data is extracted. It is impossible to store the data into memory at once.

[0003] When massive amounts of data are generated, the data retrieval time of the power grid master station will be extended and the number of retrievals will increase. There may be situations where previous data expires before it has been stored and parsed, which greatly affects the processing efficiency of power grid data. Summary of the Invention

[0004] This invention provides a method and system for processing massive amounts of data from a power grid master station, which solves the technical problem that the data retrieval time of the power grid master station will be extended, the number of retrievals will be increased, and the previous data may expire before it has been stored and parsed, which greatly affects the processing efficiency of power grid data.

[0005] In view of this, the first aspect of the present invention provides a method for processing massive amounts of data from a power grid master station, comprising the following steps:

[0006] In response to the data processing request from the power grid master station, the power grid master station data is segmented to obtain multiple unit data groups;

[0007] Clustering is performed on the multiple unit data groups to obtain multiple unit data groups under each cluster family;

[0008] Multiple unit data groups under each cluster are combined to obtain multiple target data groups;

[0009] The feature identifier of the target data group is determined based on the number of data items and the number of data groups in the target data group.

[0010] The feature identifier is used as the key value of the corresponding target data group and the unit data group. The key value, the target data group and the unit data group are stored in memory, and the key value is used as the unique identifier corresponding to the target data group and the unit data group respectively.

[0011] Preferably, the step of segmenting the power grid master station data to obtain multiple unit data groups in response to the power grid master station data processing request specifically includes:

[0012] Respond to data processing requests from the power grid master station and receive data from the power grid master station;

[0013] Obtain common characters from the power grid master station data, and segment the power grid master station data according to the common characters to obtain multiple unit data groups.

[0014] Preferably, the step of combining multiple unit data groups under each cluster to obtain multiple target data groups specifically includes:

[0015] Vectorize the multiple unit data groups under each cluster to obtain multiple unit data vectors, and construct a unit data vector set.

[0016] Randomly select a unit data vector from the set of unit data vectors, calculate the first Euclidean distance between the selected unit data vector and the other unit data vectors, and determine the unit data vector with the smallest first Euclidean distance as the cluster center.

[0017] Calculate the second Euclidean distance between the cluster center and other unit data vectors, compare the second Euclidean distance with a preset Euclidean distance, filter out unit data vectors whose second Euclidean distance is less than the preset Euclidean distance and assign them to the range of the cluster center, and combine the cluster center with the unit data vectors in its range;

[0018] Based on the remaining unit data vectors, the steps of randomly selecting a unit data vector, calculating the first Euclidean distance between the selected unit data vector and the other unit data vectors, and determining the unit data vector with the smallest first Euclidean distance as the cluster center are repeated until all unit data vectors are combined to obtain multiple target data groups.

[0019] Preferably, the step of determining the feature identifier of the corresponding target data group based on the number of data items and the unit data group of the target data group specifically includes:

[0020] Based on the number of data points and the number of data groups in the target data group, the corresponding feature identifier of the target data group is calculated using the following formula:

[0021] Q = S0 × 31 (n-1) +S1×31 (n-2) +...+S n-1

[0022] In the formula, Q is the feature identifier, and S is the feature identifier. i S is the normalized unit data vector corresponding to the i-th unit data group in the target data group.i =S0,S1,+...+S n-1 , where n is the number of data points in the target data set.

[0023] Preferably, the step of using the feature identifier as the key value of the corresponding target data group and the unit data group, storing the key value, the target data group, and the unit data group in memory, and using the key value as a unique identifier corresponding to the target data group and the unit data group respectively, specifically includes:

[0024] The feature identifier is used as the key value of the corresponding target data group and the unit data group, and the key value mapping is defined for the target data group and the unit data group.

[0025] Multiple storage threads are created, and each storage thread stores the key value, the target data group, and the unit data group into memory.

[0026] Preferably, the method further includes:

[0027] In response to a memory-based data request query, the corresponding key value in the data request query is parsed, and the target data group or unit data group corresponding to the data request query is determined based on the mapping relationship between the key value and the target data group and the unit data group, respectively.

[0028] Secondly, the present invention also provides a system for processing massive amounts of data from a power grid master station, comprising:

[0029] The data response module is used to respond to the processing requests of the power grid master station data, and to segment the power grid master station data to obtain multiple unit data groups;

[0030] The data clustering module is used to perform clustering processing on multiple unit data groups to obtain multiple unit data groups under each cluster family;

[0031] The data combination module is used to combine multiple unit data groups under each cluster to obtain multiple target data groups;

[0032] The identifier generation module is used to determine the feature identifier of the target data group based on the number of data items and the unit data group of the target data group.

[0033] The data storage module is used to use the feature identifier as the key value of the corresponding target data group and the unit data group, store the key value, the target data group and the unit data group in memory, and use the key value as a unique identifier corresponding to the target data group and the unit data group respectively.

[0034] Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described above.

[0035] Fourthly, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.

[0036] As can be seen from the above technical solutions, the present invention has the following advantages:

[0037] This invention segments power grid master station data into multiple unit data groups, clusters these unit data groups to obtain multiple unit data groups under each cluster, and combines these multiple unit data groups under each cluster to obtain multiple target data groups. Based on the number of data points in each target data group and the unit data groups, a feature identifier for each target data group is determined. This feature identifier is used as the key value for the corresponding target data group and unit data group, and stored in memory. The key value also serves as a unique identifier for each target data group and unit data group. This significantly reduces the data retrieval time and number of retrievals from the power grid master station, avoids situations where previous data expires before it has been fully stored and parsed, and improves the processing efficiency of power grid data. Attached Figure Description

[0038] Figure 1 A flowchart illustrating a method for processing massive amounts of data from a power grid master station, provided as an embodiment of the present invention;

[0039] Figure 2 This is a schematic diagram of the structure of a power grid master station massive data processing system provided in an embodiment of the present invention. Detailed Implementation

[0040] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0041] For easier understanding, please refer to Figure 1 The present invention provides a method for processing massive amounts of data from a power grid master station, comprising the following steps:

[0042] Step 1: Respond to the data processing request from the power grid master station, segment the power grid master station data, and obtain multiple unit data groups.

[0043] It should be noted that the execution entity in this embodiment is usually a power plant server, which has the ability to receive massive amounts of data. In the scenario of receiving massive amounts of data, the main station data includes multiple data that far exceed the processing capacity of conventional solutions. This data is usually used to realize the storage, processing or distribution of power grid data.

[0044] Among them, the unit data group can be the unit data after minimizing the segmentation.

[0045] Step 2: Perform clustering on multiple unit data groups to obtain multiple unit data groups under each cluster family.

[0046] Clustering can be performed using clustering algorithms such as K-MEANS, K-MEDOIDS, or CLARANS.

[0047] By clustering multiple unit data groups, the resulting unit data groups under each cluster have a high degree of commonality and characteristics.

[0048] Step 3: Combine the multiple unit data groups under each cluster to obtain multiple target data groups.

[0049] By categorizing unit data according to data type, the target data group can exhibit commonalities and characteristics. This also facilitates skipping expired data during subsequent calls and parsing, optimizing the data's functional structure and reducing data processing costs.

[0050] Step 4: Determine the corresponding feature identifiers of the target data group based on the number of data points and the number of data units in the target data group.

[0051] Feature identifiers are typically used to identify target data groups. Under this understanding, feature identifiers need to have a short data length and uniqueness so that feature identifiers can correspond one-to-one with target data groups.

[0052] Step 5: Use the feature identifier as the key value for the corresponding target data group and unit data group, store the key value, target data group and unit data group in memory, and use the key value as the unique identifier for the target data group and unit data group respectively.

[0053] Among them, key-value pairs make subsequent data retrieval and parsing more convenient and orderly. When data expires, it is easier to skip expired data directly, reducing the resources required by the server. It can quickly process massive amounts of data in a short time and store massive amounts of data in memory at once, reducing data processing costs and improving data utilization efficiency.

[0054] It should be noted that this invention segments the power grid master station data to obtain multiple unit data groups, clusters these unit data groups to obtain multiple unit data groups under each cluster, and combines these multiple unit data groups under each cluster to obtain multiple target data groups. Based on the number of data in the target data group and the unit data groups, a feature identifier for the corresponding target data group is determined. The feature identifier is used as the key value for the corresponding target data group and unit data group and stored in memory. The key value is also used as a unique identifier for the target data group and unit data group respectively. This greatly reduces the interception time and number of interceptions of power grid master station data, avoids the situation where the previous data has expired before it has been fully stored and parsed, and improves the processing efficiency of power grid data.

[0055] In one specific embodiment, step one specifically includes:

[0056] 101. Respond to the power grid master station's data processing request and receive data from the power grid master station.

[0057] 102. Obtain common characters from the power grid master station data, and segment the power grid master station data according to the common characters to obtain multiple unit data groups.

[0058] Among them, common characters are the most basic characters in the power grid master station data, which can minimize the segmentation of the power grid master station data.

[0059] Since the main site parses the entire array into a JSON string before distributing the data, and the JSON string of the array is: [data1, data2, ..., datam], taking the character format as a JSON string as an example, after determining that the data format is a JSON string, the array is based on the square brackets before and after, and each data is separated by a comma by default. Therefore, the common character of the string in the main site data is the comma.

[0060] In one specific embodiment, step three specifically includes:

[0061] 301. Vectorize the multiple unit data groups under each cluster to obtain multiple unit data vectors, and construct a unit data vector set.

[0062] 302. Randomly select a unit data vector from the set of unit data vectors, calculate the first Euclidean distance between the selected unit data vector and the other unit data vectors, and determine the unit data vector with the smallest first Euclidean distance as the cluster center.

[0063] 303. Calculate the second Euclidean distance between the cluster center and other unit data vectors, compare the second Euclidean distance with the preset Euclidean distance, filter out the unit data vectors whose second Euclidean distance is less than the preset Euclidean distance and assign them to the range of the cluster center, and combine the cluster center with the unit data vectors in its range.

[0064] In practical applications, it is also necessary to combine the cluster centers with the unit data groups corresponding to the unit data vectors within their respective ranges.

[0065] 304. Based on the remaining unit data vectors, re-execute the step of randomly selecting a unit data vector, calculating the first Euclidean distance between the selected unit data vector and the other unit data vectors, and determining the unit data vector with the smallest first Euclidean distance as the cluster center, until all unit data vectors are combined to obtain multiple target data groups.

[0066] In one specific embodiment, step four specifically includes:

[0067] Based on the number of data points and the number of data groups in the target data set, the corresponding feature identifier of the target data set is calculated using the following formula:

[0068] Q = S0 × 31 (n-1) +S1×31 (n-2) +...+S n-1

[0069] In the formula, Q is the feature identifier, and S is the feature identifier. i S is the normalized unit data vector corresponding to the i-th unit data group in the target data group. i =S0,S1,+...+S n-1 , where n is the number of data points in the target data set.

[0070] It should be noted that the formula for calculating the feature identifier uses the prime number 31 as the multiplier, as multiplying by a prime number makes it easier to produce a unique feature identifier. In Java, however, multiplying by a very large number can lead to memory overflow and data loss. Therefore, using 31 as the multiplier is the optimal choice for application in power grid master stations.

[0071] In one specific embodiment, step five specifically includes:

[0072] 501. Use the feature identifier as the key value of the corresponding target data group and unit data group, and define the target data group and unit data group corresponding to the key value mapping.

[0073] Specifically, a key-value type is defined in the stack memory to map all target data groups and unit data groups, where the key is the feature identifier Q and the value is the target data group and unit data group corresponding to the feature identifier.

[0074] 502. Create multiple storage threads, and use each storage thread to store the key-value pair, target data set, and unit data set into memory.

[0075] It should be noted that, since the target data groups in this embodiment are independent of each other, the storage of the mapping relationship between each key value and the target data group is more flexible. At this time, by creating multiple storage threads, each thread can store one or more target data groups into memory at the same time, which can effectively improve the storage efficiency of massive data.

[0076] In one specific embodiment, the method further includes:

[0077] Respond to memory-based data request query requests, parse the corresponding key values ​​in the data request query requests, and determine the target data group or unit data group corresponding to the data request query requests based on the mapping relationship between the key values ​​and the target data group and the unit data group respectively.

[0078] It should be noted that this embodiment can perform calculations and data transmission operations directly in memory, without storing the data on a local hard drive. When processing operations such as calculations or data transmission of power grid data are required, data requests can be made directly based on memory. The purpose of the data request instruction is determined by parsing the corresponding feature identifier and identifying the requested target data group or unit data group according to the key-value mapping relationship. Then, the corresponding data group's calculations or data transmission operations are executed.

[0079] The above is a detailed description of an embodiment of a method for processing massive data from a power grid master station provided by the present invention. The following is a detailed description of an embodiment of a system for processing massive data from a power grid master station provided by the present invention.

[0080] For easier understanding, please refer to Figure 2 The present invention also provides a system for processing massive amounts of data from a power grid master station, comprising:

[0081] The data response module 100 is used to respond to the processing requests of the power grid master station data, and to segment the power grid master station data to obtain multiple unit data groups;

[0082] The data clustering module 200 is used to perform clustering processing on multiple unit data groups to obtain multiple unit data groups under each cluster family;

[0083] The data combination module 300 is used to combine multiple unit data groups under each cluster to obtain multiple target data groups;

[0084] The identifier generation module 400 is used to determine the corresponding feature identifier of the target data group based on the number of data groups and the unit data group of the target data group.

[0085] The data storage module 500 is used to use the feature identifier as the key value of the corresponding target data group and unit data group, store the key value, target data group and unit data group in memory, and use the key value as the unique identifier corresponding to the target data group and unit data group respectively.

[0086] The present invention also provides an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described above.

[0087] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method.

[0088] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, electronic devices, and computer-readable storage media described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0089] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0090] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0091] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0092] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing mass data of a power grid master station, characterized in that, The method comprises the following steps: In response to a processing request of power grid master station data, the power grid master station data is segmented to obtain a plurality of unit data groups; A clustering processing is performed on the plurality of unit data groups to obtain a plurality of unit data groups under each cluster family, comprising: The plurality of unit data groups under each cluster family are respectively vectorized to obtain a plurality of unit data vectors, and a unit data vector set is constructed; A unit data vector is randomly selected from the unit data vector set, a first Euclidean distance between the selected unit data vector and other unit data vectors is calculated, and a unit data vector with the minimum first Euclidean distance is determined as a cluster center; A second Euclidean distance between the cluster center and other unit data vectors is calculated, the second Euclidean distance is compared with a preset Euclidean distance, and a unit data vector with a second Euclidean distance smaller than the preset Euclidean distance is filtered out and attributed to a range to which the cluster center belongs, and the cluster center and the unit data vectors in the range to which the cluster center belongs are combined; The step of randomly selecting a unit data vector from the remaining unit data vectors, calculating a first Euclidean distance between the selected unit data vector and other unit data vectors, and determining a unit data vector with the minimum first Euclidean distance as a cluster center is repeatedly performed until all unit data vectors are combined to obtain a plurality of target data groups; The plurality of unit data groups under each cluster family are respectively combined to obtain a plurality of target data groups; A feature identifier of a corresponding target data group is determined according to a data quantity of the target data group and a unit data group, comprising: The feature identifier of the corresponding target data group is calculated according to the data quantity of the target data group and the unit data group by the following formula: Q = S0x31 (n-1) + S1x31 (n-2) +... + S n-1 In the formula, Q is a characteristic mark, S i is the normalized unit data vector corresponding to the i th unit data group in the target data group, S i =S0,S1,+... + S n-1 n is the data number of the target data group; The feature identifier is taken as a key value of the corresponding target data group and the unit data group, the key value, the target data group and the unit data group are stored in a memory, and the key value is taken as a unique identifier corresponding to the target data group and the unit data group respectively.

2. The method of claim 1, wherein, The step of segmenting the power grid master station data to obtain a plurality of unit data groups in response to a processing request of the power grid master station data specifically comprises: In response to a processing request of power grid master station data, the power grid master station data is segmented to obtain a plurality of unit data groups; A public character in the power grid master station data is acquired, and the power grid master station data is segmented according to the public character to obtain a plurality of unit data groups.

3. The method of claim 1, wherein, The step of taking the feature identifier as a key value of the corresponding target data group and the unit data group, storing the key value, the target data group and the unit data group in a memory, and taking the key value as a unique identifier corresponding to the target data group and the unit data group respectively specifically comprises: The feature identifier is taken as a key value of the corresponding target data group and the unit data group, and the key value is defined to correspond to the target data group and the unit data group; A plurality of storage threads are created, and the key value, the target data group and the unit data group are stored in the memory through each storage thread.

4. The method of claim 1, wherein, Further comprising: In response to a memory-based data request query request, a corresponding key value in the data request query request is parsed, and a target data group or a unit data group corresponding to the data request query request is determined according to a mapping relationship between the key value and the target data group and the unit data group.

5. A power grid master station mass data processing system, characterized in that, Comprise: A data response module is configured to respond to a processing request of power grid master station data, segment the power grid master station data, and obtain a plurality of unit data groups; A data clustering module is configured to perform clustering processing on the plurality of unit data groups, and obtain a plurality of unit data groups under each cluster family; The clustering processing on the plurality of unit data groups to obtain the plurality of unit data groups under each cluster family comprises: The plurality of unit data groups under each cluster family are respectively vectorized to obtain a plurality of unit data vectors, and a unit data vector set is constructed; A unit data vector is randomly selected from the unit data vector set, a first Euclidean distance between the selected unit data vector and other unit data vectors is calculated, and a unit data vector with the smallest first Euclidean distance is determined as a cluster center; A second Euclidean distance between the cluster center and other unit data vectors is calculated, the second Euclidean distance is compared with a preset Euclidean distance, and a unit data vector with a second Euclidean distance smaller than the preset Euclidean distance is filtered out and belongs to a range in the cluster center, and the cluster center and the unit data vectors in the range thereof are combined; The steps of randomly selecting a unit data vector from the remaining unit data vectors, calculating a first Euclidean distance between the selected unit data vector and other unit data vectors, and determining a unit data vector with the smallest first Euclidean distance as a cluster center are repeatedly performed until all unit data vectors are combined to obtain a plurality of target data groups; A data combination module is configured to combine the plurality of unit data groups under each cluster family to obtain a plurality of target data groups; An identifier generation module is configured to determine a characteristic identifier of a corresponding target data group according to a data quantity of the target data group and a unit data group; The characteristic identifier of the corresponding target data group is calculated according to the data quantity of the target data group and the unit data group by the following formula: A data storage module is configured to store the characteristic identifier as a key value of the corresponding target data group and the unit data group, store the key value, the target data group, and the unit data group in a memory, and store the key value as a unique identifier corresponding to the target data group and the unit data group, respectively. Q = S0x31 (n-1) + S1x31 (n-2) +... + S n-1 In the formula, Q is a characteristic mark, S i is the normalized unit data vector corresponding to the i th unit data group in the target data group, S i =S0,S1,+... + S n-1 n is the data number of the target data group; The memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method of any one of claims 1-4. 6.An electronic device comprising a memory and a processor, the electronic device characterized by, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-4.

7. A computer readable storage medium characterized in that, ​

Images