Network data processing method and device for cloud edge-end fusion

Abstract

The invention discloses a network data processing method and device for cloud side-end fusion, and the method comprises the steps: receiving time series data uploaded by all devices, calling a first function through a processing logic class, carrying out the cleaning of the time series data, obtaining a cleaning result, decomposing the cleaning result, obtaining a plurality of data sets, and carrying out the processing of the data sets. And performing key value mapping operation on the preset number of data sets to obtain key value pairs, concluding the key value pairs by using a second function to obtain a classification result, and storing the time sequence data into the distributed system according to the classification result. And the network data processing efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of power grid data processing technology, and in particular to a cloud-edge-end integrated network data processing method and device. Background Art

[0002] With the explosive growth of minute-level data collection across the entire link, network data collection and processing face unprecedented challenges. Traditional centralized data storage and processing systems can no longer meet the needs of massive information processing, which puts higher demands on network latency performance and computing power.

[0003] At present, centralized data storage and processing systems usually adopt a centralized data storage architecture, storing large amounts of data on a few servers and then processing them in batches or in real time. This architecture often suffers from problems such as slow processing speed, high latency, and data security due to the limitations of processing power and network bandwidth when facing large amounts of data, and cannot meet the processing requirements of full-link minute-level data collection. Summary of the invention

[0004] In order to solve the above technical problems, an embodiment of the present invention provides a cloud-edge-end integrated network data processing method and device, which realizes the standardization and quality level control processing of network data and improves the network data processing efficiency.

[0005] A first aspect of an embodiment of the present invention provides a network data processing method for cloud-edge-device fusion, the method comprising:

[0006] Receive timing data uploaded by each device;

[0007] Using the processing logic class to call the first function to clean the time series data, obtain the cleaning results, decompose the cleaning results to obtain several data sets, perform key-value mapping operations on a preset number of data sets to obtain key-value pairs, and use the second function to summarize the key-value pairs to obtain classification results;

[0008] According to the classification results, the time series data is stored in a distributed system.

[0009] In a possible implementation of the first aspect, the processing logic class is used to call the first function to clean the time series data to obtain a cleaning result, including:

[0010] The first processing class is used to perform basic field processing on the time series data to obtain the fields of the time series data, and the fields of the time series data are saved in a table form to the second processing class to obtain a log class;

[0011] Utilize the first filter to parse the log class according to the filtering condition to obtain the time series data field to be classified;

[0012] According to the time series data field to be classified, the third processing class is used for output to obtain a cleaning result, wherein the rules in the first processing class, the second processing class and the third processing class can be configured as needed.

[0013] In a possible implementation manner of the first aspect, performing a key-value mapping operation on a preset number of data sets to obtain key-value pairs includes:

[0014] Perform attribute normalization operations on each data set to obtain a normalized data set;

[0015] According to the importance of the data set, the weight of each data set is obtained, based on the weight, the weighted Euclidean distance of each data set is obtained, and according to the weighted Euclidean distance, the neighbor metric between each two data sets is calculated;

[0016] A key-value mapping operation is performed according to the normalized data set and the neighbor metric value to obtain a key-value pair, wherein the key-value pair includes a node, a node attribute, and a neighbor metric value.

[0017] In a possible implementation manner of the first aspect, the key-value pairs are summarized by using the second function to obtain a classification result, including:

[0018] The nodes and node attributes in the key-value pairs are sorted using the second function, and the key-value pairs belonging to the same node and node attribute are classified into one category to obtain a classification result.

[0019] In a possible implementation manner of the first aspect, obtaining a classification result further includes:

[0020] Use the trained Spark algorithm to calculate the time series data to obtain the optimal solution, and then obtain the classification result based on the optimal solution. The training process of the Spark algorithm is as follows:

[0021] The time series data samples are divided into a test set and a prediction set. The population size and population speed are set according to the test set. The individual historical optimal solution is set as the population initial position, and the group historical optimal value is set as the population position with the minimum fitness of the initial position.

[0022] Update the population position and population speed, calculate the fitness of each population, update the individual historical optimal solution of each population according to the fitness, update the inertia factor according to the individual historical optimal solution, update the group historical optimal value, and when the inertia factor is less than the preset value, stop updating to obtain the optimal solution.

[0023] A second aspect of an embodiment of the present invention provides a network data processing device for cloud-edge-device fusion, the device comprising:

[0024] The receiving module is used to receive the timing data uploaded by each device;

[0025] A classification module is used to use a processing logic class to call a first function to clean the time series data to obtain a cleaning result, decompose the cleaning result to obtain a number of data sets, perform a key-value mapping operation on a preset number of data sets to obtain key-value pairs, and use a second function to summarize the key-value pairs to obtain a classification result;

[0026] The storage module is used to store the time series data in a distributed system according to the classification results.

[0027] In a possible implementation manner of the second aspect, the classification module includes a first processing unit, a parsing unit, and an output unit.

[0028] The first processing unit is used to perform basic field processing on the time series data using the first processing class to obtain the fields of the time series data, and save the fields of the time series data in a table form to the second processing class to obtain a log class;

[0029] The parsing unit is used to parse the log class according to the filtering condition using the first filter to obtain the time series data field to be classified;

[0030] The output unit is used to output the cleaning result by using the third processing class according to the time series data field to be classified, wherein the rules in the first processing class, the second processing class and the third processing class can be configured as needed.

[0031] In a possible implementation manner of the second aspect, the classification module further includes a normalization unit, a calculation unit, and a mapping unit.

[0032] Among them, the normalization unit is used to perform attribute normalization operations on each data set to obtain a normalized data set;

[0033] The calculation unit is used to obtain the weight of each data set according to the importance of the data set, obtain the weighted Euclidean distance of each data set based on the weight, and calculate the neighbor measurement value between each two data sets according to the weighted Euclidean distance;

[0034] The mapping unit is used to perform a key-value mapping operation according to the normalized data set and the neighbor metric value to obtain a key-value pair, wherein the key-value pair includes a node, a node attribute and a neighbor metric value.

[0035] A third aspect of an embodiment of the present invention provides a computer device, including a processor and a memory, the memory being used to store a computer program, and the computer program, when executed by the processor, implements the above-mentioned cloud-edge-device integrated network data processing method.

[0036] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium storing a computer program, which implements the above-mentioned cloud-edge-device integrated network data processing method when the computer program is executed by a processor.

[0037] The present invention receives the time series data uploaded by each device, uses the processing logic class to call the first function to clean the time series data, obtains the cleaning result, decomposes the cleaning result to obtain several data sets, performs key-value mapping operation on a preset number of data sets to obtain key-value pairs, uses the second function to summarize the key-value pairs to obtain classification results, and stores the time series data in a distributed system according to the classification results. The present invention realizes the standardization and quality level control processing of network data, and improves the network data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 : A processing flow chart of an embodiment of a network data processing method for cloud-edge-end fusion provided by the present invention;

[0039] Figure 2 : A schematic diagram of smart grid time series data of an embodiment of a network data processing method for cloud-edge-end fusion provided by the present invention;

[0040] Figure 3 : A schematic diagram of the core design of the Map function of an embodiment of the network data processing method for cloud-edge-end integration provided by the present invention;

[0041] Figure 4 : A schematic diagram of the Spark ecosystem of an embodiment of the cloud-edge-end integrated network data processing method provided by the present invention;

[0042] Figure 5 : A device block diagram of an embodiment of the cloud-edge-end integrated network data processing method provided by the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Please refer to Figure 1 , which is a flow chart of an embodiment of a network data processing method for cloud-edge-device integration provided by an embodiment of the present invention, including steps S101 to S103, each of which is specifically as follows:

[0045] S101, receiving time series data uploaded by each device.

[0046] In this embodiment, in the smart grid, the smart grid time series big data generally refers to a type of data that is generated by equipment or instruments, collected by sensors, has a specific association with a certain object or equipment, and is associated with events one after another, such as Figure 2 As shown, Figure 2 It contains the time series data uploaded by receiving various devices, where the time series data is stored in the form of data records and can be represented by triples:

[0047] Z={D e , T i , V a}

[0048] Where D e Indicates the measurement point flag that generates this time series data, T i Indicates the timestamp of time series data generation, V a Represents a specific value, which can be in the form of an integer, floating point, string, or byte array.

[0049] In smart grid, an independent time series data is an event. The event set is defined as follows:

[0050]

[0051] Where N>0, D ti Indicates that at time t i The event rate is e t , then the expression is:

[0052]

[0053] In the formula, e t Refers to the number of events that occur per unit time.

[0054] S102. Use the processing logic class to call the first function to clean the time series data to obtain the cleaning results, decompose the cleaning results to obtain several data sets, perform key-value mapping operations on a preset number of data sets to obtain key-value pairs, and use the second function to summarize the key-value pairs to obtain classification results.

[0055] In this embodiment, after obtaining the time series data, the time series data is processed using an all-round, full-process quality control system based on a cloud platform, edge computing, and terminal perception. A high-performance big data processing and analysis engine, such as Hadoop or Spark, is used in the cloud platform to process and analyze a large amount of data in real time. The engine has efficient data processing and computing capabilities, and can perform batch data processing and analysis, as well as real-time data processing and analysis. Specifically, the first function is called by the processing logic class to clean the time series data to obtain the cleaning result, and then the cleaning result is decomposed to obtain several data sets, and a key-value mapping operation is performed on a preset number of data sets to obtain key-value pairs, and the key-value pairs are summarized by the second function to obtain classification results.

[0056] In some embodiments, step S102 includes but is not limited to the following steps:

[0057] The first processing class is used to perform basic field processing on the time series data to obtain the fields of the time series data, and the fields of the time series data are saved in a table form to the second processing class to obtain a log class;

[0058] Utilize the first filter to parse the log class according to the filtering condition to obtain the time series data field to be classified;

[0059] According to the time series data field to be classified, the third processing class is used for output to obtain a cleaning result, wherein the rules in the first processing class, the second processing class and the third processing class can be configured as needed.

[0060] Specifically, it is worth explaining that the first processing class refers to a class that can process log text containing time series data to obtain the basic fields of time series data; the second processing class refers to a class that can save the basic fields of time series data and obtain the log class based on the basic fields of time series data; the first filter refers to a class that filters the log class according to the set filtering rules to obtain the final required fields; the third processing class refers to writing the final required fields into the corresponding files to obtain the final data.

[0061] In the Hadoop-based smart grid time series big data processing method, big data cleaning is an indispensable step. The three functions of Map, Reduce and Partition are carefully designed, and the corresponding dynamic collection of cleaning classes and setting of cleaning rules are realized through XML configuration. In the data cleaning process, the Map function is the most core part of the architecture, which cleans the data from the original state to usable data. A series of patterns such as filters, factories and listeners are used in the Map design, which can make the cleaning system have good scalability. Figure 3 This is a schematic diagram of the core design of the Map function. Figure 3In the Map phase, the LogProcess class is mainly responsible for the preliminary preparation of the basic configuration files and the use of the scheme to call the Handler class and sort out the actual data processing logic. Among them, the Handler class is the actual control class in the processing logic, which must be completed in the initialization phase of the Map phase. In the actual processing process, according to the Handler call Figure 3 The process part in is completed.

[0062] FilterChain is used to perform step-by-step parsing and cleaning in a pipeline-like mode. The main function of FilterChain is to configure the filtering conditions. In the Map design process, all classes use a configurable method and can be replaced and modified at will based on the actual data cleaning requirements. In the log processing process, the original log uses the JournalClean class to implement basic processing, thereby generating some basic fields, and saves them to the Journal in the form of a table to form a Journal log class. This class is parsed in the filter chain according to the logic under specific requirements to generate the final required fields, and the JournalWriter log is written to the corresponding file to output the final data.

[0063] In some embodiments, step S102 includes but is not limited to the following steps:

[0064] Perform attribute normalization operations on each data set to obtain a normalized data set;

[0065] According to the importance of the data set, the weight of each data set is obtained, based on the weight, the weighted Euclidean distance of each data set is obtained, and according to the weighted Euclidean distance, the neighbor metric between each two data sets is calculated;

[0066] A key-value mapping operation is performed according to the normalized data set and the neighbor metric value to obtain a key-value pair, wherein the key-value pair includes a node, a node attribute, and a neighbor metric value.

[0067] Specifically, this method solves the classification problem of smart grid time series big data based on the idea of ​​distributed computing. According to the advantages of the nearest neighbor classification method, a parallel classification hybrid method - PCHA is designed by combining the Map-Reduce model and its fusion. First, the PCHA algorithm should be implemented, and the interface should be built to provide Map function and Reduce function, and indicate the input, output and other operating parameters. First, the input link is used to decompose the big data set into several independent data sets for the convenience of subsequent processing. Here, it is set to M' data sets. After submitting them to the JobTracker, the corresponding TaskTracker is used to execute the task.

[0068] In the Map-Reduce model, the Map function is usually used first: a large data set is decomposed into small data sets. This step is to perform mapping operations on the {ke, va} pairs of each decomposed data set. At this time, TaskTracker calls idle data resources to perform Map and Reduce tasks. The focus of the Map process is to perform key-value mapping operations on classified data sets. The tasks are based on the normalization operations of each attribute. At the same time, according to the importance and the weighted Euclidean distance results, the ((node, attribute), (neighborhood metric)) key-value pairs are obtained, and then classification is implemented based on the neighbor metric. Among them, the weighted Euclidean distance is calculated as follows:

[0069] Assume x i and x j is an n-dimensional data object. If each variable is assigned weights w1, w2, ..., wn according to its importance, then the weighted Euclidean distance can be expressed as:

[0070] x i =(x i1 , x i2 , …, x in )

[0071] x j =(x j1 , x j2 , …, x jn )

[0072]

[0073] In the formula, dis(x i , x j ) represents the weighted Euclidean distance.

[0074] Then, the neighbor metric is obtained based on the weighted Euclidean distance. Assume that there are two data set objects x in the n-dimensional space. i and x j , then the neighbor metric calculation formula is:

[0075] ||x i -x j ||=dis(x i , x j )

[0076] In the formula, ||x i -x j || represents the nearest neighbor metric, based on which we can judge x i and x j Whether it is a neighbor.

[0077] Attribute normalization operation means assuming that A represents a numerical attribute, A MAX , A MINRepresents the maximum and minimum values ​​of the attribute, in order to avoid the dominant influence of large number attributes on data classification, based on attribute normalization, the value falls within the range of 0-1. Among them, the attribute normalization expression is:

[0078]

[0079] In some embodiments, step S102 includes but is not limited to the following steps:

[0080] The nodes and node attributes in the key-value pairs are sorted using the second function, and the key-value pairs belonging to the same node and node attribute are classified into one category to obtain a classification result.

[0081] Specifically, the Reduce function is used to sort the nodes and node attributes in the key-value pairs, and the key-value pairs belonging to the same node and node attribute are classified into one category to obtain the classification results. The Reduce function mainly traverses the intermediate result set generated after all Map links are processed, and outputs ((node, attribute), (neighborhood metric)) in a unified manner according to the sorting and induction of the same (node, attribute) value, and then realizes parallel classification based on the neighbor metric.

[0082] In the output link, the output link corresponds to the input link, that is, its function is to perform output operations on the output result set of the Reduce link, and save the output to the specified location. This link obtains the classification result set obtained by running the PCHA algorithm, which is convenient for the next step of secure storage and processing.

[0083] In some embodiments, step S102 includes but is not limited to the following steps:

[0084] Use the trained Spark algorithm to calculate the time series data to obtain the optimal solution, and then obtain the classification result based on the optimal solution. The training process of the Spark algorithm is as follows:

[0085] The time series data samples are divided into a test set and a prediction set. The population size and population speed are set according to the test set. The individual historical optimal solution is set as the population initial position, and the group historical optimal value is set as the population position with the minimum fitness of the initial position.

[0086] Update the population position and population speed, calculate the fitness of each population, update the individual historical optimal solution of each population according to the fitness, update the inertia factor according to the individual historical optimal solution, update the group historical optimal value, and when the inertia factor is less than the preset value, stop updating to obtain the optimal solution.

[0087] Specifically, Spark can also be used to classify time series data. Figure 4It is the Spark ecosystem, in which SparkCore is the core component of Spark, responsible for Spark Context management, RDD encapsulation, and memory-based computing.

[0088] 1) Initialize Spark

[0089] Load data from HDFS and divide them into test sets and prediction sets. Since training and testing are performed in subtasks, the data will be downloaded from the cluster each time a subtask is started. This will increase unnecessary memory overhead on the node and reduce network transmission consumption from data to nodes. Therefore, broadcast variables are used so that each node copies one copy of the data instead of multiple copies during the entire life cycle of load forecasting, thereby reducing network transmission consumption and memory consumption.

[0090] The initial position and initial velocity of the particles in the particle swarm are randomly initialized, and the population size N is set to 40, that is, the number of particles is 40; for the support vector machine, the penalty factor C and the insensitive loss coefficient ε are optimized, and the particle space dimension D in the model is 2, that is, the velocity and position of each individual in the particle swarm are a 2-dimensional entity vector. When initializing, create a 3-dimensional array X i，j，k , where i is the number of particles N, j is a sequence from 0 to 2: 0 represents the current position, 1 represents the best historical position of the current particle, and k is the dimension D. Initial position X i，0，k =14×Rand-7, initial speed X i，2，k =2×Rand-1, the initial value of the individual's historical optimal solution is equal to the initial position of the particle, and the group's historical optimal value is the particle position with the lowest fitness at the initial position. The expression is:

[0091]

[0092] Among them, X[i][1] is the historical optimal position vector of the i-th particle, and f is the fitness function.

[0093] The fitness in this model is the root mean square error of the load forecast result. The test data is sent to all nodes as broadcast variables to avoid network transmission consumption caused by each training node requesting data from the cluster.

[0094] 2) Parallelize the inner loop of the particle swarm algorithm

[0095] The particle swarm algorithm updates the global optimal point at the end of each cycle, and the same operation is performed in parallel programming to find the optimal value of this round and compare and update the global optimal value. This means that there will be an operation to take the minimum value at the end of each cycle. On the one hand, it will wait for the slowest node in the previous stage to complete the task before entering the calculation of the current operator, resulting in a waste of computing resources of all nodes except the slowest node. On the other hand, the minimum value is a wide-dependency operator, which will pull data from all other nodes for calculation, resulting in the consumption of network resources. Therefore, the loop process of the particle swarm algorithm is modified in this model, and a double loop method is adopted.

[0096] 3) Parallelize the outer loop of the particle swarm algorithm

[0097] The adaptability of adaptive particle swarm optimization is reflected in the inertia factor w that is continuously updated during the algorithm. The update formula is as follows:

[0098]

[0099] Among them, w i is the current inertia factor value of particle No. i, w min is the preset minimum value of the inertia factor, w max is the preset maximum value of the inertia factor, f ave is the average value of the current fitness of particles in the population, f min is the minimum value of the current fitness of particles in the population, f min is the fitness value of the optimal particle in the current population. When the fitness value of particle i differs greatly from the optimal solution, the inertia factor is large, and the optimal solution can be found in a larger range. Otherwise, the inertia term is small, and the particle is pulled by the optimal solution to complete rapid convergence. In each outer loop, the inertia factor is updated before the inner loop, and the global optimal solution is updated after the inner loop.

[0100] The above model is used to make the algorithm more suitable for distributed environments, thereby improving the parallelization capability of the algorithm.

[0101] S103: According to the classification result, the time series data is stored in a distributed system.

[0102] In this embodiment, a scalable distributed storage system is used to store data, such as HDFS. The system can store massive data and achieve high scalability, high availability and fault tolerance, and ensure the security and reliability of data. At the same time, a distributed data synchronization system is used to achieve data synchronization and sharing between different nodes. The system has efficient data transmission and synchronization capabilities, and can perform real-time data synchronization and sharing, as well as batch data synchronization and sharing.

[0103] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0104] In some embodiments, Figure 5 As shown, it shows a block diagram of a network data processing device 500 for cloud-edge-end integration provided in an embodiment of the present application, including: a receiving module 501, a classification module 502 and a storage module 503, wherein:

[0105] The receiving module 501 is used to receive the time series data uploaded by each device;

[0106] The classification module 502 is used to use the processing logic class to call the first function to clean the time series data to obtain a cleaning result, decompose the cleaning result to obtain a number of data sets, perform a key-value mapping operation on a preset number of data sets to obtain key-value pairs, and use the second function to summarize the key-value pairs to obtain a classification result;

[0107] The storage module 503 is used to store the time series data in a distributed system according to the classification result.

[0108] In some embodiments, the classification module includes a first processing unit, a parsing unit, and an output unit.

[0109] The first processing unit is used to perform basic field processing on the time series data using the first processing class to obtain the fields of the time series data, and save the fields of the time series data in a table form to the second processing class to obtain a log class;

[0110] The parsing unit is used to parse the log class according to the filtering condition using the first filter to obtain the time series data field to be classified;

[0111] The output unit is used to output the cleaning result by using the third processing class according to the time series data field to be classified, wherein the rules in the first processing class, the second processing class and the third processing class can be configured as needed.

[0112] In some embodiments, the classification module further includes a normalization unit, a calculation unit, and a mapping unit.

[0113] Among them, the normalization unit is used to perform attribute normalization operations on each data set to obtain a normalized data set;

[0114] The calculation unit is used to obtain the weight of each data set according to the importance of the data set, obtain the weighted Euclidean distance of each data set based on the weight, and calculate the neighbor measurement value between each two data sets according to the weighted Euclidean distance;

[0115] The mapping unit is used to perform a key-value mapping operation according to the normalized data set and the neighbor metric value to obtain a key-value pair, wherein the key-value pair includes a node, a node attribute and a neighbor metric value.

[0116] In some embodiments, the key-value pairs are summarized using the second function to obtain a classification result, including:

[0117] The nodes and node attributes in the key-value pairs are sorted using the second function, and the key-value pairs belonging to the same node and node attribute are classified into one category to obtain a classification result.

[0118] In some embodiments, obtaining the classification result further includes:

[0119] Use the trained Spark algorithm to calculate the time series data to obtain the optimal solution, and then obtain the classification result based on the optimal solution. The training process of the Spark algorithm is as follows:

[0120] The time series data samples are divided into a test set and a prediction set. The population size and population speed are set according to the test set. The individual historical optimal solution is set as the population initial position, and the group historical optimal value is set as the population position with the minimum fitness of the initial position.

[0121] Update the population position and population speed, calculate the fitness of each population, update the individual historical optimal solution of each population according to the fitness, update the inertia factor according to the individual historical optimal solution, update the group historical optimal value, and when the inertia factor is less than the preset value, stop updating to obtain the optimal solution.

[0122] In one embodiment of the present application, a computer device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and the above steps are implemented when the processor executes the computer program; the computer device provided in this embodiment has an implementation principle and technical effects similar to those of the above method embodiments, and will not be repeated here.

[0123] In one embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and the above steps are implemented when the computer program is executed by a processor; the computer-readable storage medium provided in this embodiment has an implementation principle and technical effects similar to those of the above method embodiment, and will not be repeated here.

[0124] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0125] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0126] The specific embodiments described above further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A network data processing method for cloud-edge-end fusion, characterized in that: include: Receive timing data uploaded by each device; Using the processing logic class to call the first function to clean the time series data to obtain a cleaning result, decomposing the cleaning result to obtain a number of data sets, performing a key-value mapping operation on a preset number of the data sets to obtain key-value pairs, and using the second function to summarize the key-value pairs to obtain a classification result; According to the classification result, the time series data is stored in a distributed system.

2. The network data processing method for cloud-edge-end fusion according to claim 1 is characterized in that: The using the processing logic class to call the first function to clean the time series data to obtain a cleaning result includes: Using the first processing class to perform basic field processing on the time series data to obtain the fields of the time series data, and saving the fields of the time series data in a table form to the second processing class to obtain a log class; Utilizing the first filter to parse the log class according to the filtering condition, to obtain the time series data field to be classified; According to the time series data field to be classified, the third processing class is used for output to obtain a cleaning result, wherein the rules in the first processing class, the second processing class and the third processing class can be configured as needed.

3. The network data processing method for cloud-edge-end fusion according to claim 1, characterized in that: The step of performing a key-value mapping operation on a preset number of data sets to obtain key-value pairs includes: Performing attribute normalization operation on each of the data sets to obtain a normalized data set; According to the importance of the data sets, a weight of each of the data sets is obtained, based on the weight, a weighted Euclidean distance of each of the data sets is obtained, and according to the weighted Euclidean distance, a neighbor metric value between each of the data sets is calculated; A key-value mapping operation is performed according to the normalized data set and the neighbor metric value to obtain a key-value pair, wherein the key-value pair includes a node, a node attribute, and a neighbor metric value.

4. The network data processing method for cloud-edge-end fusion according to claim 3 is characterized in that: The using the second function to summarize the key-value pairs to obtain a classification result includes: The nodes and node attributes in the key-value pairs are sorted by using a second function, and the key-value pairs belonging to the same node and node attribute are classified into one category to obtain a classification result.

5. The network data processing method for cloud-edge-end fusion according to claim 1, characterized in that: The obtaining of the classification result further includes: The time series data is calculated using the trained Spark algorithm to obtain an optimal solution, and a classification result is obtained according to the optimal solution. The training process of the Spark algorithm is as follows: Divide the time series data samples into a test set and a prediction set, set the population size and population speed according to the test set, set the individual historical optimal solution as the population initial position, and set the group historical optimal value as the population position with the minimum fitness of the initial position; Update the population position and the population speed, calculate the fitness of each population, update the individual historical optimal solution of each population according to the fitness, update the inertia factor according to the individual historical optimal solution, update the group historical optimal value, and stop updating to obtain the optimal solution when the inertia factor is less than a preset value.

6. A network data processing device integrating cloud, edge and end, characterized in that: include: The receiving module is used to receive the timing data uploaded by each device; A classification module, configured to use a processing logic class to call a first function to clean the time series data to obtain a cleaning result, decompose the cleaning result to obtain a plurality of data sets, perform a key-value mapping operation on a preset number of the data sets to obtain key-value pairs, and use a second function to summarize the key-value pairs to obtain a classification result; A storage module is used to store the time series data in a distributed system according to the classification result.

7. The cloud-edge-end fusion network data processing device according to claim 6, characterized in that: The classification module includes a first processing unit, a parsing unit and an output unit. The first processing unit is used to perform basic field processing on the time series data using the first processing class to obtain the fields of the time series data, and save the fields of the time series data in a table form to the second processing class to obtain a log class; The parsing unit is used to parse the log class according to the filtering condition using the first filter to obtain the time series data field to be classified; The output unit is used to output according to the time series data field to be classified using the third processing class to obtain a cleaning result, wherein the rules in the first processing class, the second processing class and the third processing class can be configured as needed.

8. The cloud-edge-end fusion network data processing device according to claim 6, characterized in that: The classification module also includes a normalization unit, a calculation unit and a mapping unit. The normalization unit is used to perform attribute normalization operation on each of the data sets to obtain a normalized data set; The calculation unit is used to obtain the weight of each data set according to the importance of the data set, obtain the weighted Euclidean distance of each data set based on the weight, and calculate the neighbor metric value between each two data sets according to the weighted Euclidean distance; The mapping unit is used to perform a key-value mapping operation according to the normalized data set and the neighbor metric value to obtain a key-value pair, wherein the key-value pair includes a node, a node attribute and a neighbor metric value.

9. A computer device, characterized in that: It includes a processor and a memory, the memory is used to store a computer program, and when the computer program is executed by the processor, it implements the network data processing method of cloud-edge-end integration as described in any one of claims 1 to 5.

10. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the network data processing method for cloud-edge-end integration as described in any one of claims 1 to 5.

Images