A distributed data real-time processing method based on cloud-edge collaboration

Abstract

The present invention relates to the technical field of cloud-edge collaborative computing, and discloses a distributed data real-time processing method of cloud-edge collaboration, the method comprising: collecting network traffic data, and identifying task types according to the collected network traffic, and dividing tasks into computing-intensive tasks and storage-consuming tasks; performing branch processing according to task types, storage-consuming tasks directly access and store data through edge nodes, and computing-intensive tasks are divided into subtasks and assigned to different computing nodes by constructing a cloud-edge collaborative scheduling model to optimize cloud-edge distributed resource scheduling; optimizing and solving the cloud-edge collaborative scheduling model to obtain a distributed resource scheduling strategy, dividing the computing-intensive tasks into subtasks and assigning them to different computing nodes for processing according to the calculated strategy, and aggregating the calculation results after processing by each computing node to realize real-time data processing, thereby improving data processing efficiency and the overall performance of the system.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud-edge collaborative computing, and in particular to a cloud-edge collaborative distributed data real-time processing method. Background Art

[0002] With the rapid development of big data and IoT technologies, the amount of data generated by various sensors and devices is growing exponentially. The traditional centralized data processing model has been unable to cope with such a huge amount of data traffic, and cannot guarantee real-time processing and rapid response of data. Cloud computing has become the mainstream platform for big data processing with its powerful computing and storage capabilities. However, with the rise of edge computing, the combination of cloud computing and edge computing has gradually become a trend. Edge computing sinks data processing and storage capabilities to the edge nodes of the network and processes them close to the data source, thereby reducing the delay of data transmission and improving processing efficiency. At the same time, cloud computing provides centralized management, massive storage and complex computing capabilities, and can analyze and process global data. However, how to reasonably allocate and schedule computing tasks between cloud and edge nodes according to the characteristics of the tasks and the current status of resources to maximize resource utilization and processing efficiency is still a difficult point. Summary of the invention

[0003] In view of this, the present invention proposes a distributed data real-time processing method with cloud-edge collaboration. Through cloud-edge collaboration, the respective advantages of cloud computing and edge computing are fully utilized to provide efficient, reliable and real-time data processing capabilities.

[0004] To achieve the above object, the present invention provides a cloud-edge collaborative distributed data real-time processing method, comprising the following steps:

[0005] S1: Collect network traffic data through port mirroring, and identify task types based on the collected network traffic, and divide the tasks into computing-intensive tasks and storage-consuming tasks, wherein traffic identification based on spatial structure features is an implementation method of the task type identification;

[0006] S2: Branch processing is performed according to the identified task type. Storage-consuming tasks directly access and store data through edge nodes. Computation-intensive tasks are optimized by building a cloud-edge collaborative scheduling model to divide computing-intensive tasks into subtasks and assign them to different computing nodes. Multi-objective optimization is the implementation method of the cloud-edge collaborative scheduling model.

[0007] S3: Optimize and solve the constructed cloud-edge collaborative scheduling model to obtain the distributed resource scheduling strategy, where proximal gradient optimization is the implementation method for optimizing and solving the scheduling model;

[0008] S4: According to the calculated distributed resource scheduling strategy, the computing-intensive task is divided into subtasks and assigned to different computing nodes for processing. The calculation results after processing by each computing node are aggregated to realize real-time data processing.

[0009] As a further improvement method of the present invention:

[0010] Optionally, in step S1, task type identification is performed according to the collected network traffic, and tasks are divided into computing-intensive tasks and storage-intensive tasks, including:

[0011] S11: preprocessing the collected network traffic to obtain preprocessed network traffic data, wherein the preprocessing operation includes traffic deduplication and flow segmentation, wherein the traffic deduplication operation performs duplication detection on the collected traffic packets according to the characteristics of the data packets {source IP, destination IP, source port, destination port, protocol number, timestamp}, and filters the duplicate data packets to obtain deduplicated network traffic, and the flow segmentation operation performs flow extraction on the deduplicated network traffic according to the five-tuple {source IP, destination IP, source port, destination port, protocol number}, and divides the deduplicated network traffic into different communication flows;

[0012] S12: performing vector processing on the divided communication flow to obtain a communication flow vector, wherein the communication flow vector includes: a start timestamp, a communication flow duration, a protocol number, an average data packet size, a data packet size variance, a number of data packets, an average time interval between data packets, and a time interval variance between data packets;

[0013] S13: extracting structural features from the communication flow vector to obtain a spatial feature vector of the communication flow;

[0014] S14: using a deep neural network to identify the communication flow task type, wherein the deep neural network takes the spatial feature vector of the communication flow as input and takes the task type identification probability as output;

[0015] S15: Construct the target loss function under non-equilibrium conditions according to the task type recognition probability, and optimize and solve the model parameters. Instantiate the model according to the solved model parameters to perform task type recognition. The calculation formula is:

[0016] ;

[0017] in:

[0018] represents the first The task type recognition probability of each communication flow;

[0019] Indicates the number of communication flows;

[0020] Represents the equalized The task type identification result of the communication flow is calculated as follows:

[0021] ;

[0022] in:

[0023] Represents the balance factor, which is used to adjust the weight of communication flow samples. The value range is (0,1);

[0024] Represents the adjustment factor, which is used to adjust the importance of classification samples. The value range is (0,5);

[0025] In the objective loss function The role of is to reduce the contribution of communication flow samples of easy-to-identify task types and increase the contribution of communication flow samples of difficult-to-identify task types.

[0026] Optionally, extracting structural features from the communication flow vector in step S13 to obtain a spatial feature vector of the communication flow includes:

[0027] S13.1: Perform outer product operation on the communication flow vector to obtain the network flow feature matrix, and the calculation formula is:

[0028] ;

[0029] in:

[0030] is the communication flow vector;

[0031] represents the network flow feature matrix;

[0032] S13.2: Use convolution operation to perform preliminary feature extraction on the network flow feature matrix to obtain the communication flow feature graph. The calculation formula is:

[0033] ;

[0034] in:

[0035] * indicates convolution operation;

[0036] represents a nonlinear activation function;

[0037] represents the network flow feature matrix;

[0038] represents the convolution kernel;

[0039] S13.3: The communication flow feature graph is passed through an initial capsule layer to obtain a capsule vector set, wherein the initial capsule layer is composed of a preset number of primary capsules, each of which converts the communication flow feature graph into a capsule vector , the calculation formula is:

[0040] ;

[0041] in:

[0042] Representing a communication flow characteristic graph;

[0043] represents the first The convolution kernel of the primary capsule;

[0044] Indicates The bias vector of the primary capsule;

[0045] represents a nonlinear activation function;

[0046] Represents a vector concatenation operation;

[0047] S13.4: For each primary capsule output capsule vector , through the weight matrix linear transformation, get the prediction vector , the calculation formula is:

[0048] ;

[0049] in:

[0050] Indicates The capsule vector of the primary capsules;

[0051] Indicates that according to The capsule vector prediction of the primary capsule is about The prediction vector of the high-level capsules;

[0052] Indicates Primary capsule and The transformation matrix between high-level capsules linearly transforms the primary capsule output vector to the high-level capsule space;

[0053] Indicates Primary capsule and The bias vector between high-level capsules;

[0054] S13.5: Dynamically update the coupling coefficient between the primary capsule and the advanced capsule through the bias vector, and calculate the advanced capsule vector in combination with the obtained primary capsule prediction vector , the calculation formula is:

[0055] ;

[0056] in:

[0057] represents the coupling coefficient, which represents the coupling coefficient between primary capsule i and advanced capsule Contribution of

[0058] S13.6: The high-level capsule vector is compressed and normalized to obtain a compressed high-level capsule vector, and all compressed high-level capsule vectors are concatenated to obtain the spatial feature vector of the communication flow. The high-level capsule vector compression and normalization calculation formula is:

[0059] ;

[0060] in:

[0061] Indicates A compressed and normalized high-level capsule vector.

[0062] Optionally, in step S15, dynamically updating the coupling coefficient between the primary capsule and the advanced capsule by using the bias vector includes:

[0063] The coupling coefficient is calculated based on the bias vector between the primary capsule and the advanced capsule, and the calculation formula is:

[0064] ;

[0065] in:

[0066] Indicates The bias vector between the kth primary capsule and the kth higher-level capsule.

[0067] Optionally, in step S2, a cloud-edge collaborative scheduling model is constructed to perform cloud-edge distributed resource scheduling optimization to divide computing-intensive tasks into subtasks and assign them to different computing nodes, including:

[0068] The cloud-edge collaborative scheduling model achieves cloud-edge distributed resource scheduling optimization by dividing computing-intensive tasks into multiple non-interdependent subtasks and assigning them to different computing nodes for parallel solving. The model jointly optimizes the global computing node CPU load balancing level and computing efficiency in the task scheduling process through multi-objective optimization. The objective function is:

[0069] ;

[0070] ;

[0071] in:

[0072] represents the weight parameter;

[0073] Indicates the global computing node CPU load balancing level. The specific calculation process includes:

[0074] S21: Divide the computationally intensive task to be processed into a preset number of subtasks, and assign each subtask as an independent computing unit to a computing node for execution, wherein the computing node includes an edge device node and a cloud server node;

[0075] S22: Statistics on CPU utilization on each computing node And calculate the global CPU load balancing level, the calculation formula is:

[0076] ;

[0077] in:

[0078] Indicates the number of computing nodes;

[0079] Indicates CPU utilization of each computing node;

[0080] Indicates the number of subtasks;

[0081] is an indicator function, indicating whether Subtasks are assigned to If there are computing nodes, the value is 1, otherwise the value is 0;

[0082] Indicates the average CPU utilization of the global computing node;

[0083] Indicates the task calculation efficiency. The specific calculation process includes:

[0084] S23: Evaluate the computational efficiency of the subtask on each computing node. The calculation formula is:

[0085] ;

[0086] in:

[0087] Indicates The computing power of the CPU of each computing node, expressed in floating point operations per second (FLOPS);

[0088] Indicates The computational complexity of each subtask is expressed in terms of algorithmic complexity;

[0089] Indicates The expected completion time of each subtask;

[0090] S24: Determine the task computing efficiency according to the computing efficiency of the subtask on each computing node, and the calculation formula is:

[0091] ;

[0092] in:

[0093] Indicates The subtask is The computation efficiency on each computing node.

[0094] Optionally, in step S3, optimizing and solving the constructed cloud-edge collaborative scheduling model to obtain a distributed resource scheduling strategy includes:

[0095] S31: Randomly initialize distributed resource scheduling strategy variables , step size parameter , proximal parameters ;

[0096] S32: For each round of iteration ;

[0097] The iterative process includes:

[0098] S32.1: Calculate current variables The gradient at ,in represents the objective function;

[0099] S32.2: Applying the proximal operator Iteratively update variables , the variable update calculation formula is:

[0100] ;

[0101] represents the regularization function The calculation formula of the proximal operator under the constraint is:

[0102] ;

[0103] in:

[0104] represents the optimization variable of the proximal operator;

[0105] represents the proximal parameter;

[0106] z represents the proximal endpoint variable;

[0107] S33: Determine whether the stop condition is met according to a preset convergence criterion and output the optimal distributed resource scheduling strategy. For example, the convergence criterion includes that the change in the variable value is less than a preset threshold, the norm of the gradient is less than a preset threshold, or the maximum number of iterations is reached.

[0108] Optionally, in step S4, the computing-intensive task is divided into subtasks and assigned to different computing nodes for processing according to the calculated distributed resource scheduling strategy, including:

[0109] According to the calculated distributed resource scheduling strategy, the decomposed subtasks without mutual dependence are assigned to the computing nodes specified by the optimal strategy. Each computing node executes the decomposed subtasks in parallel, uses local computing resources for data processing and calculation operations, accelerates the processing of computing-intensive tasks, and sends the local results processed by each computing node back to the central node for aggregation, thereby realizing real-time data processing.

[0110] In order to solve the above problem, the present invention provides an electronic device, the electronic device comprising:

[0111] A memory storing at least one instruction;

[0112] Communication interface, enabling electronic equipment to communicate; and

[0113] The processor executes the instructions stored in the memory to implement the above-mentioned cloud-edge collaborative distributed data real-time processing method.

[0114] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one instruction is stored. The at least one instruction is executed by a processor in an electronic device to implement the above-mentioned cloud-edge collaborative distributed data real-time processing method.

[0115] Compared with the prior art, the present invention proposes a distributed data real-time processing method with cloud-edge collaboration, which has the following advantages:

[0116] (1) This scheme proposes a network traffic collection method based on port mirroring technology. This method uses port mirroring technology to collect network traffic data and combines spatial structure features to identify task types. It uses the spatial structure features of network traffic data to classify task types. Compared with traditional classification methods based on data volume or processing time, this method is more accurate and effective.

[0117] (2) This solution proposes a cloud-edge collaborative scheduling model, which dynamically schedules tasks based on the identified task types. Storage-consuming tasks are directly processed on edge nodes, while computing-intensive tasks are divided into subtasks and optimally allocated to cloud and edge nodes for processing. This fully utilizes the advantages of edge computing and cloud computing resources, achieving efficient resource utilization and optimized task processing.

[0118] (3) This solution introduces a proximal gradient optimization method to optimize the cloud-edge collaborative scheduling model, which can reduce communication overhead and latency while ensuring task processing efficiency, and is particularly suitable for real-time data processing scenarios;

[0119] (4) This solution integrates multiple technologies such as port mirroring, task type identification, cloud-edge collaborative scheduling and distributed resource scheduling, and proposes a complete real-time data processing solution that can effectively cope with the challenges of big data processing in modern network environments and improve data processing efficiency and the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0120] Figure 1 A flowchart of a distributed data real-time processing method for cloud-edge collaboration provided by an embodiment of the present invention;

[0121] Figure 2 A schematic diagram of the structure of an electronic device for implementing a distributed data real-time processing method for cloud-edge collaboration provided by an embodiment of the present invention;

[0122] In the figure: 1 electronic device, 10 processor, 11 memory, 12 program, 13 communication interface.

[0123] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0124] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0125] The embodiment of the present application provides a distributed data real-time processing method in cloud-edge collaboration. The execution subject of the distributed data real-time processing method in cloud-edge collaboration includes but is not limited to at least one of the electronic devices such as a server, a terminal, etc. that can be configured to execute the method provided in the embodiment of the present application. In other words, the distributed data real-time processing method in cloud-edge collaboration can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc.

[0126] S1: Collect network traffic data through port mirroring, and identify task types based on the collected network traffic, and divide the tasks into computing-intensive tasks and storage-consuming tasks, wherein traffic identification based on spatial structure characteristics is an implementation method of the task type identification.

[0127] In the step S1, task types are identified based on the collected network traffic, and tasks are divided into computing-intensive tasks and storage-intensive tasks, including:

[0128] S11: preprocessing the collected network traffic to obtain preprocessed network traffic data, wherein the preprocessing operation includes traffic deduplication and flow segmentation, wherein the traffic deduplication operation performs duplication detection on the collected traffic packets according to the characteristics of the data packets {source IP, destination IP, source port, destination port, protocol number, timestamp}, and filters the duplicate data packets to obtain deduplicated network traffic, and the flow segmentation operation performs flow extraction on the deduplicated network traffic according to the five-tuple {source IP, destination IP, source port, destination port, protocol number}, and divides the deduplicated network traffic into different communication flows;

[0129] S12: performing vector processing on the divided communication flow to obtain a communication flow vector, wherein the communication flow vector includes: a start timestamp, a communication flow duration, a protocol number, an average data packet size, a data packet size variance, a number of data packets, an average time interval between data packets, and a time interval variance between data packets;

[0130] S13: extracting structural features from the communication flow vector to obtain a spatial feature vector of the communication flow;

[0131] S14: using a deep neural network to identify the communication flow task type, wherein the deep neural network takes the spatial feature vector of the communication flow as input and takes the task type identification probability as output;

[0132] S15: Construct the target loss function under non-equilibrium conditions according to the task type recognition probability, and optimize and solve the model parameters. Instantiate the model according to the solved model parameters to perform task type recognition. The calculation formula is:

[0133] ;

[0134] in:

[0135] represents the first The task type recognition probability of each communication flow;

[0136] Indicates the number of communication flows;

[0137] Represents the equalized The task type identification result of the communication flow is calculated as follows:

[0138] ;

[0139] in:

[0140] Represents the balance factor, which is used to adjust the weight of communication flow samples. The value range is (0,1);

[0141] Represents the adjustment factor, which is used to adjust the importance of classification samples, and its value range is (0,5);

[0142] In the objective loss function The role of is to reduce the contribution of communication flow samples of easy-to-identify task types and increase the contribution of communication flow samples of difficult-to-identify task types.

[0143] The step S13 extracts structural features from the communication flow vector to obtain a spatial feature vector of the communication flow, including:

[0144] S13.1: Perform outer product operation on the communication flow vector to obtain the network flow feature matrix, and the calculation formula is:

[0145] ;

[0146] in:

[0147] is the communication flow vector;

[0148] represents the network flow feature matrix;

[0149] S13.2: Use convolution operation to perform preliminary feature extraction on the network flow feature matrix to obtain the communication flow feature graph. The calculation formula is:

[0150] ;

[0151] in:

[0152] * indicates convolution operation;

[0153] represents a nonlinear activation function;

[0154] represents the network flow feature matrix;

[0155] represents the convolution kernel;

[0156] S13.3: The communication flow feature graph is passed through an initial capsule layer to obtain a capsule vector set, wherein the initial capsule layer is composed of a preset number of primary capsules, each of which converts the communication flow feature graph into a capsule vector , the calculation formula is:

[0157] ;

[0158] in:

[0159] Representing a communication flow characteristic graph;

[0160] represents the first The convolution kernel of the primary capsule;

[0161] Indicates The bias vector of the primary capsule;

[0162] represents a nonlinear activation function;

[0163] Represents a vector concatenation operation;

[0164] S13.4: For each primary capsule output capsule vector , through the weight matrix linear transformation, get the prediction vector , the calculation formula is:

[0165] ;

[0166] in:

[0167] Indicates The capsule vector of the primary capsules;

[0168] Indicates that according to The capsule vector prediction of the primary capsule is about The prediction vector of the high-level capsules;

[0169] Indicates Primary capsule and The transformation matrix between high-level capsules linearly transforms the primary capsule output vector to the high-level capsule space;

[0170] Indicates Primary capsule and The bias vector between high-level capsules;

[0171] S13.5: Dynamically update the coupling coefficient between the primary capsule and the advanced capsule through the bias vector, and calculate the advanced capsule vector in combination with the obtained primary capsule prediction vector , the calculation formula is:

[0172] ;

[0173] in:

[0174] represents the coupling coefficient, which represents the coupling coefficient between primary capsule i and advanced capsule Contribution of

[0175] S13.6: The high-level capsule vector is compressed and normalized to obtain a compressed high-level capsule vector, and all compressed high-level capsule vectors are concatenated to obtain the spatial feature vector of the communication flow. The high-level capsule vector compression and normalization calculation formula is:

[0176] ;

[0177] in:

[0178] Indicates A compressed and normalized high-level capsule vector.

[0179] In the step S15, the coupling coefficient between the primary capsule and the advanced capsule is dynamically updated by the bias vector, including:

[0180] The coupling coefficient is calculated based on the bias vector between the primary capsule and the advanced capsule, and the calculation formula is:

[0181] ;

[0182] in:

[0183] Indicates The bias vector between the kth primary capsule and the kth higher-level capsule.

[0184] S2: Branch processing is performed according to the identified task types. Storage-consuming tasks directly access and store data through edge nodes. Computation-intensive tasks are optimized by building a cloud-edge collaborative scheduling model to divide the computing-intensive tasks into subtasks and assign them to different computing nodes. Multi-objective optimization is the implementation method of the cloud-edge collaborative scheduling model.

[0185] In step S2, a cloud-edge collaborative scheduling model is constructed to optimize cloud-edge distributed resource scheduling, dividing computing-intensive tasks into subtasks and assigning them to different computing nodes, including:

[0186] The cloud-edge collaborative scheduling model achieves cloud-edge distributed resource scheduling optimization by dividing computing-intensive tasks into multiple non-interdependent subtasks and assigning them to different computing nodes for parallel solving. The model jointly optimizes the global computing node CPU load balancing level and computing efficiency in the task scheduling process through multi-objective optimization. The objective function is:

[0187] ;

[0188] ;

[0189] in:

[0190] represents the weight parameter;

[0191] Indicates the global computing node CPU load balancing level. The specific calculation process includes:

[0192] S21: Divide the computationally intensive task to be processed into a preset number of subtasks, and assign each subtask as an independent computing unit to a computing node for execution, wherein the computing node includes an edge device node and a cloud server node;

[0193] S22: Statistics on CPU utilization on each computing node And calculate the global CPU load balancing level, the calculation formula is:

[0194] ;

[0195] in:

[0196] Indicates the number of computing nodes;

[0197] Indicates CPU utilization of each computing node;

[0198] Indicates the number of subtasks;

[0199] is an indicator function, indicating whether Subtasks are assigned to If there are computing nodes, the value is 1, otherwise the value is 0;

[0200] Indicates the average CPU utilization of the global computing node;

[0201] Indicates the task calculation efficiency. The specific calculation process includes:

[0202] S23: Evaluate the computational efficiency of the subtask on each computing node. The calculation formula is:

[0203] ;

[0204] in:

[0205] Indicates The computing power of the CPU of each computing node, expressed in floating point operations per second (FLOPS);

[0206] Indicates The computational complexity of each subtask is expressed in terms of algorithmic complexity;

[0207] Indicates The expected completion time of each subtask;

[0208] S24: Determine the task computing efficiency according to the computing efficiency of the subtask on each computing node, and the calculation formula is:

[0209] ;

[0210] in:

[0211] Indicates The subtask is The computation efficiency on each computing node.

[0212] S3: The constructed cloud-edge collaborative scheduling model is optimized and solved to obtain the distributed resource scheduling strategy, among which the proximal gradient optimization is the implementation method for optimizing and solving the scheduling model.

[0213] In the S3 step, the constructed cloud-edge collaborative scheduling model is optimized and solved to obtain a distributed resource scheduling strategy, including:

[0214] S31: Randomly initialize distributed resource scheduling strategy variables , step size parameter , proximal parameters ;

[0215] S32: For each round of iteration ;

[0216] The iterative process includes:

[0217] S32.1: Calculate current variables The gradient at ,in represents the objective function;

[0218] S32.2: Applying the proximal operator Iteratively update variables , the variable update calculation formula is:

[0219] ;

[0220] represents the regularization function The calculation formula of the proximal operator under the constraint is:

[0221] ;

[0222] in:

[0223] represents the optimization variable of the proximal operator;

[0224] represents the proximal parameter;

[0225] z represents the proximal endpoint variable;

[0226] S33: Determine whether the stop condition is met according to a preset convergence criterion and output the optimal distributed resource scheduling strategy. For example, the convergence criterion includes that the change in the variable value is less than a preset threshold, the norm of the gradient is less than a preset threshold, or the maximum number of iterations is reached.

[0227] S4: According to the calculated distributed resource scheduling strategy, the computing-intensive task is divided into subtasks and assigned to different computing nodes for processing. The calculation results after processing by each computing node are aggregated to realize real-time data processing.

[0228] In the step S4, the computing-intensive task is divided into subtasks and assigned to different computing nodes for processing according to the calculated distributed resource scheduling strategy, including:

[0229] According to the calculated distributed resource scheduling strategy, the decomposed subtasks without mutual dependence are assigned to the computing nodes specified by the optimal strategy. Each computing node executes the decomposed subtasks in parallel, uses local computing resources for data processing and calculation operations, accelerates the processing of computing-intensive tasks, and sends the local results processed by each computing node back to the central node for aggregation, thereby realizing real-time data processing.

[0230] like Figure 2 The figure is a schematic diagram of the structure of an electronic device for implementing a distributed data real-time processing method with cloud-edge collaboration provided by an embodiment of the present invention.

[0231] The electronic device 1 may include a processor 10 , a memory 11 , a communication interface 13 , and a bus, and may also include a computer program stored in the memory 11 and executable on the processor 10 , such as a program 12 .

[0232] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 11 may also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 11 may also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 may not only be used to store application software and various types of data installed in the electronic device 1, such as the code of the program 12, etc., but also be used to temporarily store data that has been output or is to be output.

[0233] The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and a combination of various control chips. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect the various components of the entire electronic device, and executes or executes the programs or modules stored in the memory 11 (such as the program 12 for implementing distributed data real-time processing of cloud-edge collaboration), and calls the data stored in the memory 11 to execute various functions of the electronic device 1 and process data.

[0234] The communication interface 13 may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices, and to achieve connection and communication between internal components of the electronic device.

[0235] The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0236] Figure 2 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 2 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0237] For example, although not shown, the electronic device 1 may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0238] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0239] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0240] The program 12 stored in the memory 11 in the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can achieve:

[0241] Collecting network traffic data through port mirroring, and identifying task types based on the collected network traffic, dividing the tasks into computing-intensive tasks and storage-consuming tasks, wherein the traffic based on spatial structure characteristics is identified as the task;

[0242] Branch processing is performed according to the identified task type. Storage-consuming tasks directly access and store data through edge nodes. Computation-intensive tasks are optimized by building a cloud-edge collaborative scheduling model to optimize cloud-edge distributed resource scheduling and divide computation-intensive tasks into subtasks and assign them to different computing nodes.

[0243] The constructed cloud-edge collaborative scheduling model is optimized and solved to obtain the distributed resource scheduling strategy;

[0244] According to the calculated distributed resource scheduling strategy, the computing-intensive tasks are divided into subtasks and assigned to different computing nodes for processing. The calculation results after processing by each computing node are aggregated to realize real-time data processing.

[0245] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to Figure 1 to Figure 2 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0246] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0247] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0248] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A distributed data real-time processing method with cloud-edge collaboration, characterized in that: The method comprises: S1: Collect network traffic data through port mirroring, and identify task types based on the collected network traffic, and divide the tasks into computing-intensive tasks and storage-consuming tasks, wherein traffic identification based on spatial structure features is an implementation method of the task type identification; S2: Branch processing is performed according to the identified task type. Storage-consuming tasks directly access and store data through edge nodes. Computation-intensive tasks are optimized by building a cloud-edge collaborative scheduling model to divide computing-intensive tasks into subtasks and assign them to different computing nodes. Multi-objective optimization is the implementation method of the cloud-edge collaborative scheduling model. S3: Optimize and solve the constructed cloud-edge collaborative scheduling model to obtain the distributed resource scheduling strategy, where proximal gradient optimization is the implementation method for optimizing and solving the scheduling model; The constructed cloud-edge collaborative scheduling model is optimized and solved to obtain the distributed resource scheduling strategy, including: S31: Randomly initialize distributed resource scheduling strategy variables , step size parameter , proximal parameters ; S32: For each iteration ; The iterative process includes: S32.1: Calculate current variables The gradient at ,in represents the objective function; S32.2: Applying the proximal operator Iteratively update variables , the variable update calculation formula is: ； represents the regularization function The calculation formula of the proximal operator under the constraint is: ； in: represents the optimization variable of the proximal operator; represents the proximal parameter; z represents the proximal endpoint variable; S33: judging whether the stop condition is met according to a preset convergence criterion and outputting an optimal distributed resource scheduling strategy, wherein the convergence criterion includes that the change in the variable value is less than a preset threshold, the norm of the gradient is less than a preset threshold, or the maximum number of iterations is reached; S4: According to the calculated distributed resource scheduling strategy, the computing-intensive task is divided into subtasks and assigned to different computing nodes for processing. The calculation results after processing by each computing node are aggregated to realize real-time data processing.

2. A distributed data real-time processing method with cloud-edge collaboration as claimed in claim 1, characterized in that: In the step S1, task types are identified based on the collected network traffic, and tasks are divided into computing-intensive tasks and storage-intensive tasks, including: S11: performing a preprocessing operation on the collected network traffic to obtain preprocessed network traffic data, wherein the preprocessing operation includes traffic deduplication and flow segmentation, wherein the traffic deduplication operation performs a repeatability detection on the collected traffic packets according to the characteristics of the data packets {source IP, destination IP, source port, destination port, protocol number, timestamp}, and filters the repeated data packets to obtain the deduplicated network traffic, and the flow segmentation operation performs flow extraction on the deduplicated network traffic according to the five-tuple {source IP, destination IP, source port, destination port, protocol number}, and divides the deduplicated network traffic into different communication flows; S12: performing vector processing on the divided communication flow to obtain a communication flow vector, wherein the communication flow vector includes: a start timestamp, a communication flow duration, a protocol number, an average data packet size, a data packet size variance, a number of data packets, an average time interval between data packets, and a time interval variance between data packets; S13: extracting structural features from the communication flow vector to obtain a spatial feature vector of the communication flow; S14: using a deep neural network to identify the communication flow task type, wherein the deep neural network takes the spatial feature vector of the communication flow as input and takes the task type identification probability as output; S15: Construct the target loss function under non-equilibrium conditions according to the task type recognition probability, and optimize and solve the model parameters. Instantiate the model according to the solved model parameters to perform task type recognition. The calculation formula is: ； in: represents the first The task type recognition probability of each communication flow; Indicates the number of communication flows; Represents the equalized The task type identification result of the communication flow is calculated as follows: ； in: Represents the balance factor, which is used to adjust the weight of communication flow samples. The value range is (0,1); Represents the adjustment factor, which is used to adjust the importance of classification samples. The value range is (0,5).

3. A distributed data real-time processing method with cloud-edge collaboration as claimed in claim 2, characterized in that: The step S13 extracts structural features from the communication flow vector to obtain a spatial feature vector of the communication flow, including: S13.1: Perform outer product operation on the communication flow vector to obtain the network flow feature matrix, and the calculation formula is: ； in: is the communication flow vector; represents the network flow feature matrix; S13.2: Use convolution operation to perform preliminary feature extraction on the network flow feature matrix to obtain the communication flow feature graph. The calculation formula is: ； in: * indicates convolution operation; represents a nonlinear activation function; represents the network flow feature matrix; represents the convolution kernel; S13.3: The communication flow feature graph is passed through an initial capsule layer to obtain a capsule vector set, wherein the initial capsule layer is composed of a preset number of primary capsules, each of which converts the communication flow feature graph into a capsule vector , the calculation formula is: ； in: Representing a communication flow characteristic graph; represents the first The convolution kernel of the primary capsule; Indicates The bias vector of each primary capsule; represents a nonlinear activation function; Represents a vector concatenation operation; S13.4: For each primary capsule output capsule vector , through the weight matrix linear transformation, get the prediction vector , the calculation formula is: ； in: Indicates The capsule vector of the primary capsules; Indicates that according to The capsule vector prediction of the primary capsule is about The prediction vector of the high-level capsules; Indicates Primary capsule and The transformation matrix between high-level capsules linearly transforms the primary capsule output vector to the high-level capsule space; Indicates Primary capsule and The bias vector between high-level capsules; S13.5: Dynamically update the coupling coefficient between the primary capsule and the advanced capsule through the bias vector, and calculate the advanced capsule vector in combination with the obtained primary capsule prediction vector , the calculation formula is: ； in: represents the coupling coefficient, which represents the coupling coefficient between primary capsule i and advanced capsule Contribution of S13.6: The high-level capsule vector is compressed and normalized to obtain a compressed high-level capsule vector, and all compressed high-level capsule vectors are concatenated to obtain the spatial feature vector of the communication flow. The high-level capsule vector compression and normalization calculation formula is: ； in: Indicates A compressed and normalized high-level capsule vector.

4. The distributed data real-time processing method of cloud-edge collaboration as claimed in claim 3 is characterized in that: In the step S13.5, the coupling coefficient between the primary capsule and the advanced capsule is dynamically updated by the bias vector, including: The coupling coefficient is calculated based on the bias vector between the primary capsule and the advanced capsule, and the calculation formula is: ； in: Indicates The bias vector between the kth primary capsule and the kth higher-level capsule.

5. The distributed data real-time processing method of cloud-edge collaboration as claimed in claim 1, characterized in that: In step S2, a cloud-edge collaborative scheduling model is constructed to optimize cloud-edge distributed resource scheduling, dividing computing-intensive tasks into subtasks and assigning them to different computing nodes, including: The cloud-edge collaborative scheduling model achieves cloud-edge distributed resource scheduling optimization by dividing computing-intensive tasks into multiple non-interdependent subtasks and assigning them to different computing nodes for parallel solving. The model jointly optimizes the global computing node CPU load balancing level and computing efficiency in the task scheduling process through multi-objective optimization. The objective function is: ； ； in: represents the weight parameter; Indicates the global computing node CPU load balancing level. The specific calculation process includes: S21: Divide the computationally intensive task to be processed into a preset number of subtasks, and assign each subtask as an independent computing unit to a computing node for execution, wherein the computing node includes an edge device node and a cloud server node; S22: Statistics on CPU utilization on each computing node And calculate the global CPU load balancing level, the calculation formula is: ； in: Indicates the number of computing nodes; Indicates CPU utilization of each computing node; Indicates the number of subtasks; is an indicator function, indicating whether Subtasks are assigned to If there are computing nodes, the value is 1, otherwise the value is 0; Indicates the average CPU utilization of the global computing node; Indicates the task calculation efficiency. The specific calculation process includes: S23: Evaluate the computational efficiency of the subtask on each computing node. The calculation formula is: ； in: Indicates The computing power of the CPU of each computing node, expressed in floating point operations per second (FLOPS); Indicates The computational complexity of each subtask is expressed in terms of algorithmic complexity; Indicates The expected completion time of each subtask; Indicates The subtask is Computational efficiency on each computing node; S24: Determine the task computing efficiency according to the computing efficiency of the subtask on each computing node, and the calculation formula is: .

6. The distributed data real-time processing method of cloud-edge collaboration as claimed in claim 1, characterized in that: In the step S4, the computing-intensive task is divided into subtasks and assigned to different computing nodes for processing according to the calculated distributed resource scheduling strategy, including: According to the calculated distributed resource scheduling strategy, the decomposed subtasks without mutual dependence are assigned to the computing nodes specified by the optimal strategy. Each computing node executes the decomposed subtasks in parallel, uses local computing resources for data processing and calculation operations, accelerates the processing of computing-intensive tasks, and sends the local results processed by each computing node back to the central node for aggregation, thereby realizing real-time data processing.

Images