Data processing method and system based on cloud edge collaboration

Abstract

The invention discloses a data processing method and system based on cloud edge collaboration, and relates to the technical field of artificial intelligence and data mining. Comprising the steps of 1, performing data preprocessing on edge nodes based on a cloud edge collaborative architecture, performing multi-stage filtering, feature extraction, compression and coding on data, and uploading the preprocessed data to a cloud end, and 2, performing dynamic task scheduling on the cloud end according to the obtained data; the method comprises the following steps: establishing an edge computing resource pool, performing unified management and scheduling on edge computing resources by utilizing the edge computing resource pool, encrypting data by adopting a security protocol in a data transmission process, and performing user identity verification and authorization management by utilizing an access control and authentication method.

Description

Technical Field

[0001] The present invention discloses a data processing method and system based on cloud-edge collaboration, which relates to the technical fields of artificial intelligence and data mining. Background Art

[0002] Cloud computing, with its powerful data processing capabilities and resource pooling characteristics, has greatly promoted digital transformation. However, with the explosive growth of Internet of Things devices, the data volume has increased sharply, and there are more and more application scenarios with high real-time requirements. The method of processing all data in the cloud has been unable to meet the explosive growth demand. Summary of the Invention

[0003] In view of the problems of the prior art, the present invention provides a data processing method and system based on cloud-edge collaboration. By deploying computing, storage, and application programs at the network edge, local processing and analysis of data are realized, latency is reduced, and the pressure on the cloud center is alleviated.

[0004] The specific solution proposed by the present invention is as follows:

[0005] The present invention provides a data processing method based on cloud-edge collaboration, including:

[0006] Step 1: Based on the cloud-edge collaboration architecture, perform data preprocessing at the edge node: perform multi-level filtering, feature extraction, and compression and encoding on the data,

[0007] Upload the preprocessed data to the cloud,

[0008] Step 2: Perform dynamic task scheduling in the cloud according to the obtained data:

[0009] Step 21: Considering the real-time requirements, data importance, and computing resource consumption factors of the task comprehensively, construct a decision-making model for multi-objective optimized QoS, and use the decision-making model to perform intelligent allocation of tasks between the cloud and the edge,

[0010] Use machine learning algorithms to establish a prediction model, use the prediction model to predict the network status and edge node load in the next period of time, and dynamically adjust the task allocation in combination with the current cloud system status,

[0011] Use the load balancing algorithm to evenly distribute tasks among multiple edge nodes to avoid single-point overload, and at the same time schedule cloud resources to intervene to cope with sudden traffic;

[0012] Step 22: Establish an edge computing resource pool, use the edge computing resource pool to uniformly manage and schedule edge computing resources, and at the same time encrypt data using a security protocol during the data transmission process, and use access control and authentication methods to perform user identity verification and authorization management.

[0013] Further, in step 1 of the data processing method based on cloud-edge collaboration, multi-level filtering of data is performed, including: first, initially screening the original data through a rule engine to eliminate irrelevant or redundant information; secondly, using statistical analysis methods to identify outliers and avoid uploading invalid data to the cloud.

[0014] Feature extraction of data is performed, including: for specific application scenarios, initially performing feature extraction through edge nodes.

[0015] Data compression and encoding are carried out: using data compression algorithms to reduce the data volume while ensuring the quality of the restored data.

[0016] Further, in step 22 of the data processing method based on cloud-edge collaboration, the edge computing resource pool is used to uniformly manage and schedule edge computing resources, including:

[0017] Resource virtualization management is carried out: using containerization methods to virtualize the edge computing resources, facilitating the rapid deployment and elastic expansion of resources.

[0018] Resource status monitoring is carried out: deploying an edge monitoring system to collect the usage of CPU, memory, and storage resources in real time, providing accurate data support for resource scheduling.

[0019] Resource allocation is carried out: matching appropriate edge resources according to task requirements, and at the same time considering resource reservation strategies to cope with future demand fluctuations.

[0020] Further, in step 22 of the data processing method based on cloud-edge collaboration, the TLS / SSL protocol is used to encrypt data during the data transmission process. A key management system is adopted between the edge node and the cloud to achieve end-to-end encryption of the data. Differential privacy and homomorphic encryption methods are also used to protect personal privacy information from being leaked while not affecting the data analysis results.

[0021] The present invention also provides a data processing system based on cloud-edge collaboration, including a data preprocessing module, a scheduling module, and a security management module.

[0022] The data preprocessing module is based on the cloud-edge collaboration architecture and performs data preprocessing at the edge node: performing multi-level filtering, feature extraction, and compression and encoding of the data, and uploading the preprocessed data to the cloud.

[0023] The scheduling module performs dynamic task scheduling in the cloud according to the obtained data:

[0024] The scheduling module comprehensively considers factors such as the real-time requirements of tasks, data importance, and computing resource consumption, constructs a decision-making model for multi-objective optimized QoS, and uses the decision-making model to intelligently allocate tasks between the cloud and the edge.

[0025] A prediction model is established using machine learning algorithms. The prediction model is used to predict the network conditions and edge node loads in a future period of time, and the task allocation is dynamically adjusted in combination with the current cloud system status.

[0026] The load balancing algorithm is used to evenly distribute tasks among multiple edge nodes to avoid single-point overload. At the same time, cloud resources are scheduled to intervene to cope with sudden traffic.

[0027] The scheduling module establishes an edge computing resource pool to uniformly manage and schedule edge computing resources. At the same time, the security management module encrypts data using a security protocol during data transmission, and uses access control and authentication methods to authenticate and authorize user identities.

[0028] Furthermore, the data preprocessing module of the data processing system based on cloud-edge collaboration performs multi-level filtering on data, including: first, initially screening the raw data through a rule engine to eliminate irrelevant or redundant information; secondly, using statistical analysis methods to identify outliers to avoid uploading invalid data to the cloud.

[0029] Feature extraction is performed on the data, including: for specific application scenarios, preliminary feature extraction is performed by edge nodes.

[0030] Data compression and encoding are performed: data compression algorithms are used to reduce the data volume while ensuring the quality of the data after recovery.

[0031] Furthermore, the scheduling module of the data processing system based on cloud-edge collaboration uses the edge computing resource pool to uniformly manage and schedule edge computing resources, including:

[0032] Resource virtualization management is carried out: the containerization method is used to virtualize the edge computing resources, which is convenient for the rapid deployment and elastic expansion of resources.

[0033] Resource status monitoring is carried out: an edge monitoring system is deployed to collect the usage of CPU, memory, and storage resources in real time, providing accurate data support for resource scheduling.

[0034] Resource allocation is carried out: appropriate edge resources are matched according to task requirements, and resource reservation strategies are considered to cope with future demand fluctuations.

[0035] Furthermore, the security management module of the data processing system based on cloud-edge collaboration encrypts data using the TLS / SSL protocol during data transmission. A key management system is used between the edge nodes and the cloud to achieve end-to-end encryption of the data. Differential privacy and homomorphic encryption methods are also used to protect personal privacy information from being leaked while not affecting the data analysis results.

[0036] The advantages of the method of the present invention are as follows:

[0037] 1. Real-time and low-latency optimization: Through cloud-edge collaboration, data processing and analysis are performed at edge nodes close to the data source, significantly reducing the response time, which is suitable for application scenarios with strict real-time requirements such as autonomous driving and remote surgery. The technological innovation focuses on efficient data transmission protocols, edge caching strategies, and dynamic task scheduling algorithms.

[0038] 2. Data security and privacy protection: Edge computing can process sensitive data at the source, reducing data upload and lowering the risk of leakage. Combining technologies such as blockchain and homomorphic encryption, the cloud-edge collaboration system can further enhance data security and privacy protection.

[0039] 3. Resource optimization and energy consumption management: Cloud-edge collaboration requires an efficient resource allocation mechanism to ensure the rational use of computing resources. By predicting future loads through machine learning algorithms, dynamically adjusting cloud-edge resource allocation, and considering energy consumption issues, green computing is achieved. Patents may involve resource scheduling algorithms, energy efficiency ratio optimization models, etc.

[0040] 4. Intelligent application and service innovation: In scenarios such as smart cities and industrial Internet, cloud-edge collaboration supports the implementation of a large number of intelligent applications, such as intelligent monitoring and predictive maintenance. These applications rely on the efficient data processing capabilities of cloud-edge collaboration and algorithm optimization for specific scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the application framework of the method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0042] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, so that those skilled in the art can better understand the present invention and be able to implement it, but the embodiments given are not intended to limit the present invention.

[0043] Embodiment 1

[0044] The present invention provides a data processing method based on cloud-edge collaboration, including:

[0045] Step 1: Based on the cloud-edge collaboration architecture, perform data preprocessing at the edge node: perform multi-level filtering, feature extraction, and compression and encoding on the data,

[0046] and upload the preprocessed data to the cloud.

[0047] Among them, performing multi-level filtering on the data includes: first, initially screening the original data through a rule engine to eliminate irrelevant or redundant information; secondly, using statistical analysis methods to identify outliers to avoid uploading invalid data to the cloud,

[0048] Extract features from the data, including: for specific application scenarios, perform preliminary feature extraction through edge nodes, such as edge detection in image recognition and spectral analysis in sound recognition, to reduce the dimension of the uploaded data and relieve the processing burden on the cloud.

[0049] Compress and encode the data: use data compression algorithms to reduce the data volume while ensuring the quality of the restored data. For example, use H.265 video encoding and gzip data compression methods to reduce the data volume while ensuring the quality of the restored data.

[0050] Step 2: Perform dynamic task scheduling in the cloud based on the obtained data:

[0051] Step 21: Considering the real-time requirements of tasks, data importance, and computing resource consumption factors, construct a decision-making model for multi-objective optimized QoS, and use the decision-making model to intelligently allocate tasks between the cloud and the edge.

[0052] Use machine learning algorithms to establish a prediction model, use the prediction model to predict the network status and edge node load in the next period of time, and dynamically adjust task allocation in combination with the current cloud system status.

[0053] Use load balancing algorithms to evenly distribute tasks among multiple edge nodes to avoid single-point overload, and at the same time schedule cloud resources to intervene to handle sudden traffic.

[0054] Step 22: Establish an edge computing resource pool to uniformly manage and schedule edge computing resources. At the same time, use security protocols to encrypt data during data transmission, and use access control and authentication methods to perform user identity verification and authorization management.

[0055] Among them, the unified management and scheduling of edge computing resources by using the edge computing resource pool includes:

[0056] Perform resource virtualization management: use Docker containerization methods to virtualize edge computing resources for easy rapid deployment and elastic expansion of resources.

[0057] Perform resource status monitoring: deploy an edge monitoring system to collect the usage of CPU, memory, and storage resources in real time to provide accurate data support for resource scheduling.

[0058] Perform resource allocation: match appropriate edge resources according to task requirements, and at the same time consider resource reservation strategies to cope with future demand fluctuations.

[0059] When performing security and privacy protection, use the TLS / SSL protocol to encrypt data during data transmission, and use a key management system between the edge node and the cloud to achieve end-to-end encryption of data.

[0060] Implement a strict access control policy, and use technologies such as OAuth and JWT to implement user authentication and authorization management to ensure that only authorized users can access data and resources.

[0061] Also adopt differential privacy and homomorphic encryption methods to protect personal privacy information from being leaked while not affecting the data analysis results.

[0062] Embodiment 2

[0063] The present invention also provides a data processing system based on cloud-edge collaboration, including a data preprocessing module, a scheduling module, and a security management module.

[0064] The data preprocessing module performs data preprocessing at the edge node based on the cloud-edge collaboration architecture: performs multi-level filtering, feature extraction, and compression and encoding on the data, and uploads the preprocessed data to the cloud.

[0065] The scheduling module performs dynamic task scheduling in the cloud based on the acquired data:

[0066] The scheduling module comprehensively considers the real-time requirements of tasks, data importance, and computing resource consumption factors, constructs a decision-making model for multi-objective optimized QoS, and uses the decision-making model to intelligently allocate tasks between the cloud and the edge.

[0067] Establish a prediction model using machine learning algorithms, use the prediction model to predict the network status and edge node load in the next period of time, and dynamically adjust task allocation in combination with the current cloud system status.

[0068] Use the load balancing algorithm to evenly distribute tasks among multiple edge nodes to avoid single-point overload, and at the same time schedule cloud resources to intervene to cope with sudden traffic.

[0069] The scheduling module establishes an edge computing resource pool, uses the edge computing resource pool to uniformly manage and schedule edge computing resources, and at the same time the security management module encrypts data using a security protocol during data transmission, and performs user authentication and authorization management using access control and authentication methods.

[0070] For the content of information interaction and execution process between the above modules in the system, since it is based on the same concept as the method embodiment of the present invention, the specific content can be referred to the description in the method embodiment of the present invention, and will not be elaborated here.

[0071] Similarly, the advantages of the system of the present invention are:

[0072] 1. Real-time and low-latency optimization: Cloud-edge collaboration significantly reduces response time by performing data processing and analysis at edge nodes close to the data source, making it suitable for application scenarios with strict real-time requirements such as autonomous driving and remote surgery. Technological innovations focus on efficient data transmission protocols, edge caching strategies, and dynamic task scheduling algorithms.

[0073] 2. Data security and privacy protection: Edge computing can process sensitive data at the source, reducing data upload and thus lowering the risk of leakage. Combining technologies such as blockchain and homomorphic encryption, cloud-edge collaboration systems can further enhance data security and privacy protection.

[0074] 3. Resource optimization and energy consumption management: Cloud-edge collaboration requires an efficient resource allocation mechanism to ensure the rational use of computing resources. By using machine learning algorithms to predict future loads, cloud-edge resource allocation can be dynamically adjusted while considering energy consumption issues to achieve green computing. Patents may involve resource scheduling algorithms, energy efficiency ratio optimization models, etc.

[0075] 4. Intelligent application and service innovation: In scenarios such as smart cities and industrial Internet, cloud-edge collaboration supports the implementation of a large number of intelligent applications, such as intelligent monitoring and predictive maintenance. These applications rely on the efficient data processing capabilities of cloud-edge collaboration and algorithm optimizations for specific scenarios.

[0076] It should be noted that not all steps and modules in the above processes and system structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted as needed. The system structures described in the above embodiments can be physical structures or logical structures, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities separately, or some components in multiple independent devices may be jointly implemented.

[0077] The above-described embodiments are merely preferred embodiments given to fully illustrate the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or transformations made by those skilled in the art on the basis of the present invention are all within the protection scope of the present invention. The protection scope of the present invention is subject to the claims.

Claims

1. A data processing method based on cloud-edge collaboration, characterized in that include: Step 1: Based on the cloud-edge collaborative architecture, data preprocessing is performed at the edge node: multi-level filtering, feature extraction, compression and encoding of the data. Upload the preprocessed data to the cloud. Step 2: Dynamically schedule tasks based on the acquired data in the cloud: Step 21: Consider the real-time requirements of the task, the importance of data, and the consumption of computing resources, and build a multi-objective optimization QoS decision model. Use the decision model to intelligently allocate tasks between the cloud and the edge. Use machine learning algorithms to build prediction models, use the prediction models to predict the network status and edge node load in the future, and dynamically adjust task allocation based on the current cloud system status. Use load balancing algorithms to evenly distribute tasks among multiple edge nodes to avoid single-point overload, and schedule cloud resources to intervene to cope with sudden traffic; Step 22: Establish an edge computing resource pool, and use the edge computing resource pool to uniformly manage and schedule edge computing resources. At the same time, use security protocols to encrypt data during data transmission, and use access control and authentication methods to perform user identity authentication and authorization management.

2. According to the data processing method based on cloud-edge collaboration according to claim 1, it is characterized by In step 1, the data is filtered at multiple levels, including: first, the original data is initially screened through the rule engine to remove irrelevant or redundant information; second, statistical analysis methods are used to identify outliers to avoid invalid data being uploaded to the cloud. Extract features from data, including: performing preliminary feature extraction through edge nodes for specific application scenarios, Compress and encode data: Use data compression algorithms to reduce data volume while ensuring data quality after recovery.

3. According to the data processing method based on cloud-edge collaboration according to claim 1, it is characterized by In step 22, the edge computing resource pool is used to uniformly manage and schedule edge computing resources, including: Virtualize resource management: Use containerization to virtualize edge computing resources, which facilitates rapid deployment and elastic expansion of resources. Monitor resource status: deploy edge monitoring systems to collect real-time CPU, memory, and storage resource usage to provide accurate data support for resource scheduling. Allocate resources: Match appropriate edge resources according to task requirements, and consider resource reservation strategies to cope with future demand fluctuations.

4. According to the data processing method based on cloud-edge collaboration according to claim 1, it is characterized by In step 22, the TLS / SSL protocol is used to encrypt data during data transmission, and a key management system is used between the edge node and the cloud to achieve end-to-end encryption of data. Differential privacy and homomorphic encryption methods are also used to protect personal privacy information from being leaked without affecting the data analysis results.

5. A data processing system based on cloud-edge collaboration, characterized by Including data preprocessing module, scheduling module and security management module, The data preprocessing module is based on the cloud-edge collaborative architecture and performs data preprocessing at the edge node: multi-level filtering, feature extraction, compression and encoding of the data, and uploading the preprocessed data to the cloud. The scheduling module performs dynamic task scheduling in the cloud based on the acquired data: The scheduling module comprehensively considers the real-time requirements of the task, the importance of data, and the consumption of computing resources, and builds a multi-objective optimized QoS decision model. The decision model is used to intelligently allocate tasks between the cloud and the edge. Use machine learning algorithms to build prediction models, use the prediction models to predict the network status and edge node load in the future, and dynamically adjust task allocation based on the current cloud system status. Use load balancing algorithms to evenly distribute tasks among multiple edge nodes to avoid single-point overload, and schedule cloud resources to intervene to cope with sudden traffic; The scheduling module establishes an edge computing resource pool and uses the edge computing resource pool to uniformly manage and schedule edge computing resources. At the same time, the security management module uses security protocols to encrypt data during data transmission, and uses access control and authentication methods to perform user identity authentication and authorization management.

6. The data processing system based on cloud-edge collaboration according to claim 5 is characterized in that The data preprocessing module performs multi-level filtering on the data, including: firstly, preliminary screening of the original data through the rule engine to eliminate irrelevant or redundant information; secondly, using statistical analysis methods to identify outliers to avoid invalid data being uploaded to the cloud. Extract features from data, including: performing preliminary feature extraction through edge nodes for specific application scenarios, Compress and encode data: Use data compression algorithms to reduce data volume while ensuring data quality after recovery.

7. The data processing system based on cloud-edge collaboration according to claim 5 is characterized by: The scheduling module uses the edge computing resource pool to uniformly manage and schedule edge computing resources, including: Virtualize resource management: Use containerization to virtualize edge computing resources, which facilitates rapid deployment and elastic expansion of resources. Monitor resource status: deploy edge monitoring systems to collect real-time CPU, memory, and storage resource usage to provide accurate data support for resource scheduling. Allocate resources: Match appropriate edge resources according to task requirements, and consider resource reservation strategies to cope with future demand fluctuations.

8. According to claim 5, a data processing system based on cloud-edge collaboration is characterized in that the security management module uses TLS / SSL protocol to encrypt data during data transmission, and uses a key management system between the edge node and the cloud to achieve end-to-end encryption of data. It also uses differential privacy and homomorphic encryption methods to protect personal privacy information from being leaked without affecting the data analysis results.

Images