Data Processing Method Based on Cloud Computing Platform and Cloud Computing Platform

Abstract

The present invention relates to the field of data processing, and discloses a data processing method and a cloud computing platform based on a cloud computing platform, including: a data processing task parsing module obtains a task processing type according to a data processing task sent by a client, and according to the type of distributed data server required by the data processing task, a sequence of distributed data servers of the same type, matches the corresponding distributed data server according to the sequence of distributed data servers of the same type, and sends the data processing task to the matched distributed data server to complete the data processing based on the cloud computing platform. Through refined task parsing and intelligent server matching, the present invention ensures that tasks are executed by the most suitable server, significantly improving the data processing efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and specifically to a data processing method based on a cloud computing platform and a cloud computing platform. Background Art

[0002] With the rapid development of information technology, the amount of data has increased explosively, and the traditional single-machine processing method has been difficult to meet the requirements of efficient and fast data processing. As a new type of IT service model that integrates computing, storage, and network services, the cloud computing platform has become an ideal choice for processing large-scale data due to its characteristics such as elastic resource expansion, on-demand allocation, and high availability. However, when the cloud computing platform processes complex data processing tasks, it still faces many challenges:

[0003] Diversity of task types: Data processing tasks may involve single computing tasks (such as big data analysis, image processing, etc.) or combined tasks (that is, multiple interdependent tasks are executed in a specific order). When dealing with combined tasks, existing methods often lack a flexible task parsing and scheduling mechanism, resulting in low processing efficiency.

[0004] Unreasonable allocation of server resources: The cloud computing platform usually consists of various types of distributed data servers, including computing power data servers, storage data servers, and comprehensive data servers, etc. How to quickly and accurately match the most suitable server resources according to task requirements and avoid resource idleness or overload is an urgent problem to be solved currently.

[0005] Task scheduling delay: During the execution of combined tasks, data transfer and status synchronization between tasks are key factors affecting the overall processing efficiency. Unreasonable task scheduling strategies will increase data transmission delay and reduce processing speed.

[0006] System scalability and compatibility: As the scale of the cloud computing platform expands, how to ensure that the data processing method can efficiently adapt to newly added server resources and be compatible with different types of data processing tasks is an important consideration for improving the flexibility and scalability of the system. Summary of the Invention

[0007] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a data processing method based on a cloud computing platform, including the following steps:

[0008] Step 1, a data processing task parsing module obtains the task processing type according to the data processing task sent by the client. If it is a single task, it proceeds to Step 2; if it is a combined task, it proceeds to Step 3;

[0009] Step 2: According to the type of distributed data server required for the data processing task, the sequence of distributed data servers of the same type, match the corresponding distributed data server according to the sequence of distributed data servers of the same type, and send the data processing task to the matched distributed data server, then enter Step 7;

[0010] Step 3: The data processing task parsing module obtains the task processing order sequence according to the combined task, and obtains the sequence of distributed data server types according to the type of distributed data server required for each task in the task sequence;

[0011] Step 4: According to the sequence of distributed data server types, the distributed data server management module respectively obtains the sequence of distributed data servers corresponding to the type of distributed data server;

[0012] Step 5: According to the first task information in the task processing order sequence and the sequence of distributed data servers corresponding to the type of distributed data server, match to obtain the first distributed data server. According to the obtained first distributed data server and the sequence of distributed data servers corresponding to the type of distributed data server of the next task, match to obtain the distributed data server of the next task until the matching of the distributed data servers for all tasks is completed, obtaining the task processing distributed server sequence, and package the task processing distributed server sequence and the combined task and send them to the first distributed data server;

[0013] Step 6: After the first distributed data server finishes executing the task, send the task to the next distributed data server according to the task processing distributed server sequence until all tasks are completed;

[0014] Step 7: Complete the data processing based on the cloud computing platform.

[0015] Furthermore, the data processing task parsing module obtains the task processing type according to the data processing task sent by the client, including:

[0016] If the data processing task sent by the client is a single type of computing task, it is a single task; otherwise it is a combined task; the single type of computing task means that all tasks in the data processing task are of the same type of computing task.

[0017] Furthermore, the obtaining the sequence of distributed data servers of the same type according to the type of distributed data server required for the data processing task, and matching the corresponding distributed data server according to the sequence of distributed data servers of the same type, includes:

[0018] The distributed data server types for the described data processing task requirements include computing power data servers, storage data servers, and comprehensive data servers; obtaining a sequence of distributed data servers of the same type according to the distributed data server types for the data processing task requirements; obtaining the matching distributed data servers according to the set matching features and the sequence of distributed data servers; the set matching features include one or more of network latency, transmission bandwidth, and data processing speed.

[0019] Further, the data processing task parsing module obtains a task processing order sequence according to the combined task, and obtains a sequence of distributed data server types according to the distributed data server types required for each task in the task sequence, including:

[0020] The data processing task parsing module obtains each task and the task processing order in the combined task, obtains a task processing order sequence, and obtains a sequence of distributed data server types according to the distributed data server types required for each task in the task processing order sequence.

[0021] Further, matching to obtain the first distributed data server according to the first task information in the task processing order sequence and the sequence of distributed data servers of the corresponding distributed data server type, includes:

[0022] Obtaining the matching first distributed data server according to the matching features set for the first task, the first task information, and the sequence of distributed data servers of the corresponding distributed data server type.

[0023] Further, matching to obtain the distributed data server for the next task according to the obtained first distributed data server and the sequence of distributed data servers of the corresponding distributed data server type for the next task, includes:

[0024] According to the matching features set for the next task, respectively obtaining the degree of feature compliance of the matching features of each distributed data server in the sequence of distributed data servers of the corresponding distributed data server type between the first distributed data server and the next task; the distributed data server corresponding to the maximum value of the degree of feature compliance is the matching distributed data server for the next task;

[0025] The degree of feature compliance is: the feature difference between the matching features of the distributed data server in the sequence of distributed data servers of the corresponding distributed data server type between the first distributed data server and the next task and the matching features set for the task; the smaller the difference, the greater the degree of feature compliance.

[0026] Further, after the first distributed data server finishes executing a task, it sends the task to the next distributed data server according to the task processing distributed server sequence, including:

[0027] After the first distributed data server finishes executing the task, it packages and sends the first task processing result, the remaining tasks, and the task processing distributed server sequence to the next distributed data server.

[0028] A cloud computing platform, characterized in that it applies the data processing method based on the cloud computing platform, including a distributed data server, a distributed data server management module, a data processing task parsing module, and a communication module;

[0029] The distributed data server, the distributed data server management module, and the data processing task parsing module are respectively communicatively connected to the communication module.

[0030] The beneficial effects of the present invention are as follows: improving processing efficiency: through refined task parsing and intelligent server matching, it ensures that tasks are executed by the most suitable server, significantly improving data processing efficiency.

[0031] Reducing processing latency: optimizing the task scheduling mechanism, reducing the waiting time between tasks and data transmission time, and effectively reducing the overall processing latency.

[0032] Enhancing system flexibility: supporting the flexible processing of single tasks and combined tasks, and adapting to diverse data processing requirements.

[0033] Improving resource utilization: real-time monitoring of the server resource status, dynamically adjusting task allocation, avoiding resource idleness or overload, and improving resource utilization.

[0034] Enhancing system scalability: the design method is easy to expand, can quickly integrate newly added server resources, and adapt to the growth of the cloud computing platform scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic flowchart of the data processing method based on the cloud computing platform;

[0036] Figure 2 It is a schematic diagram of the cloud computing platform principle. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0037] The technical solution of the present invention will be further described in detail below with reference to the drawings, but the protection scope of the present invention is not limited to the following.

[0038] The features and performance of the present invention will be further described in detail below with reference to the embodiments.

[0039] Such as Figure 1As shown in the figure, the data processing method based on the cloud computing platform includes the following steps:

[0040] Step 1: The data processing task parsing module obtains the task processing type according to the data processing task sent by the client. If it is a single task, go to Step 2; if it is a combined task, go to Step 3.

[0041] Step 2: According to the distributed data server type required by the data processing task, the same type of distributed data server sequence, match the corresponding distributed data server according to the same type of distributed data server sequence, and send the data processing task to the matched distributed data server, and enter Step 7.

[0042] Step 3: The data processing task parsing module obtains the task processing order sequence according to the combined task, and obtains the distributed data server type sequence according to the distributed data server type required by each task in the task sequence.

[0043] Step 4: According to the distributed data server type sequence, the distributed data server management module respectively obtains the distributed data server sequences corresponding to the distributed data server types.

[0044] Step 5: According to the first task information in the task processing order sequence and the distributed data server sequence corresponding to the distributed data server type, match the first distributed data server. According to the obtained first distributed data server and the distributed data server sequence corresponding to the distributed data server type of the next task, match the distributed data server of the next task until all tasks are distributed. The data server matching is completed to obtain the task processing distributed server sequence, and the task processing distributed server sequence and the combined task are packaged and sent to the first distributed data server.

[0045] Step 6: After the first distributed data server finishes executing the task, send the task to the next distributed data server according to the task processing distributed server sequence until all tasks are completed.

[0046] Step 7: Complete the data processing based on the cloud computing platform.

[0047] The data processing task parsing module obtains the task processing type according to the data processing task sent by the client, including:

[0048] If the data processing task sent by the client is a single type of computing task, it is a single task; otherwise, it is a combined task; the single type of computing task means that all tasks in the data processing task are of the same type of computing task.

[0049] The type of distributed data server according to the data processing task requirements is obtained to get a sequence of distributed data servers of the same type, and the corresponding distributed data server is matched according to the sequence of distributed data servers of the same type, including:

[0050] The types of distributed data servers for the data processing task requirements include computing power data servers, storage data servers, and comprehensive data servers; a sequence of distributed data servers of the same type is obtained according to the types of distributed data servers for the data processing task requirements; a matched distributed data server is obtained according to the set matching features and the sequence of distributed data servers; the set matching features include one or more of network latency, transmission bandwidth, and data processing speed.

[0051] The data processing task parsing module obtains a task processing order sequence according to the combined task, and obtains a sequence of distributed data server types according to the types of distributed data servers required for each task in the task sequence, including:

[0052] The data processing task parsing module obtains each task and the task processing order in the combined task to get a task processing order sequence, and obtains a sequence of distributed data server types according to the types of distributed data servers required for each task in the task processing order sequence.

[0053] The first distributed data server is matched according to the first task information in the task processing order sequence and the sequence of distributed data servers of the corresponding distributed data server type, including:

[0054] According to the matching features set by the first task, the first task information, and the sequence of distributed data servers of the corresponding distributed data server type, the matched first distributed data server is obtained.

[0055] The distributed data server for the next task is matched according to the obtained first distributed data server and the sequence of distributed data servers of the corresponding distributed data server type for the next task, including:

[0056] According to the matching features set by the next task, the degree of feature compliance of the matching features of each distributed data server in the sequence of distributed data servers of the corresponding distributed data server type between the first distributed data server and the next task is obtained respectively. The distributed data server corresponding to the maximum value of the degree of feature compliance is the matched distributed data server for the next task.

[0057] The degree of feature compliance is as follows: The feature difference between the matching features of the distributed data server in the distributed data server sequence of the corresponding distributed data server type of the first distributed data server and the next task and the matching features set by the task. The smaller the difference, the greater the degree of feature compliance.

[0058] After the first distributed data server finishes executing the task, according to the task processing distributed server sequence, the task is sent to the next distributed data server, including:

[0059] After the first distributed data server finishes executing the task, it packs and sends the first task processing result, the remaining tasks, and the task processing distributed server sequence to the next distributed data server.

[0060] Such as Figure 2 As shown, the cloud computing platform applies the data processing method based on the cloud computing platform, including a distributed data server, a distributed data server management module, a data processing task parsing module, and a communication module;

[0061] The distributed data server, the distributed data server management module, and the data processing task parsing module are respectively communicatively connected to the communication module.

[0062] Specifically, the present invention provides a data processing method based on a cloud computing platform, including the following steps:

[0063] Step 1: Task type parsing

[0064] The data processing task parsing module receives the data processing task sent by the client, and by analyzing the task content, determines whether the task type is a single task or a combined task. A single task refers to a computing task in which all operations in the task are of the same type, such as only involving data analysis; a combined task contains multiple different types of sub-tasks or sub-tasks that need to be executed in sequence.

[0065] Step 2: Single task processing

[0066] If the task is a single task, the data processing task parsing module determines the required type of distributed data server according to the task requirements (such as a computing power data server, a storage data server, or a comprehensive data server).

[0067] The distributed data server management module selects the most suitable server from the server sequence of the same type according to the server type based on the set matching features (such as network latency, transmission bandwidth, data processing speed, etc.).

[0068] Send the data processing task to the matched distributed data server, and directly enter Step 7 after execution.

[0069] Step 3: Combined task parsing

[0070] For a combined task, the data processing task parsing module first parses the execution order of the tasks to form a task processing order sequence. According to the requirements of each subtask, the types of distributed data servers required are determined to form a distributed data server type sequence.

[0071] Step Four: Obtaining the server sequence

[0072] The distributed data server management module generates an available server sequence for each type of server according to the distributed data server type sequence.

[0073] Step Five: Task-server matching

[0074] According to the first task information in the task processing order sequence and its corresponding server type sequence, combined with the set matching features, the first distributed data server is selected.

[0075] For subsequent tasks, based on the matching features of the server in the current task execution server and the next task server type sequence (such as considering data transmission efficiency, processing speed, etc.), the execution server of each task is determined one by one to form a task processing distributed server sequence. The task processing distributed server sequence and the combined task are packaged and sent to the first distributed data server.

[0076] Step Six: Task scheduling and execution

[0077] After the first distributed data server completes its task, according to the task processing distributed server sequence, it passes the task processing result and the remaining task information to the next distributed data server and executes in turn until all tasks are completed.

[0078] Step Seven: Task completion

[0079] After all tasks are executed, the data processing based on the cloud computing platform is completed, and the processing result is returned to the client.

[0080] Embodiment 1: Cloud processing of a single data analysis task

[0081] An e-commerce company needs to perform real-time analysis on its massive user behavior data to gain insights into user purchase preferences and optimize the recommendation algorithm. This task is a single type of data analysis task, mainly involving the aggregation, statistics, and pattern recognition of a large amount of data.

[0082] Implementation steps:

[0083] Task type parsing: The e-commerce company submits a data analysis task to the cloud computing platform through the client. After receiving the task, the data processing task parsing module identifies that this task is a single task through parsing the task script, and the main requirement is data analysis.

[0084] Single-task processing: The data processing task parsing module determines that a high-computing power data server is required to execute this task according to the task requirements. The distributed data server management module monitors the status of all current computing power data servers, including CPU usage, available memory, network latency, etc., and generates a sequence of available servers.

[0085] An intelligent matching algorithm is used to comprehensively consider the processing speed and network latency, and the optimal server is selected from the sequence of computing power data servers.

[0086] Task execution and completion: The data analysis task is sent to the optimal computing power data server found for execution. After the server completes the data analysis, it directly returns the processing result to the client to complete the entire data processing process.

[0087] Embodiment 2: Cloud processing of combined data processing tasks

[0088] A research institution needs to preprocess, extract features, and train machine learning models on a large amount of scientific research data collected. These tasks need to be executed in sequence, and each step has different requirements for computing resources and storage resources.

[0089] Implementation steps:

[0090] Task type parsing: The research institution submits a combined data processing task to the cloud computing platform through the client. After receiving the task, the data processing task parsing module parses that the task includes three subtasks: data preprocessing, feature extraction, and model training, and they need to be executed in sequence.

[0091] Combined task parsing: The data processing task parsing module forms a task processing sequence: data preprocessing -> feature extraction -> model training. According to the requirements of each subtask, it is determined that data preprocessing requires a storage data server, feature extraction requires a computing power data server, and model training requires a comprehensive data server.

[0092] Server sequence acquisition: The distributed data server management module generates a sequence of storage data servers, a sequence of computing power data servers, and a sequence of comprehensive data servers respectively according to the server type requirements.

[0093] Task-server matching: According to the task processing sequence, first select the optimal storage data server for the data preprocessing task. Then, considering the data transmission efficiency and processing speed, select the optimal computing power data server for the feature extraction task. Finally, select the optimal comprehensive data server for the model training task to form a task processing distributed server sequence.

[0094] Task scheduling execution: The combined tasks and the sequence of distributed servers for task processing are packaged and sent to the first data storage server. After the data storage server completes data preprocessing, the results are passed to the computing power data server for feature extraction. After feature extraction is completed, the results are then passed to the comprehensive data server for model training. After model training is completed, the final processing results are returned to the client.

[0095] Task completion: All subtasks are successfully executed in sequence, and the research institution obtains a scientific research data analysis report after preprocessing, feature extraction, and model training.

Claims

1. A data processing method based on a cloud computing platform, characterized in that, It includes the following steps: Step 1: The data processing task parsing module obtains the task processing type according to the data processing task sent by the client. If it is a single task, go to Step 2; if it is a combined task, go to Step 3; if the data processing task sent by the client is a single type of computing task, it is a single task; otherwise, it is a combined task; the single type of computing task means that all tasks in the data processing task are of the same type of computing task; Step 2: According to the type of distributed data server required by the data processing task, obtain a sequence of distributed data servers of the same type. Match the corresponding distributed data server according to the sequence of distributed data servers of the same type, and send the data processing task to the matched distributed data server, and go to Step 7; Step 3: The data processing task parsing module obtains a task processing order sequence according to the combined task, and obtains a distributed data server type sequence according to the type of distributed data server required by each task in the task sequence; Step 4: According to the distributed data server type sequence, the distributed data server management module respectively obtains a sequence of distributed data servers of the corresponding distributed data server type; Step 5: According to the matching characteristics set by the first task, the first task information, and the sequence of distributed data servers of the corresponding distributed data server type, match the first distributed data server. According to the obtained first distributed data server and the sequence of distributed data servers of the corresponding distributed data server type of the next task, match the distributed data server of the next task until the matching of the distributed data servers of all tasks is completed, obtain a task processing distributed server sequence, and package and send the task processing distributed server sequence and the combined task to the first distributed data server; Step 6: After the first distributed data server finishes executing the task, send the task to the next distributed data server according to the task processing distributed server sequence until all tasks are completed; Step 7: Complete the data processing based on the cloud computing platform; The types of distributed data servers required by the data processing task include computing power data servers, storage data servers, and comprehensive data servers; obtain a sequence of distributed data servers of the same type according to the type of distributed data server required by the data processing task; obtain the matched distributed data server according to the set matching characteristics and the sequence of distributed data servers; the set matching characteristics include one or more of network latency, transmission bandwidth, and data processing speed.

2. The data processing method based on a cloud computing platform according to claim 1, wherein The data processing task parsing module obtains a task processing order sequence according to the combined task, and obtains a distributed data server type sequence according to the type of distributed data server required by each task in the task sequence, including: The data processing task parsing module obtains each task and the task processing order in the combined task, obtains a task processing order sequence, and obtains a distributed data server type sequence according to the type of distributed data server required by each task in the task processing order sequence.

3. The data processing method based on a cloud computing platform according to claim 2, characterized in that Matching the distributed data server for the next task according to the obtained first distributed data server and the sequence of distributed data servers of the corresponding distributed data server type for the next task includes: According to the matching features set for the next task, respectively obtaining the degree of feature compliance of the matching features of each distributed data server in the first distributed data server and the sequence of distributed data servers of the corresponding distributed data server type for the next task, wherein the distributed data server corresponding to the maximum value of the degree of feature compliance is the distributed data server for the next task to be matched; The degree of feature compliance is: the feature difference between the matching features of the distributed data server in the first distributed data server and the sequence of distributed data servers of the corresponding distributed data server type for the next task and the matching features set for the task. The smaller the difference, the greater the degree of feature compliance.

4. The data processing method based on a cloud computing platform according to claim 3, wherein After the first distributed data server finishes executing the task, sending the task to the next distributed data server according to the task processing distributed server sequence includes: After the first distributed data server finishes executing the task, it packages and sends the first task processing result, the remaining tasks, and the task processing distributed server sequence to the next distributed data server.

5. Cloud computing platform, characterized in that, Applying the data processing method based on the cloud computing platform according to any one of claims 1-4, including a distributed data server, a distributed data server management module, a data processing task parsing module, and a communication module; The distributed data server, the distributed data server management module, and the data processing task parsing module are respectively communicatively connected to the communication module.

Images