Data processing system and method of cloud computer data center
By setting monitoring time points and early warning time periods, short-term, medium-term and long-term data characteristics are obtained, emergency data processing thresholds are set, edge node traffic is cleaned, real-time data characteristics are monitored, resource replication requests are triggered, and resources are dynamically allocated, which solves the performance problems of cloud computer data centers when handling massive heterogeneous tasks, and improves the flexibility and overall performance of the system.
Patent Information
- Application Number
- CN202510445102.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-10
- Publication Date
- 2025-07-18
AI Technical Summary
Traditional static resource allocation and load balancing strategies are difficult to meet the dynamic needs of cloud computer data centers to handle massive heterogeneous tasks, and cannot cope with emergencies of large amounts of data, resulting in poor overall system performance.
By setting monitoring time points and early warning time periods, short-term, medium-term and long-term data processing characteristics are obtained, emergency data processing thresholds are set, resource allocation requests are triggered, edge node traffic is cleaned, real-time data processing characteristics are monitored, resource replication requests are triggered, and resources are dynamically allocated using multi-level data processing prediction methods.
It improves the overall system performance of cloud computer data centers in dealing with large amounts of data emergencies, and achieves flexible response to dynamic needs.
Smart Images
Figure CN120336019A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of cloud computer data centers, and particularly to a data processing system and method for a cloud computer data center. Background Art
[0002] Cloud computers can efficiently support various application requirements such as big data processing, high throughput rate, and high-security information services. Their computing power and storage capacity can be dynamically scaled and infinitely expanded. Cloud computers have a distributed new architecture, and multiple inexpensive computing resources perform parallel computing, greatly improving the computing speed and storage capacity of the infrastructure.
[0003] With the rapid development of cloud computing technology, data centers need to process a large amount of heterogeneous tasks, such as batch processing jobs, streaming data, AI training tasks, etc. Traditional static resource allocation and load balancing strategies are difficult to meet dynamic requirements. When dealing with a large amount of heterogeneous tasks, they cannot cope with sudden situations of processing a large amount of data, resulting in poor overall system performance. Summary of the Invention
[0004] The purpose of the present invention is to provide a data processing system and method for a cloud computer data center, aiming to solve the technical problem that traditional static resource allocation and load balancing strategies in the prior art are difficult to meet dynamic requirements, and cannot cope with sudden situations of processing a large amount of data when dealing with a large amount of heterogeneous tasks, resulting in poor overall system performance.
[0005] To achieve the above purpose, a data processing method for a cloud computer data center adopted by the present invention includes the following steps:
[0006] Set monitoring time points and early warning time periods, and respectively obtain short-term, medium-term, and long-term data processing characteristics before the monitoring time points;
[0007] Set a threshold for data processing in case of emergencies, trigger a resource allocation request according to the monitoring situation, and clean the traffic of edge nodes;
[0008] Monitor real-time data processing characteristics and trigger a resource leveling request.
[0009] Among them, in the step of setting monitoring time points and early warning time periods and respectively obtaining short-term, medium-term, and long-term data processing characteristics before the monitoring time points:
[0010] Set key monitoring time points, and divide the short-term early warning time period, medium-term early warning time period, and long-term early warning time period of the key monitoring time points;
[0011] Obtain the data processing characteristics of the short-term early warning time period of the key monitoring time points and output short-term processing values;
[0012] Obtain the data processing characteristics of the medium-term early warning time period at the key monitoring time point, and output the medium-term processing value;
[0013] Obtain the data processing characteristics of the long-term early warning time period at the key monitoring time point, and output the long-term processing value.
[0014] Wherein, after the step of obtaining the data processing characteristics of the long-term early warning time period at the key monitoring time point and outputting the long-term processing value:
[0015] Integrate the short-term, medium-term, and long-term processing values, and output the processed value pair.
[0016] Wherein, in the step of setting the data processing threshold for emergency situations, triggering a resource allocation request according to the monitoring situation, and cleaning the traffic of edge nodes:
[0017] Set a two-layer threshold for data processing in emergency situations; the two-layer threshold includes a historical statistical data processing threshold and a real-time data processing threshold;
[0018] Set multi-level early warnings and allocate resource allocation request data;
[0019] Clean the traffic of edge nodes and discard invalid data packets in real time.
[0020] Wherein, in the step of setting multi-level early warnings and allocating resource allocation request data:
[0021] The multi-level early warnings include light-level early warnings, medium-level early warnings, and heavy-level early warnings.
[0022] Wherein, in the step of setting multi-level early warnings and allocating resource allocation request data:
[0023] The resource allocation request for the light-level early warning is to trigger the expansion of the local cache;
[0024] The resource allocation request for the medium-level early warning is to trigger cross-node load balancing;
[0025] The resource allocation request for the heavy-level early warning is to trigger the elastic scaling of cloud services.
[0026] Wherein, in the step of monitoring the real-time data processing characteristics and triggering a resource leveling request:
[0027] Collect key metrics according to a time window and monitor the abnormal situation of the metric combination; the key metrics include processing delay and throughput;
[0028] Set a resource leveling request, and trigger a resource leveling request according to the key metrics and the situation of the metric combination.
[0029] The present invention also provides a data processing system for a cloud computer data center, including a feature monitoring module, a resource allocation trigger module, and a resource leveling trigger module, wherein:
[0030] The feature monitoring module is used to set the monitoring time point and the early warning time period, and respectively obtain the short-term, medium-term, and long-term data processing features before the monitoring time point;
[0031] The resource allocation trigger module is used to set the data processing threshold for emergencies, trigger a resource allocation request according to the monitoring situation, and clean the traffic of the edge nodes;
[0032] The resource leveling trigger module is used to monitor the real-time data processing features and trigger a resource leveling request.
[0033] A data processing system and method for a cloud computer data center according to the present invention uses a feature monitoring module, a resource allocation trigger module, and a resource leveling trigger module to perform the following steps: setting the monitoring time point and the early warning time period, and respectively obtaining the short-term, medium-term, and long-term data processing features before the monitoring time point; setting the data processing threshold for emergencies, triggering a resource allocation request according to the monitoring situation, and cleaning the traffic of the edge nodes; monitoring the real-time data processing features and triggering a resource leveling request; by performing a multi-level data processing prediction method, dynamically allocating resources, being able to handle emergencies of processing a large amount of data, and improving the overall performance of the system. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.
[0035] Figure 1 is a flowchart of the steps of the data processing method for the cloud computer data center of the present invention.
[0036] Figure 2 is a flowchart of the steps of S100 of the present invention.
[0037] Figure 3 is a flowchart of the steps of S200 of the present invention.
[0038] Figure 4 is a flowchart of the steps of S300 of the present invention.
[0039] Figure 5 is a schematic structural diagram of the data processing system for the cloud computer data center of the present invention.
[0040] Figure 6 is a schematic structural diagram of the electronic device of the present invention.
[0041] 401 - Feature Monitoring Module, 402 - Resource Allocation Trigger Module, 403 - Resource Leveling Trigger Module. Detailed Implementation Manner
[0042] Here, the exemplary embodiments will be described in detail, and the examples are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present application.
[0043] The terms used in the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application. The singular forms "a", "the", and "said" used in the present application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more of the associated listed items.
[0044] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "when" or "while" or "in response to determining".
[0045] Please refer to Figures 1 to 4 , the present invention provides a data processing method for a cloud computer data center, including the following steps:
[0046] S100: Set the monitoring time point and the warning time period, and respectively obtain the short-term, medium-term, and long-term data processing characteristics before the monitoring time point.
[0047] In this embodiment, the monitoring time point and the warning time period are set, and the short-term, medium-term, and long-term data processing characteristics before the monitoring time point are respectively obtained; the specific steps are as follows:
[0048] S101: Set the key monitoring time point, and divide the short-term warning time period, medium-term warning time period, and long-term warning time period of the key monitoring time point;
[0049] S102: Obtain the data processing characteristics of the short-term warning time period of the key monitoring time point, and output the short-term processing value;
[0050] S103: Obtain the data processing characteristics of the medium-term warning time period of the key monitoring time point, and output the medium-term processing value;
[0051] S104: Obtain the data processing characteristics of the long-term warning time period at the key monitoring time point, and output the long-term processing value;
[0052] S105: Integrate the short-term, medium-term, and long-term processing values, and output the processing value integration pair.
[0053] In the above process, set the key monitoring time point according to business requirements, such as a fixed time every day, a specific working day of each week, or a special event trigger node, and divide the short-term warning time period, medium-term warning time period, and long-term warning time period of the key monitoring time point. Among them, the short-term prediction time window is 15 minutes, the medium-term prediction time window is 2 hours, and the long-term prediction adopts cross-data center collaboration. Respectively obtain the data processing characteristics of the short-term, medium-term, and long-term warning time periods of the key monitoring time point, and output the short-term, medium-term, and long-term processing values. Finally, integrate the short-term, medium-term, and long-term processing values, and output the processing value integration pair.
[0054] S200: Set the data processing threshold for emergencies, trigger a resource allocation request according to the monitoring situation, and clean the traffic of the edge node.
[0055] In this embodiment, set the data processing threshold for emergencies, trigger a resource allocation request according to the monitoring situation, and clean the traffic of the edge node; the specific steps are as follows:
[0056] S201: Set the double-layer threshold for data processing of emergencies; the double-layer threshold includes the historical statistical data processing threshold and the real-time data processing threshold;
[0057] S202: Set multi-level warnings and allocate resource allocation request data;
[0058] S203: Clean the traffic of the edge node and discard invalid data packets in real time.
[0059] In the above process, first set the double-layer threshold for data processing of emergencies; the double-layer threshold includes the historical statistical data processing threshold and the real-time data processing threshold; then set multi-level warnings and allocate resource allocation request data; the multi-level warnings include light-level warnings, medium-level warnings, and heavy-level warnings; the resource allocation request for the light-level warning is to trigger the expansion of the local cache; the resource allocation request for the medium-level warning is to trigger cross-node load balancing; the resource allocation request for the heavy-level warning is to trigger the elastic scaling of cloud services; finally, clean the traffic of the edge node and discard invalid data packets in real time.
[0060] S300: Monitor the real-time data processing characteristics and trigger a resource leveling request.
[0061] In this embodiment, monitor the real-time data processing characteristics and trigger a resource leveling request; the specific steps are as follows:
[0062] S301: Collect key metrics within a time window and monitor anomalies in metric combinations; the key metrics include processing latency and throughput.
[0063] S302: Set resource leveling requests and trigger resource leveling requests based on key metrics and metric combination conditions.
[0064] In the above process, collect key metrics within a time window and monitor anomalies in metric combinations.
[0065] Key metrics: Collect key metrics such as processing latency and throughput within a time window.
[0066] Metric combination conditions: Detect anomalies in metric combinations through the Esper engine, such as high latency + low throughput.
[0067] Trigger resource leveling requests based on key metrics and metric combination conditions, where the resource leveling requests are as follows:
[0068] Priority queue: Allocate dedicated computing resources for critical services;
[0069] Containerized scheduling: Automatically migrate low-priority tasks to idle nodes using Kubernetes;
[0070] Hybrid deployment optimization: Reserve a buffer resource pool at edge nodes to handle instantaneous peaks.
[0071] In the present invention, first set the monitoring time points and warning time periods, and respectively obtain the short-term, medium-term, and long-term data processing characteristics before the monitoring time points; then set the data processing threshold for emergencies, trigger resource allocation requests according to the monitoring situation, and clean the traffic at edge nodes; then monitor the real-time data processing characteristics and trigger resource leveling requests; through a multi-level data processing prediction method, dynamically allocate resources, which can handle emergencies of processing a large amount of data and improve the overall performance of the system.
[0072] Corresponding to the embodiment of the data processing method for a cloud computer data center described above, the present application also provides an embodiment of a data processing system for a cloud computer data center.
[0073] Figure 5 is a block diagram of a data processing system for a cloud computer data center shown according to an exemplary embodiment. Refer to Figure 5 , the system may include: a feature monitoring module 401, a resource allocation trigger module 402, and a resource leveling trigger module 403, where:
[0074] The feature monitoring module 401 is configured to set monitoring time points and warning time periods, and respectively obtain the short-term, medium-term, and long-term data processing characteristics before the monitoring time points;
[0075] The resource allocation trigger module 402 is configured to set a threshold for handling emergency situation data, trigger a resource allocation request according to the monitoring situation, and clean the traffic of edge nodes.
[0076] The resource leveling trigger module 403 is configured to monitor the characteristics of real-time data processing and trigger a resource leveling request.
[0077] In this embodiment, the feature monitoring module 401 sets monitoring time points and warning time periods, and respectively obtains the short-term, medium-term, and long-term data processing characteristics before the monitoring time points; the resource allocation trigger module 402 sets a threshold for handling emergency situation data, triggers a resource allocation request according to the monitoring situation, and cleans the traffic of edge nodes; the resource leveling trigger module 403 monitors the characteristics of real-time data processing and triggers a resource leveling request. First, set the monitoring time points and warning time periods, and respectively obtain the short-term, medium-term, and long-term data processing characteristics before the monitoring time points; then set the threshold for handling emergency situation data, trigger a resource allocation request according to the monitoring situation, and clean the traffic of edge nodes; then monitor the characteristics of real-time data processing and trigger a resource leveling request. By adopting a multi-level data processing prediction method to dynamically allocate resources, it is possible to handle emergencies of processing a large amount of data and improve the overall performance of the system.
[0078] Regarding the system in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.
[0079] For the system embodiments, since they basically correspond to the method embodiments, reference can be made to the partial descriptions of the method embodiments for the relevant parts. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this application. Those of ordinary skill in the art can understand and implement it without creative efforts.
[0080] Correspondingly, the present application further provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method of the cloud computer data center as described above. As Figure 6 shown, it is a hardware structure diagram of any device with data processing capabilities where the data processing system of the cloud computer data center provided by the embodiment of the present invention is located. Except for Figure 6In addition to the processor, memory, and network interface shown, any device with data processing capabilities where the device in the embodiment is located may generally include other hardware according to the actual functions of the device with data processing capabilities, which will not be elaborated here.
[0081] Correspondingly, the present application also provides a computer-readable storage medium, on which computer instructions are stored. When the instructions are executed by a processor, the data processing method of the cloud computer data center as described above is implemented. The computer-readable storage medium may be an internal storage unit of any device with data processing capabilities described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium may also be an external storage device, such as a plug-in hard disk, a Smart Media Card (SMC), an SD card, a Flash Card, etc. equipped on the device. Further, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capabilities and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capabilities, and may also be used to temporarily store data that has been output or is to be output.
[0082] After considering the specification and practicing the content disclosed herein, those skilled in the art will readily think of other implementation manners of the present application. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include known common knowledge or conventional technical means in the technical field not disclosed in the present application.
[0083] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and various modifications and changes can be made without departing from its scope.
Claims
1. A data processing method for a cloud computer data center, characterized in that, It includes the following steps: Set the monitoring time points and warning time periods, and respectively obtain the short-term, medium-term, and long-term data processing characteristics before the monitoring time points; Set the data processing threshold for emergencies, trigger a resource allocation request according to the monitoring situation, and clean the traffic of the edge nodes; Monitor the real-time data processing characteristics and trigger a resource leveling request.
2. The data processing method of the cloud computer data center according to claim 1, characterized in that In the step of setting the monitoring time points and warning time periods, and respectively obtaining the short-term, medium-term, and long-term data processing characteristics before the monitoring time points: Set the key monitoring time points, and divide the short-term warning time period, medium-term warning time period, and long-term warning time period of the key monitoring time points; Obtain the data processing characteristics of the short-term warning time period of the key monitoring time point, and output the short-term processing value; Obtain the data processing characteristics of the medium-term warning time period of the key monitoring time point, and output the medium-term processing value; Obtain the data processing characteristics of the long-term warning time period of the key monitoring time point, and output the long-term processing value.
3. The data processing method of the cloud computer data center according to claim 2, wherein After the step of obtaining the data processing characteristics of the long-term warning time period of the key monitoring time point and outputting the long-term processing value: Integrate the short-term, medium-term, and long-term processing values, and output the processing value integration pair.
4. The data processing method of the cloud computer data center according to claim 1, characterized in that In the step of setting the data processing threshold for emergencies, triggering a resource allocation request according to the monitoring situation, and cleaning the traffic of the edge nodes: Set the double-layer data processing threshold for emergencies; the double-layer threshold includes the historical statistical data processing threshold and the real-time data processing threshold; Set multi-level warnings and allocate resource allocation request data; Clean the traffic of the edge nodes and discard invalid data packets in real time.
5. The data processing method of the cloud computer data center according to claim 4, characterized in that, In the step of setting multi-level warnings and allocating resource allocation request data: The multi-level warnings include light-level warnings, medium-level warnings, and heavy-level warnings.
6. The data processing method of the cloud computer data center according to claim 5, characterized in that, In the step of setting multi-level warnings and allocating resource allocation request data: The resource allocation request for the light-level warning is to trigger the expansion of the local cache; The resource allocation request for the medium-level warning is to trigger cross-node load balancing; The resource allocation request for the heavy-level warning is to trigger the elastic scaling of cloud services.
7. The data processing method of the cloud computer data center according to claim 1, characterized in that, In the step of monitoring the real-time data processing characteristics and triggering a resource leveling request: Collect key indicators according to the time window and monitor the abnormal situation of the indicator combination; the key indicators include processing delay and throughput; Set the resource leveling request, and trigger the resource leveling request according to the key indicators and the indicator combination situation.
8. A data processing system for a cloud computer data center, applied to the data processing method of the cloud computer data center as described in claim 1, characterized in that, It includes a feature monitoring module, a resource allocation trigger module, and a resource leveling trigger module, where: The feature monitoring module is used to set the monitoring time points and warning time periods, and respectively obtain the short-term, medium-term, and long-term data processing characteristics before the monitoring time points; The resource allocation trigger module is used to set the data processing threshold for emergencies, trigger a resource allocation request according to the monitoring situation, and clean the traffic of the edge nodes; The resource leveling trigger module is used to monitor the real-time data processing characteristics and trigger a resource leveling request.