A data processing method and system based on cloud computing

Abstract

The present invention discloses a data processing method and system based on cloud computing, which relates to the field of data processing technology, including preprocessing data and uploading it to a cloud computing platform; analyzing the amount of cloud computing resources required for data processing, and formulating a preliminary cloud computing resource allocation strategy; the cloud computing platform processes data according to the preliminary cloud computing resource allocation strategy and performs real-time monitoring and adjustment. The present invention uses a time series prediction model based on an LSTM model and a cloud computing resource allocation formula, combines a real-time monitoring system to dynamically adjust the amount of allocated cloud computing resources in real time, and retrains the LSTM model when the amount of allocated cloud computing resources has not been able to solve the shortage of cloud computing resources. This not only realizes dynamic resource allocation, improves the efficiency of data processing and resource use, but also improves the reliability and security of data, which is of great significance for improving the efficiency and flexibility of processing large-scale data sets.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data processing method and system based on cloud computing. Background Art

[0002] With the rapid development and widespread application of cloud computing technology, the demand for processing large-scale data is growing. In this context, it is crucial to efficiently manage and optimize cloud computing resources. Traditional resource allocation methods often rely on static resource allocation strategies, which may lead to different resource utilization efficiencies in different situations, thereby causing resource waste or insufficient resources and affecting data processing performance. Therefore, it is particularly important to develop a more flexible and intelligent data processing method. Summary of the invention

[0003] In view of the problems existing in the above-mentioned existing cloud computing-based data processing methods and systems, the present invention is proposed.

[0004] Therefore, the problem to be solved by the present invention is that traditional resource allocation methods often rely on static resource allocation strategies, which may lead to different resource utilization efficiencies in different situations, thereby causing resource waste or resource shortage problems.

[0005] To solve the above technical problems, the present invention provides the following technical solutions: a data processing method based on cloud computing, which includes preprocessing data; analyzing the amount of cloud computing resources required for data processing and formulating a preliminary cloud computing resource allocation strategy; the cloud computing platform processes data according to the preliminary cloud computing resource allocation strategy and performs real-time monitoring and adjustment; and designing a backup and disaster recovery plan after data processing is completed.

[0006] As a preferred solution of the cloud computing-based data processing method described in the present invention, the data preprocessing includes collecting the data to be processed, removing irrelevant and duplicate data, standardizing and formatting the data, performing preliminary data analysis and statistical data quantity, and uploading the data to the cloud computing platform after preprocessing.

[0007] As a preferred solution of the cloud computing-based data processing method of the present invention, the amount of cloud computing resources required for the analysis of data processing includes:

[0008] Determine the number of layers and units in each layer of the LSTM model, and determine the model input and output;

[0009] The processed data is divided into a training set and a test set, and the training set data is used to train the LSTM model;

[0010] Iterate the LSTM model to adjust model parameters to optimize model performance;

[0011] After each training cycle, the test set is used to evaluate the model performance.

[0012] If the verification difference does not change significantly after the training cycle, stop model training and obtain the trained LSTM model;

[0013] Use the LSTM model as a time series prediction model to predict resource demand in the future. The formula is shown below:

[0014]

[0015] in, is the predicted future resource volume, x t-n It is the historical resource data;

[0016] The processed data is input into the LSTM model, which automatically processes the input data and outputs the prediction results.

[0017] As a preferred solution of the cloud computing-based data processing method described in the present invention, the formulating of a preliminary cloud computing resource allocation strategy includes formulating a preliminary cloud computing resource allocation strategy according to the prediction results of the LSTM model. The preliminary cloud computing resource allocation strategy is: if the LSTM model prediction results show that the next required resource amount will remain stable, then maintain the existing cloud computing resource amount and continuously monitor it; if the LSTM model prediction results show that the next required resource amount will increase, then increase the allocated cloud computing resource amount; if the LSTM model prediction results show that the next required resource amount will decrease, then reduce the allocated cloud computing resource amount, and perform cloud computing resource allocation. The cloud computing resource allocation formula is:

[0018]

[0019] in, The nonlinear adjustment index γ is used to adjust the predicted future resource volume The influence of L makes it nonlinear, α is the load adjustment coefficient, β is the current load coefficient, L t is the current actual load.

[0020] As a preferred solution of the cloud computing-based data processing method of the present invention, wherein: the cloud computing platform processes data according to the preliminary cloud computing resource allocation strategy and performs real-time monitoring and adjustment, including processing data according to the preliminary cloud computing resource allocation strategy and using the Prometheus monitoring system to monitor the data processing process and the usage of the allocated cloud computing resources in real time, collecting cloud computing resource performance average utilization data on the cloud computing platform, the cloud computing resource performance average utilization data is the average value of all cloud computing resource performance utilizations, and sending the cloud computing resource performance average utilization data back to the Prometheus monitoring system for analysis:

[0021] If all allocated cloud computing resources are in use and all data are being processed, and the average utilization of cloud computing resource performance exceeds 70% within 5 minutes, the cloud computing resource allocation strategy does not need to be adjusted. The average utilization of cloud computing resource performance is continuously monitored and checked. If unprocessed data is found, the data processing rate is accelerated. After the data processing is completed, the cloud computing resources are released to provide more cloud computing resources for the unprocessed data.

[0022] If all the allocated cloud computing resources are in use but some data is still not processed, and the average utilization rate of cloud computing resource performance exceeds 70% for more than 5 minutes, it means that the allocated cloud computing resources are in short supply, and the cloud computing resource allocation strategy needs to be adjusted immediately. The cloud computing platform checks other allocated cloud computing resources. If there are allocated cloud computing resources that are not in use, this part of the cloud computing resources will be allocated to the part with unprocessed data first. If all other allocated cloud computing resources are in use, the cloud computing platform will re-analyze the required cloud computing resources according to the remaining unprocessed data, generate a cloud computing resource reallocation strategy to process the remaining data and mark the data and store it as historical data as a reference for updating the cloud computing resource allocation strategy. If the generated cloud computing resource reallocation strategy still fails to solve the problem of cloud computing resource shortage, an alarm will be sent to notify the administrator to process the data immediately. After completion, the processed data will be marked and stored as the latest historical data, and the training data of the LSTM model will be replaced with the latest historical data for retraining.

[0023] If some of the allocated cloud computing resources are not in use and all data are being processed, and the average utilization rate of cloud computing resource performance is continuously lower than 50% within 3 minutes, the cloud computing platform will check other allocated cloud computing resources. If there are some allocated cloud computing resources that have not been able to process all the data, the unused cloud computing resources will be allocated to this part first. If not, the resource recovery mechanism will be automatically triggered to perform component detection on the average utilization rate of cloud computing resource performance, and gradually recover cloud computing resources with a performance utilization rate lower than 30% until the average utilization rate of cloud computing resource performance is higher than 70%.

[0024] As a preferred solution of the cloud computing-based data processing method described in the present invention, the backup after data processing is completed includes, after the data processing is completed, determining the key data that needs to be backed up, including databases, important files and system configurations, and setting the backup frequency according to the importance and update frequency of the data.

[0025] As a preferred solution of the cloud computing-based data processing method described in the present invention, the disaster recovery plan design includes formulating recovery strategies for different types of disaster situations such as hardware failures and network attacks, regularly checking and updating backup data and recovery plans, selecting multiple data centers in different geographical locations for data backup, and deploying an automated backup system to perform regular automated backup of data.

[0026] A data processing system based on cloud computing, characterized by comprising a data preprocessing module, a cloud computing resource allocation strategy module, a real-time monitoring module, and a data backup and recovery module;

[0027] The data preprocessing module is used to preprocess the data and upload the preprocessed data to the cloud computing platform;

[0028] The cloud computing resource allocation strategy module is used to process data according to the preliminary cloud computing resource allocation strategy, analyze the resources required for data processing, and formulate a preliminary cloud computing resource allocation strategy;

[0029] The real-time monitoring module is used to automatically adjust resource allocation according to the real-time monitoring data processing process;

[0030] The data backup and recovery module is used to back up data and design a disaster recovery plan.

[0031] A computer device comprises: a memory and a processor; the memory stores a computer program, and is characterized in that: when the processor executes the computer program, the steps of a data processing method based on cloud computing are implemented.

[0032] A computer-readable storage medium stores a computer program, wherein the computer program implements the steps of a data processing method based on cloud computing when executed by a processor.

[0033] The beneficial effects of the present invention are as follows: the present invention uses a time series prediction model based on the LSTM model and a cloud computing resource allocation formula, combined with a real-time monitoring system to dynamically adjust the allocated cloud computing resource amount in real time and retrain the LSTM model when the allocated cloud computing resource amount has not been able to solve the shortage of cloud computing resources. This not only realizes dynamic resource allocation, improves the efficiency of data processing, ensures the high efficiency and real-time performance of data processing, but also improves the reliability and security of data, which is of great significance for improving the efficiency and flexibility of processing large-scale data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0035] Figure 1 A flowchart of a data processing method based on cloud computing is shown in FIG.

[0036] Figure 2 A schematic diagram of the structure of a data processing system based on cloud computing. DETAILED DESCRIPTION

[0037] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0038] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0039] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor is it a separate or selective embodiment that is mutually exclusive with other embodiments.

[0040] Example 1

[0041] Reference Figure 1, which is the first embodiment of the present invention, and provides a data processing method based on cloud computing. The data processing method based on cloud computing includes the following steps:

[0042] S1. Preprocess the data;

[0043] Specifically, data preprocessing includes collecting the data that needs to be processed, removing irrelevant and duplicate data, standardizing and formatting the data, performing preliminary data analysis and statistical analysis, and uploading the data to the cloud computing platform after preprocessing.

[0044] Use filters or queries to identify and remove data that is not related to the current analysis target. Use data processing tools to detect duplicate data and use commands to delete it. Perform consistency checks to ensure that the data is accurate and consistent after removing irrelevant and duplicate data. Perform basic statistical analysis on the data by calculating the number of records, the number of unique values, and the number of missing values. Count the total data volume and analyze the distribution of data in different categories. Statistical data refers to the mean, median, standard deviation, variance, maximum and minimum values ​​of statistical data, which can help us quickly understand the basic characteristics of the data. Based on statistical data, we can quickly identify problems in the data and serve as a reference for evaluating the performance of the LSTM model. After preprocessing, store the data in the storage service provided by the cloud platform. Use the graphical interface tools and command line tools provided by the cloud platform to upload data to ensure that the data uploaded to the cloud computing platform is accurate, clean and useful, thereby reducing the complexity and difficulty of subsequent processing links and improving processing speed and efficiency.

[0045] S2. Analyze the amount of cloud computing resources required for data processing and formulate a preliminary cloud computing resource allocation strategy;

[0046] Specifically, the amount of cloud computing resources required for analytical data processing includes:

[0047] Determine the number of layers and units in each layer of the LSTM model, and determine the model input and output;

[0048] The processed data is divided into a training set and a test set, and the training set data is used to train the LSTM model;

[0049] Iterate the LSTM model to adjust model parameters to optimize model performance;

[0050] After each training cycle, the test set is used to evaluate the model performance.

[0051] If the verification difference does not change significantly after the training cycle, stop model training and obtain the trained LSTM model;

[0052] Use the LSTM model as a time series prediction model to predict resource demand in the future. The formula is shown below:

[0053]

[0054] in, is the predicted future resource volume, x t-n It is the historical resource data;

[0055] The processed data is input into the LSTM model, and the model automatically processes the input data and outputs the prediction results.

[0056] The LSTM model is a special recurrent neural network used to process and predict time series data. It can accurately predict the future demand for cloud computing resources. Through the prediction results, cloud computing resources can be allocated more accurately, data processing efficiency can be improved, and changes in data processing needs can be quickly adapted to achieve timely resource adjustments. It can reduce processing delays and system downtime caused by insufficient resources. According to the complexity of the data and the prediction performance requirements of the LSTM model, a single-layer LSTM model is initially selected as the starting point, using 64 or 128 units. If the performance of the initial LSTM model is not good, more LSTM layers are added. The number of units in each layer is adjusted according to the training results to find the optimal model complexity. The input data is determined using the historical data series including timestamps and corresponding resource usage, and the output data is determined using the predicted resource demand in the future.

[0057] Further, formulating a preliminary cloud computing resource allocation strategy includes formulating a preliminary cloud computing resource allocation strategy according to the prediction result of the LSTM model, wherein the preliminary cloud computing resource allocation strategy is: if the prediction result of the LSTM model shows that the required resource amount will remain stable in the future, then the existing cloud computing resource amount is maintained and continuously monitored; if the prediction result of the LSTM model shows that the required resource amount will increase in the future, then the allocated cloud computing resource amount is increased; if the prediction result of the LSTM model shows that the required resource amount will decrease in the future, then the allocated cloud computing resource amount is reduced, and cloud computing resource allocation is performed, wherein the cloud computing resource allocation formula is:

[0058]

[0059] Among them, R alloc is the amount of cloud computing resources allocated, The nonlinear adjustment index γ is used to adjust the predicted future resource volume The influence of L makes it nonlinear, α is the load adjustment coefficient, β is the current load coefficient, L t is the current actual load.

[0060] It is obtained by analyzing historical load data. Both α and β can be obtained through experiments and historical data. t By reading the current resource usage, the application of nonlinear adjustment index and load regulation coefficient enables resource allocation to flexibly adapt to different load conditions. During high-demand periods, it ensures that sufficient resources are available, maintains system performance and response speed, and avoids service interruptions or performance degradation due to insufficient resources. This not only improves the accuracy and efficiency of resource allocation, but also improves the performance and cost-effectiveness of the entire cloud computing platform.

[0061] S3, the cloud computing platform processes data and performs real-time monitoring and adjustment according to the preliminary cloud computing resource allocation strategy;

[0062] Specifically, the cloud computing platform processes data according to the preliminary cloud computing resource allocation strategy and performs real-time monitoring and adjustment, including processing data according to the preliminary cloud computing resource allocation strategy and using the Prometheus monitoring system to monitor the data processing process and the usage of the allocated cloud computing resources in real time, collecting cloud computing resource performance average utilization data on the cloud computing platform, the cloud computing resource performance average utilization data is the average value of all cloud computing resource performance utilizations, and sending the cloud computing resource performance average utilization data back to the Prometheus monitoring system for analysis:

[0063] If all allocated cloud computing resources are in use and all data are being processed, and the average utilization of cloud computing resource performance exceeds 70% within 5 minutes, the cloud computing resource allocation strategy does not need to be adjusted. The average utilization of cloud computing resource performance is continuously monitored and checked. If unprocessed data is found, the data processing rate is accelerated. After the data processing is completed, the cloud computing resources are released to provide more cloud computing resources for the unprocessed data.

[0064] If all the allocated cloud computing resources are in use but some data is still not processed, and the average utilization rate of cloud computing resource performance exceeds 70% for more than 5 minutes, it means that the allocated cloud computing resources are in short supply, and the cloud computing resource allocation strategy needs to be adjusted immediately. The cloud computing platform checks other allocated cloud computing resources. If there are allocated cloud computing resources that are not in use, this part of the cloud computing resources will be allocated to the part with unprocessed data first. If all other allocated cloud computing resources are in use, the cloud computing platform will re-analyze the required cloud computing resources according to the remaining unprocessed data, generate a cloud computing resource reallocation strategy to process the remaining data and mark the data and store it as historical data as a reference for updating the cloud computing resource allocation strategy. If the generated cloud computing resource reallocation strategy still fails to solve the problem of cloud computing resource shortage, an alarm will be sent to notify the administrator to process the data immediately. After completion, the processed data will be marked and stored as the latest historical data, and the training data of the LSTM model will be replaced with the latest historical data for retraining.

[0065] If some of the allocated cloud computing resources are not in use and all data are being processed, and the average utilization rate of cloud computing resource performance is continuously lower than 50% within 3 minutes, the cloud computing platform will check other allocated cloud computing resources. If there are some allocated cloud computing resources that have not been able to process all the data, the unused cloud computing resources will be allocated to this part first. If not, the resource recovery mechanism will be automatically triggered to perform component detection on the average utilization rate of cloud computing resource performance, and gradually recover cloud computing resources with a performance utilization rate lower than 30% until the average utilization rate of cloud computing resource performance is higher than 70%.

[0066] The Prometheus monitoring system is an open source system monitoring and alerting toolkit designed to handle highly dynamic cloud service architectures and is suitable for monitoring in cloud environments. Performance utilization is calculated in real time by the Prometheus monitoring system. It is based on statistical data on the real-time usage of resources such as CPU, memory, storage, and network. By averaging the resource usage data over a certain period of time, a comprehensive performance utilization indicator is obtained. Within 70%, this range is a healthy resource utilization, with neither insufficient nor excessive resources. Resource utilization above 70% indicates that resources are saturated and may be insufficient. Resource utilization below 50% indicates excessive resources. 5 minutes and 3 minutes are short-term stability for quickly identifying and confirming resource utilization to ensure that resource utilization observation is not accidental. It should be noted that the above thresholds and time windows are based on actual operational experience and best practices in cloud computing resource management. They are designed to balance resource utilization efficiency, cost, and system performance. Use the Prometheus monitoring system to track resource utilization in real time, provide instant performance data, and automatically adjust cloud computing resources based on performance data to ensure that the cloud platform always has sufficient resources to cope with data processing tasks and effective resource utilization.

[0067] S4. After data processing is completed, backup and disaster recovery solutions are designed;

[0068] Specifically, backing up after data processing is completed includes, after data processing is completed, determining the key data that needs to be backed up, including databases, important files and system configurations, and setting the backup frequency according to the importance and update frequency of the data.

[0069] Compared with traditional backup methods, cloud computing platforms can provide more efficient and automated backup solutions to reduce the risk of data loss. Timely data backup can significantly reduce business interruption time caused by data loss. Regular backup can improve the overall reliability and user trust of cloud computing platforms. Setting the backup frequency based on the importance and update frequency of data can optimize the use and management of storage resources. In a cloud computing environment, the flexibility and scalability of backup strategies help to cope with changing business needs and data volumes.

[0070] Furthermore, the design of disaster recovery plans includes developing recovery strategies for different types of disaster situations such as hardware failures and network attacks, regularly checking and updating backup data and recovery plans, selecting multiple data centers in different geographical locations for data backup, and deploying automated backup systems to perform regular automated backup of data.

[0071] Disaster recovery solutions minimize the impact of potential disasters, ensure that organizations can continue to operate under any circumstances, protect critical data and assets, enable cloud computing platforms to effectively respond to emergencies such as hardware failures and cyber attacks, reduce the negative impact of these events on the business, and significantly enhance the robustness of data processing, ensuring that the business can continue to operate normally even in extreme cases of hardware failures and cyber attacks.

[0072] Example 2

[0073] Reference Figure 2 , which is the second embodiment of the present invention, and which is different from the previous embodiment, provides a data processing system based on cloud computing, including a data preprocessing module, a cloud computing resource allocation strategy module, a real-time monitoring module, and a data backup and recovery module;

[0074] The data preprocessing module is used to preprocess the data and upload the preprocessed data to the cloud computing platform;

[0075] The cloud computing resource allocation strategy module is used to process data according to the preliminary cloud computing resource allocation strategy, analyze the resources required for data processing, and formulate a preliminary cloud computing resource allocation strategy;

[0076] The real-time monitoring module is used to automatically adjust resource allocation according to the real-time monitoring data processing process;

[0077] The data backup and recovery module is used to back up data and design a disaster recovery plan.

[0078] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0079] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0080] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0081] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0082] Example 3

[0083] This is the third embodiment of the present invention. This embodiment is different from the previous two embodiments. In order to verify the beneficial effects of the method of the present invention, the method of the present invention is compared with the prior art to demonstrate and prove the comparison results. Table 1 shows.

[0084] Table 1: Comparison table of the method of the present invention and the prior art experiments

[0085]

[0086] In summary, the present invention uses the LSTM model and the cloud computing resource allocation formula to provide a more comprehensive and dynamic data processing solution, which significantly improves the data processing speed and resource utilization, while reducing long-term operating costs and automatic resource adjustment response time. Therefore, the present invention has shown significant advantages in improving the intelligence and automation of data processing.

[0087] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A data processing method based on cloud computing, Features: include, Preprocess the data; Analyze the amount of cloud computing resources required for data processing and formulate a preliminary cloud computing resource allocation strategy; The cloud computing platform processes data and performs real-time monitoring and adjustments based on the preliminary cloud computing resource allocation strategy; After data processing is completed, backup and disaster recovery solutions are designed; The data preprocessing includes collecting the data to be processed, removing irrelevant and duplicate data, standardizing and formatting the data, performing preliminary data analysis and statistical data quantity, and uploading the data to the cloud computing platform after preprocessing; The amount of cloud computing resources required for the analysis data processing includes: Determine the number of layers and units in each layer of the LSTM model, and determine the model input and output; The processed data is divided into a training set and a test set, and the training set data is used to train the LSTM model; Iterate the LSTM model to adjust model parameters to optimize model performance; After each training cycle, the test set is used to evaluate the model performance; If the verification difference does not change significantly after the training cycle, stop model training and obtain the trained LSTM model; Use the LSTM model as a time series prediction model to predict resource demand in the future. The formula is shown below: in, is the predicted future resource volume, x t-n It is the historical resource data; Input the processed data into the LSTM model, which automatically processes the input data and outputs the prediction results; The formulating of the preliminary cloud computing resource allocation strategy includes formulating the preliminary cloud computing resource allocation strategy according to the prediction result of the LSTM model. The preliminary cloud computing resource allocation strategy is: if the prediction result of the LSTM model shows that the required resource amount will remain stable in the future, then the existing cloud computing resource amount is maintained and continuously monitored; if the prediction result of the LSTM model shows that the required resource amount will increase in the future, then the allocated cloud computing resource amount is increased; if the prediction result of the LSTM model shows that the required resource amount will decrease in the future, then the allocated cloud computing resource amount is reduced, and cloud computing resource allocation is performed. The cloud computing resource allocation formula is: in, The nonlinear adjustment index γ is used to adjust the predicted future resource volume The influence of L makes it nonlinear, α is the load adjustment coefficient, β is the current load coefficient, L t is the current actual load; The cloud computing platform processes data according to the preliminary cloud computing resource allocation strategy and performs real-time monitoring and adjustment, including processing data according to the preliminary cloud computing resource allocation strategy and using the Prometheus monitoring system to monitor the data processing process and the usage of the allocated cloud computing resources in real time, collecting cloud computing resource performance average utilization data on the cloud computing platform, the cloud computing resource performance average utilization data is the average value of all cloud computing resource performance utilizations, and sending the cloud computing resource performance average utilization data back to the Prometheus monitoring system for analysis: If all allocated cloud computing resources are in use and all data are being processed, and the average utilization of cloud computing resource performance exceeds 70% within 5 minutes, the cloud computing resource allocation strategy does not need to be adjusted. The average utilization of cloud computing resource performance is continuously monitored and checked. If unprocessed data is found, the data processing rate is accelerated, and cloud computing resources are released after data processing is completed. If all the allocated cloud computing resources are in use but some data is still not processed, and the average utilization rate of cloud computing resource performance exceeds 70% for more than 5 minutes, it means that the allocated cloud computing resources are in short supply, and the cloud computing resource allocation strategy needs to be adjusted immediately. The cloud computing platform checks other allocated cloud computing resources. If there are allocated cloud computing resources that are not in use, this part of the cloud computing resources will be allocated to the part with unprocessed data first. If all other allocated cloud computing resources are in use, the cloud computing platform will re-analyze the required cloud computing resources according to the remaining unprocessed data, generate a cloud computing resource reallocation strategy to process the remaining data, and mark the data and store it as historical data as a reference for updating the cloud computing resource allocation strategy. If the generated cloud computing resource reallocation strategy still fails to solve the problem of cloud computing resource shortage, an alarm will be sent to notify the administrator to process the data immediately. After completion, the processed data will be marked and stored as the latest historical data, and the training data of the LSTM model will be replaced with the latest historical data for retraining. If some of the allocated cloud computing resources are not in use and all data are being processed, and the average utilization rate of cloud computing resource performance is continuously lower than 50% within 3 minutes, the cloud computing platform will check other allocated cloud computing resources. If there are some allocated cloud computing resources that have not been able to process all the data, the unused cloud computing resources will be allocated to this part first. If not, the resource recovery mechanism will be automatically triggered to perform component detection on the average utilization rate of cloud computing resource performance, and gradually recover cloud computing resources with a performance utilization rate lower than 30% until the average utilization rate of cloud computing resource performance is higher than 70%.

2. The data processing method based on cloud computing according to claim 1, Features: The backing up after data processing is completed includes storing the data in a database after data processing is completed, determining key data that needs to be backed up, including the database, important files and system configuration, and setting the backup frequency according to the importance and update frequency of the data.

3. The cloud computing-based data processing method according to claim 2, Features: The disaster recovery plan design includes formulating recovery strategies for different types of disaster situations such as hardware failures and network attacks, regularly checking and updating backup data and recovery plans, selecting multiple data centers in different geographical locations for data backup, and deploying an automated backup system to perform regular automated backup of data.

4. A data processing system based on cloud computing, It is characterized in that Including data preprocessing module, cloud computing resource allocation strategy module, real-time monitoring module, data backup and recovery module; The data preprocessing module is used to preprocess the data and upload the preprocessed data to the cloud computing platform; The cloud computing resource allocation strategy module is used to process data according to the preliminary cloud computing resource allocation strategy, analyze the resources required for data processing, and formulate a preliminary cloud computing resource allocation strategy; The real-time monitoring module is used to automatically adjust resource allocation according to the real-time monitoring data processing process; The data backup and recovery module is used to back up data and design a disaster recovery plan.

5. A computer device, include: Memory and processor; The memory stores a computer program, wherein the processor implements the steps of any one of claims 1 to 3 when executing the computer program.

6. A computer-readable storage medium having a computer program stored thereon, Features: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 3 are implemented.

Images