A data processing method and system based on the cloud

Abstract

The present invention belongs to the technical field of data processing, and provides a data processing method and system based on the cloud. The method includes: predicting the data processing load in a target period based on the stored cloud far historical data processing records and cloud near historical data processing records, and starting a resource scheduling optimization mode when the data processing load exceeds a preset percentage of the maximum data processing capacity; interacting with each terminal connected to the cloud to obtain the to-be-processed data information of each terminal; and obtaining the server attribute information of each second server; analyzing and processing each to-be-processed data information and each server attribute information by using a resource scheduling optimization model to obtain a resource scheduling optimization plan. The above solution of the present invention can cope with high-load situations in advance, effectively avoid low efficiency caused by insufficient data processing capacity, and greatly improve the timeliness of data processing.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a data processing method and system based on the cloud. Background Art

[0002] In today's digital age, the amount of data has shown an explosive growth trend. Traditional data processing methods are mostly based on local devices. With the continuous expansion of the data scale, local processing faces many challenges. On the one hand, local hardware resources such as computing power and storage capacity are limited, making it difficult to meet the rapid processing requirements of massive data, resulting in low data processing efficiency and inability to meet the requirements of real-time business scenarios. On the other hand, in order to improve local data processing capabilities, enterprises need to continuously invest a large amount of funds to upgrade hardware devices, which undoubtedly greatly increases the operating costs. In addition, the security and reliability of data are also difficult to be fully guaranteed. Once a local device fails, the risk of data loss is extremely high.

[0003] The rise of cloud computing technology has brought new solutions to data processing. The data processing method based on the cloud can utilize the powerful computing resources and massive storage capabilities of the cloud to achieve efficient data processing and storage. However, the current cloud data processing methods still have deficiencies in resource scheduling optimization, resulting in a large room for improvement in the cloud data processing efficiency, and urgent further improvement and perfection are needed. Summary of the Invention

[0004] In view of the above technical problems, the present invention provides a data processing method, system, electronic device, computer storage medium and computer program product based on the cloud.

[0005] The present invention discloses a data processing method based on the cloud, which is applied to a first server. The method includes the following steps:

[0006] Predict the data processing load in a target period based on the stored cloud far historical data processing records and cloud near historical data processing records. When the data processing load exceeds a preset percentage of the maximum data processing capacity, start the resource scheduling optimization mode;

[0007] Interact with each terminal connected to the cloud to obtain the to-be-processed data information of each terminal. The to-be-processed data information includes the data volume and data type of the to-be-processed data of each terminal predicted in the target period; and obtain the server attribute information of each second server;

[0008] A resource scheduling optimization model is used to analyze and process the information of each piece of data to be processed and the attribute information of each server to obtain a resource scheduling optimization plan. The resource scheduling optimization plan includes several second servers respectively associated with each terminal, and the second servers are used to process the data to be processed uploaded by each terminal during the target time period.

[0009] Optionally, predicting the data processing load in the target period based on the stored cloud-based long-term historical data processing records and cloud-based recent historical data processing records includes:

[0010] Obtaining identity information of each terminal accessing the cloud, retrieving corresponding cloud access records according to the identity information, and classifying each terminal into a first type terminal and a second type terminal according to the cloud access records; the first type terminal refers to a terminal whose number of cloud access records is higher than a preset number, and the second type terminal refers to a terminal whose number of cloud access records is not higher than a preset number;

[0011] Calculating a ratio of the number of the second type of terminals to the sum of the number of the second type of terminals and the first type of terminals, and determining the acquisition time of the cloud-based remote historical data processing record according to the ratio;

[0012] The cloud-based long-term historical data processing record is obtained according to the acquisition time, and the cloud-based short-term historical data processing record is obtained according to a preset time.

[0013] Optionally, the preset percentage is determined as follows:

[0014] Extracting a portion of the historical data processing records corresponding to the target period from the cloud-based remote historical data processing records, and obtaining a probability of occurrence of data processing tasks that are higher than a priority level threshold based on the portion of the historical data processing records;

[0015] The preset percentage is obtained according to the occurrence probability matching; wherein the preset percentage is positively correlated with the occurrence probability.

[0016] Optionally, the using a resource scheduling optimization model to analyze and process each of the to-be-processed data information and each of the server attribute information to obtain a resource scheduling optimization solution includes:

[0017] A first reserved number is obtained according to the occurrence probability matching, and second servers of the first reserved number are selected to be in a standby state;

[0018] The resource scheduling optimization model is used to analyze and process each of the to-be-processed data information and the server attribute information of each second server in a non-standby state to obtain the resource scheduling optimization solution.

[0019] Optionally, screening the second servers with the first reserved quantity being in a standby state includes:

[0020] Evaluating the effectiveness of the first server in generating the resource scheduling optimization plan to obtain an effectiveness evaluation value, and matching a regulation factor according to the effectiveness evaluation value; the regulation factor is a value greater than 1;

[0021] Using the regulation factor to adjust the first reserved quantity to a second reserved quantity, and screening the second servers with the second reserved quantity being in a standby state.

[0022] The present invention also discloses a monitoring system applied to a first server. The system includes a processing device and a storage device. The computer code stored in the storage device is called and executed by the processing device to implement the following steps:

[0023] Predicting the data processing load in a target period based on the stored cloud far historical data processing records and cloud near historical data processing records, and starting a resource scheduling optimization mode when the data processing load exceeds a preset percentage of the maximum data processing capacity;

[0024] Interacting with each terminal accessing the cloud to obtain the to-be-processed data information of each terminal. The to-be-processed data information includes the data volume and data type of the to-be-processed data of each terminal predicted in the target period; and, obtaining the server attribute information of each second server;

[0025] Analyzing and processing each of the to-be-processed data information and each of the server attribute information by using a resource scheduling optimization model to obtain a resource scheduling optimization plan. The resource scheduling optimization plan includes a plurality of second servers respectively associated with each terminal, and the second servers are used to process the to-be-processed data uploaded by each terminal in the target period.

[0026] The present invention also discloses an electronic device, including: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, and the processor executes the computer program to implement the method as described in any one of the foregoing.

[0027] The present invention also discloses a computer storage medium. The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method as described in any one of the foregoing.

[0028] The present invention also discloses a computer program product. The computer program product contains computer code, and when the computer code is executed by a processor of an electronic device, the method as described in any one of the foregoing is implemented.

[0029] The beneficial effects of the present invention are at least as follows:

[0030] In the above solution of the present invention, after interacting with the terminal to obtain the data information to be processed and mastering the attribute information of the second server, a precise resource scheduling plan is generated using the resource scheduling optimization model, which can realize the reasonable allocation of resources according to the data type, data volume, server hardware configuration and load status. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention, and thus should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0032] Figure 1 is a schematic flowchart of a cloud-based data processing method disclosed in an embodiment of the present invention;

[0033] Figure 2 is a schematic structural diagram of a cloud-based data processing system disclosed in an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0034] The following specific embodiments illustrate the implementation manners of the present application. Those skilled in the art can easily understand the other advantages and effects of the present application from the content disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present application without creative efforts belong to the scope protected by the present application.

[0035] In addition, the technical features involved in different implementation manners of the present application described below can be combined with each other as long as they do not conflict with each other.

[0036] For the above technical problems, as Figure 1 shown, an embodiment of the present invention discloses a cloud-based data processing method, which is applied to a first server, and the method includes the following steps:

[0037] S101, predicting the data processing load in the target period based on the stored cloud historical data processing records and cloud recent historical data processing records, and starting the resource scheduling optimization mode when the data processing load exceeds a preset percentage of the maximum data processing capacity.

[0038] The cloud includes multiple servers, and the first server is dedicated to resource scheduling optimization, while multiple second servers are used to specifically process the upload data of each terminal accessing the cloud. Two data processing modes are arranged in the first server, namely the normal mode and the resource scheduling optimization mode. In the normal mode, the first server allocates corresponding second servers to the data to be processed by each terminal according to a preset rule, and the preset rule is, for example, load balancing scheduling.

[0039] Meanwhile, the first server will refer to the data processing situations of the cloud in the past relatively long period (distant history, such as the past month or half a month) and recently (recent history, such as the most recent one or two days), such as information like the amount of data processed in different time periods and the time spent on processing. By adopting methods such as time series analysis and machine learning prediction models, based on these historical records, it estimates the possible data processing workload, that is, the data processing load, in the target time period (such as the next 10 minutes or half an hour).

[0040] Since each server has its own upper limit of data processing capacity and the data types it is good at processing. A percentage is preset. For example, when the predicted data processing load exceeds 80% of the maximum data processing capacity of the cloud (i.e., all servers), it indicates that according to the current resource configuration, the servers may not be able to efficiently complete the data processing tasks in the target time period. At this time, the first server automatically activates the resource scheduling optimization mode.

[0041] S102, interact with each terminal accessing the cloud to obtain the information of the data to be processed by each terminal. The information of the data to be processed includes the amount and data type of the data to be processed by each terminal in the target time period; and, obtain the server attribute information of each second server.

[0042] The first server actively communicates with each terminal device connected to the cloud, and through a preset protocol and interface, obtains the data content that each terminal is expected to upload to the cloud for processing in the target time period, that is, the information of the data to be processed. The information of the data to be processed is predicted by each terminal itself and includes the amount and data type of the data to be processed by each terminal in the target time period. The amount of the data to be processed clarifies the size of the data that each terminal is expected to upload, for example, in units of GB or TB; the data type indicates what category these data belong to, such as text data, image data, video data, or structured data, unstructured data, etc. Different data types have different requirements for server resources during processing. For example, the processing of image, video data, and unstructured data usually requires more computing resources. Therefore, accurately obtaining this information is crucial for subsequent reasonable resource allocation.

[0043] The first server also obtains the attributes of other second servers participating in data processing in the cloud, including the hardware configuration of the servers, such as the number of CPU cores, the memory size, and the storage capacity; it may also include the current load conditions of the servers, such as the proportion of occupied computing resources and the usage of storage resources.

[0044] S103. Use the resource scheduling optimization model to analyze and process each piece of the to-be-processed data information and each piece of the server attribute information, and obtain a resource scheduling optimization plan. The resource scheduling optimization plan includes several second servers respectively associated with each terminal, and the second servers are used to process the to-be-processed data uploaded by each terminal during the target period.

[0045] Take the to-be-processed data information (data volume, data type) of each terminal obtained in the previous step and the attribute information of each second server as inputs, and let the trained resource scheduling optimization model perform comprehensive analysis and processing on them. After the analysis and processing by the model, a detailed resource scheduling optimization plan is output. This plan includes which second server(s) the to-be-processed data of each terminal should be assigned to for processing.

[0046] Through this precise resource allocation, the overall data processing efficiency of the cloud can be maximally improved, ensuring that the data of each terminal can be efficiently and timely processed during the period when the cloud data processing load is relatively heavy.

[0047] It can be understood that the resource scheduling optimization model is preferably constructed by Transformer, and uses the actual data processing records in the cloud (corresponding to the data volume and data type actually uploaded by different terminals), and the corresponding overall data processing efficiency in the cloud (obtained by manual or machine evaluation) as labeled data to create training data, and uses these training data to train the resource scheduling optimization model. After training, the resource scheduling optimization model can analyze the optimal resource scheduling optimization plan for all terminals based on the processing efficiency of different data types on different servers, the current load balancing situation of each server, etc.

[0048] In the above solution of the present invention, after interacting with the terminal to obtain the to-be-processed data information and mastering the attribute information of the second server, the resource scheduling optimization model is used to generate a precise resource scheduling plan, which can realize the reasonable allocation of resources according to the data type, data volume, server hardware configuration, and load conditions.

[0049] Optionally, the predicting the data processing load during the target period based on the cloud's far historical data processing records and near historical data processing records stored includes:

[0050] Obtain the identity information of each terminal accessing the cloud, retrieve the corresponding cloud access records according to the identity information, and divide each terminal into a first type of terminal and a second type of terminal according to the cloud access records; the first type of terminal refers to the number of cloud access records being higher than a preset number, and the second type of terminal refers to the number of cloud access records being not higher than the preset number;

[0051] Calculate the ratio of the number of the second type of terminals to the sum of the number of the second type of terminals and the number of the first type of terminals, and determine the acquisition duration of the cloud remote historical data processing records according to the ratio;

[0052] Obtain the cloud remote historical data processing records according to the acquisition duration, and obtain the cloud recent historical data processing records according to a preset duration.

[0053] In this embodiment, the solution of the present invention is to use the historical records of the cloud for data processing to analyze the upload rules of the data to be processed, and then predict the possible data processing load in the target period. At the same time, to improve the prediction accuracy, the present invention also sets a dynamic adjustment for the data acquisition amounts of the cloud remote historical data processing records and the cloud recent historical data processing records. Specifically:

[0054] The first server first collects the identity information of each terminal connected to the cloud in the current period, which can be unique information such as device ID, user account, etc., for accurately identifying each terminal individual. Then, using the obtained terminal identity information, query the corresponding access records in the cloud database. These records detail the relevant information of the terminal accessing the cloud in the past (such as the past month, two months), including access time, access times, etc.

[0055] Preset a threshold for the number of access records, and define the terminals with the number of access records higher than the preset number as the first type of terminals. These terminals are stable users of the cloud, and their data upload requirements are relatively more regular and stable. While the terminals with the number of access records not higher than the preset number are classified as the second type of terminals. These terminals are new users of the cloud service, and their data upload requirements may have more uncertainties.

[0056] After the above division is completed, the number of terminals included in the first type of terminals and the second type of terminals is counted, and the proportion of the number of the second type of terminals in all terminals (that is, the sum of the number of the first type of terminals and the second type of terminals) is calculated. If the proportion of the second type of terminals is relatively high, it means that there are more low-frequency terminals with higher uncertainty. In order to more accurately capture the possible data processing patterns and rules, the acquisition duration of the far historical data will be correspondingly shortened, that is, the reference degree of the far historical data will be reduced, and the reference degree of the near historical data will be correspondingly increased. On the contrary, if the proportion of the second type of terminals is relatively low and the first type of terminals with high-frequency usage dominate, the data processing pattern is relatively more stable. The acquisition duration of the far historical data can be appropriately extended to more comprehensively capture the possible data processing patterns and rules, that is, the reference degree of the far historical data will be increased, and the reference degree of the near historical data will be correspondingly reduced. Among them, the acquisition duration of the near historical data is a fixed value, that is, the preset duration. The preset duration is, for example, the corresponding duration of the current time period, 1 hour, 2 hours, or a certain percentage of the current time period, such as 80% or 75%.

[0057] According to the acquisition duration determined above, query and extract the far historical data processing records within the corresponding time period from the historical data repository in the cloud. These records contain detailed information on the data processing of various types of terminals in the past period of time, such as the amount of data processed, the processing time, the processing results, etc. At the same time, obtain the near historical data processing records according to the preset fixed duration.

[0058] The near historical data can better reflect the current business trends and data processing characteristics. Combined with the far historical data, it provides comprehensive data support for accurately predicting the data processing load in the target time period. Through this targeted historical data acquisition method, it can better adapt to the data processing rules and characteristics of different types of terminals, improve the accuracy of data processing load prediction, and lay a foundation for the reasonable start and efficient operation of the subsequent resource scheduling optimization mode.

[0059] Optionally, the determination method of the preset percentage is as follows:

[0060] Obtain a partial historical data processing record corresponding to the target time period from the cloud far historical data processing record, and statistically obtain the occurrence probability of data processing tasks higher than the priority level threshold based on the partial historical data processing record;

[0061] Match the preset percentage according to the occurrence probability; wherein, the preset percentage has a positive correlation with the occurrence probability.

[0062] In this embodiment, according to the foregoing embodiment, when the predicted data processing load exceeds a preset percentage of the maximum data processing capacity of the cloud, it is determined that a more in-depth resource scheduling strategy needs to be adopted for the target period. However, there may be high-priority scheduling tasks during the target period, and the more in-depth resource scheduling strategy may cause these high-priority scheduling tasks not to be processed with the expected priority. In response to this, the present invention sets to dynamically allocate the above-mentioned preset percentage, that is, to adjust the startup difficulty of the resource scheduling optimization mode by adjusting the size of the preset percentage. Specifically as follows:

[0063] Since the cloud's long-term historical data processing records cover data with a long time span, in order to more accurately determine the preset percentage related to the current target period, it is first necessary to screen out the part corresponding to the target period from these massive historical records. For example, if the target period is from 9 am to 10 am every day, then extract the data processing records from 9 am to 10 am on each past date from the cloud's long-term historical data processing records.

[0064] In the intercepted historical data processing records related to the target period, further set a priority level threshold, which is used to distinguish the priority degree of data processing tasks (the priority level is represented by numbers 1-10, for example, the larger the number, the higher the corresponding priority level). For example, some data processing tasks may involve urgent business requirements, high-value customer data, etc., and are given a higher priority level. By traversing and analyzing some of the historical data processing records, count the proportion of data processing tasks higher than this priority level threshold among all data processing tasks (that is, all data processing tasks in some of the historical data processing records), and this proportion is the above-mentioned occurrence probability. For example, among the 100 extracted target period historical data processing records, 30 records correspond to tasks higher than the priority level threshold, then the probability is 30%.

[0065] Pre-establish a corresponding relationship between the preset percentage and the above-mentioned statistically obtained occurrence probability, and this relationship is positively correlated. When the occurrence probability of data processing tasks higher than the priority level threshold is higher, set the preset percentage to be higher. For example, a mapping rule may be preset in advance. When the occurrence probability is 10%, the preset percentage is 70%; when the occurrence probability is 15%, the preset percentage is 80%, etc.

[0066] If the probability of high-priority data processing tasks occurring during the target time period in history is relatively high, the corresponding preset percentage is set relatively large, which can increase the startup difficulty of the resource scheduling optimization mode to ensure that the high-priority data processing tasks occurring during the target time period can be processed preferentially, so as to ensure the overall preferential processing efficiency of high-priority data processing tasks. On the contrary, if the probability of high-priority data processing tasks occurring during the target time period in history is relatively low, the corresponding preset percentage is set relatively small, which can reduce the startup difficulty of the resource scheduling optimization mode to ensure the overall data processing efficiency during the target time period through the resource scheduling optimization mode.

[0067] By this way of determining the preset percentage based on the historical data characteristics, the startup of the resource scheduling optimization mode is more scientific and targeted, and can better adapt to the data processing requirements of the target time period, so as to achieve a balance between the overall data processing efficiency and the overall preferential processing efficiency of high-priority data processing tasks as much as possible.

[0068] Optionally, the using the resource scheduling optimization model to analyze and process each of the to-be-processed data information and each of the server attribute information to obtain a resource scheduling optimization plan includes:

[0069] Matching the first reserved quantity according to the occurrence probability, and screening that the second servers with the first reserved quantity are in a standby state;

[0070] Using the resource scheduling optimization model to analyze and process each of the to-be-processed data information and the server attribute information of each non-standby second server to obtain the resource scheduling optimization plan.

[0071] In this embodiment, even when the probability of high-priority data processing tasks occurring is relatively low, high-priority data processing tasks may still occur during the target time period. In response to this situation, the present invention is configured to reserve a part (i.e., the first reserved quantity) of the second servers for these possible high-priority data processing tasks, and these second servers are dedicated to the high-priority data processing tasks occurring during the target time period.

[0072] For the other non-reserved (i.e., non-standby) second servers, the resource scheduling optimization model analyzes and processes them in the foregoing manner to obtain the optimal resource scheduling optimization plan, that is, to determine the second server corresponding to each terminal. It can be understood that multiple terminals may be assigned the same second server, and a single terminal may also be assigned multiple second servers, which is not specifically limited.

[0073] Optionally, the screening that the second servers with the first reserved quantity are in a standby state includes:

[0074] Evaluate the effectiveness of the resource scheduling optimization plan generated by the first server to obtain an effectiveness evaluation value, and match a adjustment factor according to the effectiveness evaluation value; the adjustment factor is a value greater than 1.

[0075] Use the adjustment factor to adjust the first reserved quantity to a second reserved quantity, and filter out the second servers with the second reserved quantity in a standby state.

[0076] In this embodiment, in addition to the possible high-priority data processing tasks during the target period, there may also be some non-high-priority data processing tasks, and these non-high-priority data processing tasks come from some terminals newly connected during the target period.

[0077] The first server can periodically generate and update the resource scheduling optimization plan (not involving those high-priority data processing tasks) during the target period to adapt to the data processing requirements of the non-high-priority data processing tasks of these newly connected terminals. However, generating a resource scheduling optimization plan takes a certain amount of time. When the effectiveness of the first server in generating the resource scheduling optimization plan is insufficient, it may cause these newly added non-high-priority data processing tasks to not be allocated to the second server for a long time, resulting in data jams, untimely responses, and other situations.

[0078] In view of the above situation, the present invention is configured to first evaluate the effectiveness of the resource scheduling optimization plan generated by the first server to obtain an effectiveness evaluation value. This evaluation can be achieved based on factors such as the generation duration of the resource scheduling optimization plan, whether the data processing tasks allocated to each second server are balanced, and whether the data processing can be completed within the specified time. The specific evaluation rules are not limited.

[0079] At the same time, a corresponding relationship between the effectiveness evaluation value and the adjustment factor is established in advance, and the corresponding adjustment factor can be obtained by querying this corresponding relationship based on the evaluated effectiveness evaluation value. Since the adjustment factor is a value greater than 1, it can appropriately amplify and adjust the subsequent reserved quantity. It can be understood that if the effectiveness evaluation value is high, it means that the first server has a strong ability to generate the resource scheduling optimization plan. At this time, relatively few standby servers are needed to handle emergencies, and the matched adjustment factor is relatively small; on the contrary, if the effectiveness evaluation value is low, it indicates that the first server has deficiencies in generating the plan. To ensure the stability of the system, more standby servers are required, and the matched adjustment factor will be relatively large.

[0080] Among the numerous second servers in the cloud, select second servers equal in number to the second reserved quantity according to certain screening criteria (such as the load condition, hardware performance, running stability, etc. of the servers), and set them to the standby state. During the target period, these standby second servers do not participate in the data processing tasks of the currently connected terminals, but are in a standby state so that they can be put into use in a timely manner when high-priority data processing tasks or data processing requirements of other newly added terminals occur, ensuring the normal operation of the system and the smooth progress of data processing.

[0081] As Figure 2 shown, an embodiment of the present invention also discloses a data processing system based on the cloud, which is applied to a first server. The system includes a processing device and a storage device. The computer code stored in the storage device is called and executed by the processing device to implement the following steps:

[0082] Predict the data processing load during the target period based on the stored cloud far historical data processing records and cloud near historical data processing records. When the data processing load exceeds a preset percentage of the maximum data processing capacity, start the resource scheduling optimization mode;

[0083] Interact with each terminal connected to the cloud to obtain the to-be-processed data information of each terminal. The to-be-processed data information includes the data volume and data type of the to-be-processed data predicted for each terminal during the target period; and, obtain the server attribute information of each second server;

[0084] Use the resource scheduling optimization model to analyze and process each of the to-be-processed data information and each of the server attribute information to obtain a resource scheduling optimization plan. The resource scheduling optimization plan includes a number of second servers respectively associated with each terminal, and the second servers are used to process the to-be-processed data uploaded by each terminal during the target period.

[0085] An embodiment of the present invention also discloses an electronic device, including: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor. The processor executes the computer program to implement the method as described in the foregoing embodiment.

[0086] An embodiment of the present invention also discloses a computer storage medium. The computer storage medium stores a computer program, and the computer program is executed by a processor to implement the method as described in the foregoing embodiment.

[0087] An embodiment of the present invention also discloses a computer program product. The computer program product contains computer code, and when the computer code is executed by a processor of an electronic device, it implements the method as described in the foregoing embodiment.

[0088] The above-mentioned computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or apparatuses, or any suitable combination of the foregoing. Alternatively, the computer-readable storage medium may be a machine-readable signal medium. More specific examples of the machine-readable storage medium would include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0089] It should be understood that various forms of the processes shown above may be used, with steps reordered, added, or deleted. For example, the steps recited in the present invention may be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved, and no limitation is imposed herein.

[0090] The above specific embodiments do not constitute a limitation on the protection scope of the present invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A data processing method based on the cloud, applied to a first server, characterized in that: The method includes the following steps: Predict the data processing load during the target period based on the stored cloud far - historical data processing records and cloud near - historical data processing records. When the data processing load exceeds a preset percentage of the maximum data processing capacity, start the resource scheduling optimization mode; Interact with each terminal connected to the cloud to obtain the to - be - processed data information of each terminal. The to - be - processed data information includes the data volume and data type of the to - be - processed data of each terminal predicted during the target period; and, obtain the server attribute information of each second server; Use the resource scheduling optimization model to analyze and process each of the to - be - processed data information and each of the server attribute information to obtain a resource scheduling optimization plan. The resource scheduling optimization plan includes several second servers respectively associated with each terminal, and the second servers are used to process the to - be - processed data uploaded by each terminal during the target period; The predicting the data processing load during the target period based on the stored cloud far - historical data processing records and cloud near - historical data processing records includes: Obtain the identity information of each terminal connected to the cloud, retrieve the corresponding cloud access records according to the identity information, and divide each terminal into a first - type terminal and a second - type terminal according to the cloud access records; the first - type terminal refers to the number of cloud access records higher than a preset number, and the second - type terminal refers to the number of cloud access records not higher than the preset number; Calculate the ratio of the number of the second - type terminals to the sum of the number of the second - type terminals and the number of the first - type terminals, and determine the acquisition duration of the cloud far - historical data processing records according to the ratio; Obtain the cloud far - historical data processing records according to the acquisition duration, and obtain the cloud near - historical data processing records according to a preset duration.

2. The data processing method based on the cloud according to claim 1, wherein: The determination method of the preset percentage is: Extract a partial historical data processing record corresponding to the target period from the cloud far - historical data processing records, and statistically obtain the occurrence probability of data processing tasks higher than the priority level threshold based on the partial historical data processing records; Match the preset percentage according to the occurrence probability; wherein, the preset percentage has a positive correlation with the occurrence probability.

3. The data processing method based on the cloud according to claim 2, wherein: The using the resource scheduling optimization model to analyze and process each of the to - be - processed data information and each of the server attribute information to obtain a resource scheduling optimization plan includes: Match a first reserved quantity according to the occurrence probability, and screen out the second servers of the first reserved quantity in a standby state; Use the resource scheduling optimization model to analyze and process each of the to - be - processed data information and the server attribute information of each second server in a non - standby state to obtain the resource scheduling optimization plan.

4. A data processing method based on the cloud according to claim 3, characterized in that: The screening out the second servers of the first reserved quantity in a standby state includes: Evaluate the effectiveness of generating the resource scheduling optimization plan for the first server to obtain an effectiveness evaluation value, and match a adjustment factor according to the effectiveness evaluation value; the adjustment factor is a value greater than 1; Adjust the first reserved quantity to a second reserved quantity using the adjustment factor, and screen that the second servers with the second reserved quantity are in a standby state.

5. A cloud-based data processing system, applied to a first server, the system comprising a processing device and a storage device, characterized in that: The computer code stored in the storage device is called and executed by the processing device to implement the following steps: Predict the data processing load during a target period based on the stored cloud long-term historical data processing records and cloud short-term historical data processing records. When the data processing load exceeds a preset percentage of the maximum data processing capacity, start a resource scheduling optimization mode; Interact with each terminal accessing the cloud to obtain the data to be processed information of each terminal, where the data to be processed information includes the data volume and data type of the data to be processed by each terminal during the target period; and, obtain the server attribute information of each second server; Use a resource scheduling optimization model to analyze and process each of the data to be processed information and each of the server attribute information to obtain a resource scheduling optimization plan, where the resource scheduling optimization plan includes a number of second servers respectively associated with each terminal, and the second servers are used to process the data to be processed uploaded by each terminal during the target period; The predicting the data processing load during the target period based on the stored cloud long-term historical data processing records and cloud short-term historical data processing records includes: Obtain the identity information of each terminal accessing the cloud, retrieve the corresponding cloud access record according to the identity information, and divide each terminal into a first type of terminal and a second type of terminal according to the cloud access record; the first type of terminal refers to the number of cloud access records being higher than a preset number, and the second type of terminal refers to the number of cloud access records being not higher than a preset number; Calculate the ratio of the number of the second type of terminals to the sum of the number of the second type of terminals and the number of the first type of terminals, and determine the acquisition duration of the cloud long-term historical data processing records according to the ratio; Obtain the cloud long-term historical data processing records according to the acquisition duration, and obtain the cloud short-term historical data processing records according to a preset duration.

6. An electronic device, comprising: At least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, characterized in that: the processor executes the computer program to implement the method according to any one of claims 1-4.

7. A computer storage medium storing a computer program, characterized in that: The computer program is executed by a processor to implement the method according to any one of claims 1-4.

8. A computer program product, characterized in that: The computer program product contains computer code, and when the computer code is executed by a processor of an electronic device, it implements the method according to any one of claims 1-4.

Images