Cloud-based data processing method and system

Abstract

The invention belongs to the technical field of data processing, and provides a cloud-based data processing method and system. The method comprises the steps of predicting a data processing load of a target time period based on stored cloud far historical data processing records and cloud near historical data processing records, and starting a resource scheduling optimization mode when the data processing load exceeds a preset percentage of the maximum data processing capability; interacting with each terminal accessed to the cloud to obtain to-be-processed data information of each terminal; obtaining server attribute information of each second server; and analyzing and processing the to-be-processed data information and the server attribute information by using the resource scheduling optimization model to obtain a resource scheduling optimization scheme. According to the scheme, the high load condition can be dealt with in advance, low efficiency caused by insufficient data processing capacity is effectively avoided, and the timeliness of data processing is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a cloud-based data processing method and system. Background Art

[0002] In today's digital age, the amount of data is showing an explosive growth trend. Traditional data processing methods are mostly based on local devices. As the scale of data continues to expand, local processing faces many challenges. On the one hand, local hardware resources such as computing power and storage capacity are limited, and it is difficult to cope with the rapid processing needs of massive data, resulting in low data processing efficiency and failure to meet the requirements of real-time business scenarios. On the other hand, in order to improve local data processing capabilities, companies need to continuously invest a lot of money to upgrade hardware equipment, which undoubtedly greatly increases operating costs. In addition, the security and reliability of data are also difficult to be fully guaranteed. Once local equipment fails, the risk of data loss is extremely high.

[0003] The rise of cloud computing technology has brought new solutions to data processing. Cloud-based data processing methods can utilize the powerful computing resources and massive storage capabilities of the cloud to achieve efficient data processing and storage. However, current cloud data processing methods are still insufficient in terms of resource scheduling and optimization, resulting in a large room for improvement in cloud data processing efficiency, which urgently needs further improvement and perfection. Summary of the invention

[0004] In response to the above technical problems, the present invention provides a cloud-based data processing method, system, electronic device, computer storage medium and computer program product.

[0005] The present invention discloses a cloud-based data processing method, which is applied to a first server and comprises the following steps: Predicting the data processing load of the target period based on the stored cloud-based long-term historical data processing records and cloud-based recent historical data processing records, and starting a resource scheduling optimization mode when the data processing load exceeds a preset percentage of the maximum data processing capacity; Interacting with each terminal connected to the cloud to obtain the data information to be processed of each terminal, wherein the data information to be processed includes the predicted data volume and data type of the data to be processed of each terminal in the target time period; and obtaining the server attribute information of each second server; A resource scheduling optimization model is used to analyze and process the information of each piece of data to be processed and the attribute information of each server to obtain a resource scheduling optimization plan. The resource scheduling optimization plan includes several second servers respectively associated with each terminal, and the second servers are used to process the data to be processed uploaded by each terminal during the target time period.

[0006] Optionally, predicting the data processing load in the target period based on the stored cloud-based long-term historical data processing records and cloud-based recent historical data processing records includes: Obtaining identity information of each terminal accessing the cloud, retrieving corresponding cloud access records according to the identity information, and classifying each terminal into a first type terminal and a second type terminal according to the cloud access records; the first type terminal refers to a terminal whose number of cloud access records is higher than a preset number, and the second type terminal refers to a terminal whose number of cloud access records is not higher than a preset number; Calculating a ratio of the number of the second type of terminals to the sum of the number of the second type of terminals and the first type of terminals, and determining the acquisition time of the cloud-based remote historical data processing record according to the ratio; The cloud-based long-term historical data processing record is obtained according to the acquisition time, and the cloud-based short-term historical data processing record is obtained according to a preset time.

[0007] Optionally, the preset percentage is determined as follows: Extracting a portion of the historical data processing records corresponding to the target period from the cloud-based remote historical data processing records, and obtaining a probability of occurrence of data processing tasks that are higher than a priority level threshold based on the portion of the historical data processing records; The preset percentage is obtained according to the occurrence probability matching; wherein the preset percentage is positively correlated with the occurrence probability.

[0008] Optionally, the using a resource scheduling optimization model to analyze and process each of the to-be-processed data information and each of the server attribute information to obtain a resource scheduling optimization solution includes: A first reserved number is obtained according to the occurrence probability matching, and second servers of the first reserved number are selected to be in a standby state; The resource scheduling optimization model is used to analyze and process each of the to-be-processed data information and the server attribute information of each second server in a non-standby state to obtain the resource scheduling optimization solution.

[0009] Optionally, the step of screening the first reserved number of second servers to be in a standby state includes: Evaluate the performance of the resource scheduling optimization solution generated by the first server to obtain a performance evaluation value, and obtain an adjustment factor according to the performance evaluation value; the adjustment factor is a value greater than 1; The first reserved quantity is adjusted to a second reserved quantity using the adjustment factor, and second servers of the second reserved quantity are selected to be in a standby state.

[0010] The present invention also discloses a monitoring system, which is applied to a first server. The system includes a processing device and a storage device. The computer code stored in the storage device is called and executed by the processing device to implement the following steps: Predicting the data processing load of the target period based on the stored cloud-based long-term historical data processing records and cloud-based recent historical data processing records, and starting a resource scheduling optimization mode when the data processing load exceeds a preset percentage of the maximum data processing capacity; Interacting with each terminal connected to the cloud to obtain the data information to be processed of each terminal, wherein the data information to be processed includes the predicted data volume and data type of the data to be processed of each terminal in the target time period; and obtaining the server attribute information of each second server; A resource scheduling optimization model is used to analyze and process the information of each piece of data to be processed and the attribute information of each server to obtain a resource scheduling optimization plan. The resource scheduling optimization plan includes several second servers respectively associated with each terminal, and the second servers are used to process the data to be processed uploaded by each terminal during the target time period.

[0011] The present invention also discloses an electronic device, comprising: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement any of the above methods.

[0012] The present invention also discloses a computer storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement any of the above methods.

[0013] The present invention also discloses a computer program product, which includes computer codes. When the computer codes are executed by a processor of an electronic device, any of the above methods is implemented.

[0014] The beneficial effects of the present invention are at least: The above-mentioned scheme of the present invention, after interacting with the terminal to obtain the data information to be processed and mastering the attribute information of the second server, uses the resource scheduling optimization model to generate an accurate resource scheduling scheme, which can realize reasonable allocation of resources according to the data type, data volume, server hardware configuration and load conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 It is a flowchart of a cloud-based data processing method disclosed in an embodiment of the present invention; Figure 2 It is a structural diagram of a cloud-based data processing system disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following is a description of the implementation of the present application by specific specific embodiments. People familiar with the technology can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0018] In addition, the technical features involved in the different embodiments of the present application described below can be combined with each other as long as they do not conflict with each other.

[0019] In response to the above technical issues, such as Figure 1 As shown, an embodiment of the present invention discloses a cloud-based data processing method, which is applied to a first server, and the method includes the following steps: S101, predicting the data processing load of a target period based on stored cloud-based long-term historical data processing records and cloud-based recent historical data processing records, and starting a resource scheduling optimization mode when the data processing load exceeds a preset percentage of a maximum data processing capacity.

[0020] The cloud includes multiple servers, and the first server is dedicated to resource scheduling optimization, and multiple second servers are used to process the uploaded data of each terminal connected to the cloud in a targeted manner. The first server is arranged with two data processing modes, namely the normal mode and the resource scheduling optimization mode. In the normal mode, the first server allocates the corresponding second server for the data to be processed of each terminal according to the preset rules, and the preset rules are, for example, load balancing scheduling.

[0021] At the same time, the first server will refer to the data processing status of the cloud in the past (distant history, such as the past month or half a month) and in the recent (recent history, such as the last day or two days), such as the amount of data processed in different time periods, the time spent on processing, etc. Through methods such as time series analysis and machine learning prediction models, the data processing workload that may be generated in the target time period (such as the next 10 minutes or half an hour) is estimated based on these historical records, that is, the data processing load.

[0022] Since each server has its own upper limit of data processing capacity and the type of data it is good at processing, a percentage is set in advance. For example, when the predicted data processing load exceeds 80% of the maximum data processing capacity of the cloud (i.e. all servers), it means that according to the current resource configuration, the server may not be able to efficiently complete the data processing task during the target period. At this time, the first server automatically starts the resource scheduling optimization mode.

[0023] S102, interacting with each terminal connected to the cloud to obtain the data information to be processed of each terminal, wherein the data information to be processed includes the predicted data volume and data type of the data to be processed of each terminal in the target time period; and obtaining the server attribute information of each second server.

[0024] The first server actively communicates with each terminal device connected to the cloud, and obtains the data content that each terminal is expected to upload to the cloud for processing during the target period through preset protocols and interfaces, that is, the data information to be processed. The data information to be processed is predicted by each terminal itself, including the predicted data volume and data type of the data to be processed by each terminal during the target period. The data volume of the data to be processed clarifies the size of the data that each terminal is expected to upload, such as in GB or TB; the data type indicates what category the data belongs to, such as text data, image data, video data, or structured data, unstructured data, etc. Different data types have different requirements for server resources when processing. For example, the processing of images, video data, and unstructured data usually requires more computing resources, so accurately obtaining this information is crucial for the subsequent reasonable allocation of resources.

[0025] The first server also obtains the properties of other second servers participating in data processing in the cloud, including the server's hardware configuration, such as the number of CPU cores, memory size, and storage capacity; it may also include the server's current load, such as the proportion of computing resources occupied, storage resource usage, etc.

[0026] S103, using a resource scheduling optimization model to analyze and process the information of each piece of data to be processed and the attribute information of each server to obtain a resource scheduling optimization plan, wherein the resource scheduling optimization plan includes a plurality of second servers respectively associated with each terminal, and the second servers are used to process the data to be processed uploaded by each terminal during a target period.

[0027] The information of the data to be processed of each terminal (data volume, data type) and the attribute information of each second server obtained in the above steps are used as input, and the trained resource scheduling optimization model performs comprehensive analysis and processing on them. After the analysis and processing by the model, a detailed resource scheduling optimization plan is output. The plan includes which second server(s) the data to be processed of each terminal should be allocated to for processing.

[0028] Through this precise resource allocation, the overall data processing efficiency of the cloud can be maximized, ensuring that the data of each terminal can be processed efficiently and timely during periods when the cloud data processing load is heavy.

[0029] It is understandable that the resource scheduling optimization model is preferably constructed through Transformer, and uses the actual data processing records of the cloud (corresponding to the amount and type of data actually uploaded by different terminals) and the corresponding overall data processing efficiency of the cloud (obtained by manual or machine evaluation) as label data to create training data, and use these training data to train the resource scheduling optimization model. The trained resource scheduling optimization model can analyze the optimal resource scheduling optimization solution for all terminals based on the processing efficiency of different data types on different servers, the current load balancing of each server, etc.

[0030] The above-mentioned scheme of the present invention, after interacting with the terminal to obtain the data information to be processed and mastering the attribute information of the second server, uses the resource scheduling optimization model to generate an accurate resource scheduling scheme, which can realize reasonable allocation of resources according to the data type, data volume, server hardware configuration and load conditions.

[0031] Optionally, predicting the data processing load in the target period based on the stored cloud-based long-term historical data processing records and cloud-based recent historical data processing records includes: Obtaining identity information of each terminal accessing the cloud, retrieving corresponding cloud access records according to the identity information, and classifying each terminal into a first type terminal and a second type terminal according to the cloud access records; the first type terminal refers to a terminal whose number of cloud access records is higher than a preset number, and the second type terminal refers to a terminal whose number of cloud access records is not higher than a preset number; Calculating a ratio of the number of the second type of terminals to the sum of the number of the second type of terminals and the first type of terminals, and determining the acquisition time of the cloud-based remote historical data processing record according to the ratio; The cloud-based long-term historical data processing record is obtained according to the acquisition time, and the cloud-based short-term historical data processing record is obtained according to a preset time.

[0032] In this embodiment, the solution of the present invention is to use the historical records of data processing on the cloud to analyze the upload rules of the data to be processed, and then predict the data processing load that may occur in the target period. At the same time, in order to improve the prediction accuracy, the present invention also sets a dynamic adjustment for the data acquisition amount of the cloud's long-term historical data processing records and the cloud's recent historical data processing records. Specifically: The first server first collects the identity information of each terminal connected to the cloud in the current period, which can be unique information such as device ID and user account, to accurately identify each terminal individual. Then, using the obtained terminal identity information, the corresponding access record is searched in the cloud database. These records record in detail the relevant information of the terminal's access to the cloud in the past (for example, the past month or two months), including access time, access times, etc.

[0033] A threshold for the number of access records is preset, and terminals with access records higher than the preset number are defined as type 1 terminals. These terminals are stable users of the cloud, and their data upload needs are relatively regular and stable. Terminals with access records not higher than the preset number are classified as type 2 terminals. These terminals are new users of the cloud service, and their data upload needs may be more uncertain.

[0034] After the above division is completed, the number of terminals included in the first type of terminals and the second type of terminals is counted to calculate the proportion of the number of second type of terminals in all terminals (i.e., the sum of the number of first type of terminals and second type of terminals). If the proportion of the second type of terminals is high, it means that there are more low-frequency terminals with high uncertainty. In order to more accurately capture the possible data processing modes and rules, the acquisition time of the distant historical data will be shortened accordingly, that is, the reference degree of the distant historical data will be reduced, and the reference degree of the recent historical data will be correspondingly increased. On the contrary, if the proportion of the second type of terminals is low, the high-frequency first type of terminals is dominant, and the data processing mode is relatively more stable, the acquisition time of the distant historical data can be appropriately extended to more comprehensively capture the possible data processing modes and rules, that is, to increase the reference degree of the distant historical data, and to reduce the reference degree of the recent historical data. Among them, the acquisition time of the recent historical data is a fixed value, that is, a preset time, and the preset time is, for example, the corresponding time of the current time period, 1 hour, 2 hours, or a certain percentage of the current time period, such as 80%, 75%.

[0035] According to the acquisition time determined earlier, query and extract the long-term historical data processing records within the corresponding time period in the cloud historical data repository. These records contain detailed information on the data processed by various terminals in the past period of time, such as the amount of data processed, processing time, processing results, etc. At the same time, obtain the recent historical data processing records according to the pre-set fixed time.

[0036] Recent historical data can better reflect current business trends and data processing characteristics, and combined with long-term historical data, it provides comprehensive data support for accurately predicting the data processing load in the target period. Through this targeted historical data acquisition method, it can better adapt to the data processing rules and characteristics of different types of terminals, improve the accuracy of data processing load prediction, and lay the foundation for the reasonable start-up and efficient operation of subsequent resource scheduling optimization models.

[0037] Optionally, the preset percentage is determined as follows: Extracting a portion of the historical data processing records corresponding to the target period from the cloud-based remote historical data processing records, and obtaining a probability of occurrence of data processing tasks that are higher than a priority level threshold based on the portion of the historical data processing records; The preset percentage is obtained according to the occurrence probability matching; wherein the preset percentage is positively correlated with the occurrence probability.

[0038] In this embodiment, according to the aforementioned embodiment, when the predicted data processing load exceeds the preset percentage of the maximum data processing capacity of the cloud, it is determined that a more in-depth resource scheduling strategy needs to be adopted for the target period. However, there may be high-priority scheduling tasks in the target period, and a more in-depth resource scheduling strategy may cause these high-priority scheduling tasks to not receive the expected priority treatment. In this regard, the present invention is configured to dynamically allocate the above-mentioned preset percentage, that is, to adjust the difficulty of starting the resource scheduling optimization mode by adjusting the size of the preset percentage. Specifically as follows: Since the cloud-based long-term historical data processing records cover a long time span, in order to more accurately determine the preset percentage related to the current target period, we first need to filter out the part corresponding to the target period from these massive historical records. For example, if the target period is from 9:00 to 10:00 a.m. every day, then we extract the data processing records from 9:00 to 10:00 a.m. on each date in the past from the cloud-based long-term historical data processing records.

[0039] In the historical data processing records related to the target time period extracted, a priority threshold is further set, which is used to distinguish the priority level of the data processing tasks (the priority level is represented by numbers 1-10, for example, and the larger the number, the higher the corresponding priority level). For example, some data processing tasks may involve urgent business needs, high-value customer data, etc., and are given a higher priority level. By traversing and analyzing some historical data processing records, the proportion of data processing tasks that are higher than the priority threshold in all data processing tasks (that is, all data processing tasks in some historical data processing records) is calculated, and this proportion is the above-mentioned probability of occurrence. For example, among the 100 historical data processing records extracted for the target time period, there are 30 records corresponding to tasks that are higher than the priority threshold, then the probability is 30%.

[0040] A corresponding relationship between the preset percentage and the occurrence probability obtained by the above statistics is established in advance, and this relationship is positively correlated. When the occurrence probability of the data processing task higher than the priority level threshold is higher, the preset percentage is set higher. For example, a mapping rule may be set in advance, when the occurrence probability is 10%, the preset percentage is 70%; when the occurrence probability is 15%, the preset percentage is 80%, and so on.

[0041] If the probability of high-priority data processing tasks appearing in the target period is high in history, the preset percentage will be set relatively large accordingly, which can increase the difficulty of starting the resource scheduling optimization mode to ensure that high-priority data processing tasks appearing in the target period can be processed preferentially to ensure the overall priority processing efficiency of high-priority data processing tasks. If the probability of high-priority data processing tasks appearing in the target period is low in history, the preset percentage will be set relatively small accordingly, which can reduce the difficulty of starting the resource scheduling optimization mode to ensure the overall data processing efficiency in the target period through the resource scheduling optimization mode.

[0042] By determining the preset percentage based on historical data characteristics, the activation of the resource scheduling optimization mode is more scientific and targeted, and can better adapt to the data processing needs of the target period, thereby achieving a balance between the overall data processing efficiency and the overall priority processing efficiency of high-priority data processing tasks.

[0043] Optionally, the using a resource scheduling optimization model to analyze and process each of the to-be-processed data information and each of the server attribute information to obtain a resource scheduling optimization solution includes: A first reserved number is obtained according to the occurrence probability matching, and second servers of the first reserved number are selected to be in a standby state; The resource scheduling optimization model is used to analyze and process each of the to-be-processed data information and the server attribute information of each second server in a non-standby state to obtain the resource scheduling optimization solution.

[0044] In this embodiment, even when the probability of a high-priority data processing task occurring is low, a high-priority data processing task may still occur within the target period. In view of this situation, the present invention is configured to reserve a portion (i.e., a first reserved number) of second servers for these high-priority data processing tasks that may occur, and these second servers are dedicated to the high-priority data processing tasks that occur within the target period.

[0045] For other non-reserved (i.e., non-standby) second servers, the resource scheduling optimization model performs analysis and processing in the aforementioned manner to obtain the optimal resource scheduling optimization solution, i.e., determine the second server corresponding to each terminal. It is understandable that multiple terminals may be assigned the same second server, and a single terminal may be assigned multiple second servers, without specific limitation.

[0046] Optionally, the step of screening the first reserved number of second servers to be in a standby state includes: Evaluate the performance of the resource scheduling optimization solution generated by the first server to obtain a performance evaluation value, and obtain an adjustment factor according to the performance evaluation value; the adjustment factor is a value greater than 1; The first reserved quantity is adjusted to a second reserved quantity using the adjustment factor, and second servers of the second reserved quantity are selected to be in a standby state.

[0047] In this embodiment, in addition to the high-priority data processing tasks that may appear in the target time period, some non-high-priority data processing tasks may also appear. These non-high-priority data processing tasks come from some terminals that are newly connected in the target time period.

[0048] The first server can periodically generate and update resource scheduling optimization plans (not involving high-priority data processing tasks) within the target period to meet the data processing needs of the non-high-priority data processing tasks of these newly connected terminals. However, the generation of resource scheduling optimization plans takes a certain amount of time. When the efficiency of the first server in generating resource scheduling optimization plans is insufficient, these newly added non-high-priority data processing tasks may not be allocated by the second server for a long time, resulting in data jams, untimely responses, etc.

[0049] In view of the above situation, the present invention is set to first evaluate the efficiency of the first server generating the resource scheduling optimization plan to obtain the efficiency evaluation value. The evaluation can be achieved based on factors such as the generation time of the resource scheduling optimization plan, whether the data processing tasks allocated to each second server are balanced, and whether the data processing can be completed within the specified time. The specific evaluation rules are not limited.

[0050] At the same time, a correspondence between the performance evaluation value and the adjustment factor is established in advance, and the corresponding adjustment factor can be obtained by querying the correspondence based on the performance evaluation value obtained through evaluation. Since the adjustment factor is a value greater than 1, it can appropriately amplify and adjust the subsequent reserved quantity. It can be understood that if the performance evaluation value is high, it means that the first server has a strong ability to generate resource scheduling optimization solutions. At this time, relatively few backup servers are needed to deal with emergencies, and the matched adjustment factor is relatively small; conversely, if the performance evaluation value is low, it means that the first server is insufficient in generating solutions. In order to ensure the stability of the system, more backup servers are needed, and the matched adjustment factor will be relatively large.

[0051] Among the numerous second servers in the cloud, according to certain screening criteria (such as server load, hardware performance, operation stability, etc.), select the second servers equal to the second reserved number and set them to standby status. During the target period, these standby second servers do not participate in the data processing tasks of the currently connected terminals, but are in standby status, so that they can be put into use in time when high-priority data processing tasks or other data processing requirements of new terminals appear, so as to ensure the normal operation of the system and the smooth progress of data processing.

[0052] like Figure 2 As shown, an embodiment of the present invention further discloses a cloud-based data processing system, which is applied to a first server. The system includes a processing device and a storage device. The computer code stored in the storage device is called and executed by the processing device to implement the following steps: Predicting the data processing load of the target period based on the stored cloud-based long-term historical data processing records and cloud-based recent historical data processing records, and starting a resource scheduling optimization mode when the data processing load exceeds a preset percentage of the maximum data processing capacity; Interacting with each terminal connected to the cloud to obtain the data information to be processed of each terminal, wherein the data information to be processed includes the predicted data volume and data type of the data to be processed of each terminal in the target time period; and obtaining the server attribute information of each second server; A resource scheduling optimization model is used to analyze and process the information of each piece of data to be processed and the attribute information of each server to obtain a resource scheduling optimization plan. The resource scheduling optimization plan includes several second servers respectively associated with each terminal, and the second servers are used to process the data to be processed uploaded by each terminal during the target time period.

[0053] An embodiment of the present invention further discloses an electronic device, comprising: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the method described in the above embodiment.

[0054] An embodiment of the present invention further discloses a computer storage medium, wherein the computer storage medium stores a computer program, and the computer program is executed by a processor to implement the method described in the above embodiment.

[0055] An embodiment of the present invention further discloses a computer program product, which includes computer code. When the computer code is executed by a processor of an electronic device, the method described in the above embodiment is implemented.

[0056] The computer-readable storage medium described above may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. Alternatively, the computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0057] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0058] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A cloud-based data processing method, applied to a first server, characterized in that: The method comprises the following steps: Predicting the data processing load of the target period based on the stored cloud-based long-term historical data processing records and cloud-based recent historical data processing records, and starting a resource scheduling optimization mode when the data processing load exceeds a preset percentage of the maximum data processing capacity; Interacting with each terminal connected to the cloud to obtain the data information to be processed of each terminal, wherein the data information to be processed includes the predicted data volume and data type of the data to be processed of each terminal in the target time period; and obtaining the server attribute information of each second server; A resource scheduling optimization model is used to analyze and process the information of each piece of data to be processed and the attribute information of each server to obtain a resource scheduling optimization plan. The resource scheduling optimization plan includes several second servers respectively associated with each terminal, and the second servers are used to process the data to be processed uploaded by each terminal during the target time period.

2. A cloud-based data processing method according to claim 1, characterized in that: The method of predicting the data processing load of the target period based on the stored cloud-based long-term historical data processing records and cloud-based recent historical data processing records includes: Obtaining identity information of each terminal accessing the cloud, retrieving corresponding cloud access records according to the identity information, and classifying each terminal into a first type terminal and a second type terminal according to the cloud access records; the first type terminal refers to a terminal whose number of cloud access records is higher than a preset number, and the second type terminal refers to a terminal whose number of cloud access records is not higher than a preset number; Calculating a ratio of the number of the second type of terminals to the sum of the number of the second type of terminals and the first type of terminals, and determining the acquisition time of the cloud-based remote historical data processing record according to the ratio; The cloud-based long-term historical data processing record is obtained according to the acquisition time, and the cloud-based short-term historical data processing record is obtained according to a preset time.

3. The cloud-based data processing method according to claim 1, characterized in that: The preset percentage is determined as follows: Extracting a portion of the historical data processing records corresponding to the target period from the cloud-based remote historical data processing records, and obtaining a probability of occurrence of data processing tasks that are higher than a priority level threshold based on the portion of the historical data processing records; The preset percentage is obtained according to the occurrence probability matching; wherein the preset percentage is positively correlated with the occurrence probability.

4. A cloud-based data processing method according to claim 3, characterized in that: The using of the resource scheduling optimization model to analyze and process each of the to-be-processed data information and each of the server attribute information to obtain a resource scheduling optimization solution includes: A first reserved number is obtained according to the occurrence probability matching, and second servers of the first reserved number are selected to be in a standby state; The resource scheduling optimization model is used to analyze and process each of the to-be-processed data information and the server attribute information of each second server in a non-standby state to obtain the resource scheduling optimization solution.

5. A cloud-based data processing method according to claim 4, characterized in that: The step of selecting the first reserved number of second servers to be in a standby state comprises: Evaluate the performance of the resource scheduling optimization solution generated by the first server to obtain a performance evaluation value, and obtain an adjustment factor according to the performance evaluation value; the adjustment factor is a value greater than 1; The first reserved quantity is adjusted to a second reserved quantity using the adjustment factor, and second servers of the second reserved quantity are selected to be in a standby state.

6. A cloud-based data processing system, applied to a first server, the system comprising a processing device and a storage device, characterized in that: The computer code stored in the storage device is called and executed by the processing device to implement the following steps: Predicting the data processing load of the target period based on the stored cloud-based long-term historical data processing records and cloud-based recent historical data processing records, and starting a resource scheduling optimization mode when the data processing load exceeds a preset percentage of the maximum data processing capacity; Interacting with each terminal connected to the cloud to obtain the data information to be processed of each terminal, wherein the data information to be processed includes the predicted data volume and data type of the data to be processed of each terminal in the target time period; and obtaining the server attribute information of each second server; A resource scheduling optimization model is used to analyze and process the information of each piece of data to be processed and the attribute information of each server to obtain a resource scheduling optimization plan. The resource scheduling optimization plan includes several second servers respectively associated with each terminal, and the second servers are used to process the data to be processed uploaded by each terminal during the target time period.

7. A cloud-based data processing system according to claim 6, characterized in that: The method of predicting the data processing load of the target period based on the stored cloud-based long-term historical data processing records and cloud-based recent historical data processing records includes: Obtaining identity information of each terminal accessing the cloud, retrieving corresponding cloud access records according to the identity information, and classifying each terminal into a first type terminal and a second type terminal according to the cloud access records; the first type terminal refers to a terminal whose number of cloud access records is higher than a preset number, and the second type terminal refers to a terminal whose number of cloud access records is not higher than a preset number; Calculating a ratio of the number of the second type of terminals to the sum of the number of the second type of terminals and the first type of terminals, and determining the acquisition time of the cloud-based remote historical data processing record according to the ratio; The cloud-based long-term historical data processing record is obtained according to the acquisition time, and the cloud-based short-term historical data processing record is obtained according to a preset time.

8. An electronic device comprising: At least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 5.

9. A computer storage medium storing a computer program, characterized in that: The computer program is executed by a processor to implement the method according to any one of claims 1 to 5.

10. A computer program product, characterized in that: The computer program product includes computer codes, and when the computer codes are executed by a processor of an electronic device, the method according to any one of claims 1 to 5 is implemented.

Images