A cloud-based AI-driven big data processing system

By dividing task scenarios, analyzing task completion data deviation and optimizing storage distribution locations in a cloud-driven big data processing system, the problem of inability to timely discover performance bottlenecks and accurately predict storage consumption in the existing technology is solved, and the problem of further development is achieved. Efficient resource management and performance optimization.

CN118740843BActive Publication Date: 2024-12-03JILIN HENGTAI JIAXIN TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411219609.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-02
Publication Date
2024-12-03
Estimated Expiration
2044-09-02

AI Technical Summary

Technical Problem

The existing cloud-powered big data processing systems cannot detect performance bottlenecks in time, resulting in lower service stability and response speed than expected, and cannot accurately predict the storage consumption time, affecting the reliability of cloud storage services.

Method used

Through the scene division module, the tasks are divided into data storage, data access and computing power services, and the task completion data is collected and analyzed separately, the deviation between reality and prediction is calculated, and the judgment and early warning module is used to take measures based on the relationship between deviation and threshold. Optimize the storage distribution location and computing power resource configuration, and build a machine learning model to predict the completion time of computing power service.

Benefits of technology

Accurate performance management of different types of tasks, timely discover performance bottlenecks, reduce service interruption time, improve user experience, and optimize storage and computing resources use through intelligent storage layout and resource configuration, and improve system reliability Sex and efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118740843B_ABST
    Figure CN118740843B_ABST
Patent Text Reader

Abstract

The present invention discloses a cloud AI-driven big data processing system, which specifically relates to the technical field of big data processing, including: dividing cloud tasks into cloud data storage tasks, cloud data access tasks and cloud computing service tasks according to usage scenarios; collecting task completion related data of cloud servers, analyzing cloud service process data, and obtaining storage time deviation, access time deviation and computing time deviation; respectively judging the relationship between storage time deviation, access time deviation and computing time deviation and corresponding thresholds, and taking measures based on the judgment results; by dividing cloud tasks into three usage scenarios of data storage, data access and computing service, different types of tasks can be more accurately identified and processed, thereby optimizing resource allocation and performance management, helping to quickly respond to performance issues, reduce service interruption time, and improve user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of big data processing technology, and more specifically, to a cloud-based AI-driven big data processing system. Background Art

[0002] The cloud AI-driven big data processing system is a data processing architecture that integrates artificial intelligence technology and cloud computing capabilities, designed to efficiently process, analyze and mine massive amounts of data. This system achieves efficient data management by leveraging the cloud's powerful computing resources, storage capabilities, and the intelligence of AI algorithms. Existing big data processing systems usually integrate a variety of technologies, including distributed storage, parallel computing frameworks, data processing and analysis tools, etc., to support the processing of massive amounts of data.

[0003] The existing big data processing systems have the following problems: they cannot detect performance bottlenecks in cloud services in a timely manner, resulting in service stability and response speed being lower than expected, thus affecting user experience; the existing technology cannot accurately predict storage consumption time and cannot effectively supervise the cloud storage service process, resulting in abnormal reliability of cloud storage services. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides a cloud-based AI-driven big data processing system to solve the problems raised in the above-mentioned background technology.

[0005] To achieve the above objectives, the present invention provides the following technical solutions: A cloud-based AI-driven big data processing system, comprising:

[0006] A scenario division module, which is used to divide cloud tasks into cloud data storage tasks, cloud data access tasks and cloud computing service tasks according to usage scenarios;

[0007] A data collection module is used to collect task completion related data of the cloud server, wherein the task completion related data at least includes: actual storage consumption time, actual access consumption time and actual computing service completion time;

[0008] The data storage service analysis module analyzes the process data of the data to be stored in the cloud database node to obtain the predicted storage consumption time; analyzes the actual storage consumption time and the predicted storage consumption time, outputs the storage time deviation Pyc, and transmits the obtained storage time deviation to the judgment and warning module;

[0009] The access service data analysis module analyzes the process data of users accessing the cloud database to obtain the predicted access consumption time; analyzes the actual access consumption time and the predicted access consumption time, outputs the access time deviation Pyf, and transmits the obtained access time deviation to the judgment and warning module;

[0010] The computing power service analysis module inputs the computing power service process data into the computing power task prediction model and outputs the predicted computing power service completion time; analyzes the predicted computing power service completion time and the actual computing power service completion time, and outputs the computing power time deviation Pys; transmits the obtained computing power time deviation to the judgment and warning module;

[0011] The judgment and warning module judges the relationship between the storage time deviation, access time deviation and computing time deviation and the corresponding thresholds, and takes measures based on the judgment results.

[0012] Preferably, the method for obtaining the predicted storage consumption time is: generate several replica data blocks of the data to be stored, and use the data loss probability Pds, access speed fv and storage cost fc as constraints to solve the storage distribution position matrix corresponding to the data to be stored, and find the target storage position of each replica data block from the storage distribution position matrix; based on the distance between each replica data block and the corresponding target storage position and the network transmission speed, calculate the predicted storage consumption time for each replica data block to arrive at the storage distribution position matrix.

[0013] Preferably, the process of obtaining the predicted storage consumption time includes the following contents:

[0014] Assume that each database to be stored has p replica data blocks, and q represents the number of the replica data block;

[0015] The size of the qth replica data block is recorded as S q ;

[0016] Find the target storage location of the qth replica data block from the storage distribution location matrix as L q , the network transmission speed of the qth replica data block to the corresponding target storage location is V q ;

[0017] By formula The predicted storage consumption time yct_q is calculated; where Jf represents the storage node load impact coefficient: the real-time load of the storage node will affect the actual storage time, because high load may lead to a decrease in processing speed; Δtq represents the transmission distance impact constant.

[0018] Preferably, the process of acquiring the storage distribution position matrix includes the following steps:

[0019] Step S11: determine the constraint condition: data loss probability P ds , access speed fv and storage cost fc;

[0020] Step S12, preprocessing: normalize the access speed and record it as the access speed measurement parameter Nor_fv; obtain the maximum and minimum values ​​of the storage cost, and obtain the storage cost measurement parameter Nor_fc after linear normalization processing;

[0021] Step S13, comprehensively considering the data loss probability, access speed measurement parameters and storage cost measurement parameters, and constructing a cloud storage objective function QI;

[0022] Step S14, randomly generate candidate solutions for the storage distribution position matrix; represent the location where the data replica is stored by integer coding;

[0023] Step S15: taking the value of the cloud storage objective function as the fitness value, and calculating the fitness value of each candidate solution;

[0024] Step S16, genetic operation: perform genetic operation based on the candidate solution and output a new generation of candidate solutions; the genetic operation includes: selection, crossover and mutation operations;

[0025] Step S17, repeating step S15 and step S16 until a preset number of iterations is reached or the fitness value converges; after termination, selecting the candidate solution with the highest fitness as the optimal solution, that is, storing the distribution position matrix, and storing the distribution position matrix in the database.

[0026] Preferably, the access time deviation is obtained by: selecting the replica data block with the shortest distance from the database as the nearest replica based on the access request, and locating the replica data block with the nearest distance; obtaining the transmission distance between the nearest replica and the data request location; calculating the predicted access consumption time based on the real-time network speed and the replica data block size, as well as the transmission distance; obtaining the actual access consumption time by monitoring the real-time measurement; and constructing the access time deviation based on the difference between the actual access consumption time and the predicted access consumption time.

[0027] Preferably, a computing task prediction model for the completion time of a computing service is built based on a machine learning algorithm, the request data of the computing service is input into the computing task prediction model, and the predicted completion time of the computing service is output. The building process of the computing task prediction model includes the following steps:

[0028] Step S21: Collect the request data of computing power service, including but not limited to the type, size, complexity and priority of the request; predict the computing power resources required to execute the computing power service request based on the historical data;

[0029] Step S22: Select features that have a significant impact on the predicted completion time from the request data of the computing service and the required computing resources;

[0030] Step S23: Initialize the neural network model parameters as the initial model of the computing power task prediction model, input the computing power service request data into the computing power task prediction model, output the predicted computing power service completion time, use historical data to train the selected machine learning algorithm, and optimize the performance of the neural network model on the training set by adjusting the parameters; use the test set to evaluate the performance of the neural network model;

[0031] Step S24, model optimization and deployment: optimize the computing power task prediction model according to the evaluation results, including adjusting the neural network parameters and adjusting the feature weights; output the trained computing power task prediction model; deploy the trained computing power task prediction model to the environment to perform real-time prediction of new computing power service requests.

[0032] Preferably, the big data processing system further includes:

[0033] The big data cloud service comprehensive management module receives the storage time deviation, access time deviation and computing power deviation; obtains the cloud service value coefficient through joint analysis, and takes measures based on the cloud service value coefficient; matches the weight coefficients for the storage time deviation, access time deviation and computing power deviation, which are recorded as wqa, wqb and wqc respectively; and uses the formula The cloud service value coefficient YF is calculated, where Form (·) represents a linear normalization function, which is used to limit the value range of the value in brackets to the range of 0 to 1.

[0034] Preferably, the storage time deviation is represented by the data storage quality evaluation index Ya; the storage time deviation is represented by the data storage quality evaluation index Y a Indicates the storage time deviation; the data storage quality evaluation index Y a The way to obtain is:

[0035] Assume that s groups of databases to be stored are stored in the cloud database, and k represents the sequence number of the databases to be stored;

[0036] Assume that each database to be stored has p replica data blocks, and q represents the number of the replica data block;

[0037] The predicted storage consumption time of the qth replica data block is recorded as yct_q, and the actual storage consumption time of the qth replica data block is recorded as sct_q. Calculate the data storage quality assessment index .

[0038] Preferably, the data access quality evaluation index Y is used b It represents the access time deviation, and the access response time threshold is recorded as Tth;

[0039] Assume that R data accesses are performed, and t represents the sequence number of the data accesses;

[0040] The predicted access time of the t-th data access is recorded as y_t, and the actual access time of the t-th data access is recorded as y"_t;

[0041] The data access quality evaluation index Y is calculated by the following formula b

[0042] , where α and β represent weight factors, which are used to adjust the weights of time differences and abnormal accesses; 1[Tq′>Tth] represents the indicator function, which is 1 when y_t>Tth, otherwise it is 0; Nod_t is the load of the node corresponding to the tth data access, which is a value between 0 and 1, used to indicate the busyness of the node.

[0043] Technical effects and advantages of the present invention:

[0044] 1. The cloud AI-driven big data processing system provided by the present invention can more accurately identify and process different types of tasks by dividing cloud tasks into three usage scenarios: data storage, data access, and computing power services, thereby optimizing resource allocation and performance management; the data storage service analysis module, the access service data analysis module, and the computing power service analysis module respectively perform performance analysis on different types of tasks, predict the task completion time, and calculate the deviation between the actual and predicted time. This intelligent analysis helps to timely discover performance bottlenecks and anomalies; the judgment and warning module triggers the warning in time and takes corresponding measures based on the relationship between the deviation and the threshold, which helps to quickly respond to performance problems, reduce service interruption time, and improve user experience.

[0045] 2. The cloud-based AI-driven big data processing system provided by the present invention reduces the risk of data loss by generating multiple replica data blocks and optimizing their storage locations. By optimizing the storage distribution position matrix, users can access required data more quickly, thereby improving the access performance of the overall system. Through intelligent storage layout, unnecessary waste of resources is avoided and cost-effectiveness is maximized. Based on the distance between each replica data block and the corresponding target storage location and the network transmission speed, the predicted storage consumption time is calculated, providing the system with intelligent prediction capabilities. This prediction capability helps the system make more reasonable decisions and solves the problem that the prior art cannot accurately predict the storage consumption time, resulting in insufficiently intelligent supervision. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is the structural framework of the big data processing system of the present invention.

[0047] Figure 2This is a flow chart for obtaining the storage distribution position matrix of the present invention. DETAILED DESCRIPTION

[0048] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0049] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0050] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present application, its application, or uses.

[0051] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0052] Example 1, see Figure 1 The present invention provides a big data processing system structure framework. Figure 1 A cloud AI-driven big data processing system is shown, comprising:

[0053] A scenario division module, which is used to divide cloud tasks into cloud data storage tasks, cloud data access tasks and cloud computing service tasks according to usage scenarios;

[0054] A data collection module is used to collect task completion related data of the cloud server, wherein the task completion related data at least includes: actual storage consumption time, actual access consumption time and actual computing service completion time;

[0055] The data storage service analysis module analyzes the process data of the data to be stored in the cloud database node to obtain the predicted storage consumption time; analyzes the actual storage consumption time and the predicted storage consumption time, outputs the storage time deviation Pyc, and transmits the obtained storage time deviation to the judgment and warning module;

[0056] The access service data analysis module analyzes the process data of users accessing the cloud database to obtain the predicted access consumption time; analyzes the actual access consumption time and the predicted access consumption time, outputs the access time deviation Pyf, and transmits the obtained access time deviation to the judgment and warning module;

[0057] The computing power service analysis module inputs the computing power service process data into the computing power task prediction model and outputs the predicted computing power service completion time; analyzes the predicted computing power service completion time and the actual computing power service completion time, and outputs the computing power time deviation Pys; transmits the obtained computing power time deviation to the judgment and warning module;

[0058] The judgment and warning module judges the relationship between the storage time deviation, access time deviation and computing time deviation and the corresponding thresholds, and takes measures based on the judgment results.

[0059] Explanation: For storage time deviation (Pyc), access time deviation (Pyf) and computing time deviation (Pys), reasonable initial thresholds need to be set based on historical data, system performance expectations and business needs. These thresholds can be fixed or dynamically adjusted over time or system status; the system should be able to automatically monitor and analyze performance indicators (such as load, response time, resource utilization, etc.) and dynamically adjust thresholds based on changes in these indicators. For example, during peak periods, the thresholds can be appropriately relaxed to cope with possible performance fluctuations; during trough periods, the thresholds can be tightened to ensure service quality.

[0060] What needs to be further explained in the embodiments of the present invention is that the judgment and warning module can quickly identify potential problems or problems that have occurred in the service by comparing with the corresponding thresholds, such as reduced reliability of data storage, slower data access speed, and uneven distribution of computing resources; once a problem is found, corresponding measures are taken based on preset strategies (such as cloud service fault analysis knowledge graphs), such as increasing redundant storage, optimizing data access paths, adjusting computing resource allocation, etc., so as to avoid further deterioration of the problem and ensure the continuity and stability of the service.

[0061] What needs to be further explained in the embodiments of the present invention is that the method for obtaining the predicted storage consumption time is: generate several replica data blocks of the data to be stored, and use the data loss probability Pds, access speed fv and storage cost fc as constraints to solve the storage distribution position matrix corresponding to the data to be stored, and find the target storage position of each replica data block from the storage distribution position matrix; based on the distance between each replica data block and the corresponding target storage position and the network transmission speed, calculate the predicted storage consumption time for each replica data block to arrive at the storage distribution position matrix.

[0062] It needs to be further explained in the embodiment of the present invention that the process of obtaining the predicted storage consumption time includes the following contents:

[0063] Assume that each database to be stored has p replica data blocks, and q represents the number of the replica data block;

[0064] The size of the qth replica data block is recorded as Sq (Unit: Byte);

[0065] Find the target storage location of the qth replica data block from the storage distribution location matrix as L q , the network transmission speed of the qth replica data block to the corresponding target storage location is V q (Unit: bytes / second);

[0066] By formula The predicted storage consumption time yct_q is calculated; where Jf represents the storage node load impact coefficient: the real-time load of the storage node will affect the actual storage time, because high load may lead to a decrease in processing speed; Δtq represents the transmission distance impact constant.

[0067] In the further design, the transmission distance influence constant Δtq is obtained by: obtaining the physical distance d from the current position to the target storage position q , through the formula The transmission distance influence constant is obtained, where k is a constant related to network conditions.

[0068] See also Figure 2 The process of acquiring the storage distribution position matrix includes the following steps:

[0069] Step S11: determine the constraint condition: data loss probability P ds , access speed fv and storage cost fc;

[0070] Step S12, preprocessing: normalize the access speed and record it as the access speed measurement parameter Nor_fv; obtain the maximum and minimum values ​​of the storage cost, and obtain the storage cost measurement parameter Nor_fc after linear normalization processing;

[0071] Step S13, comprehensively considering the data loss probability, access speed measurement parameters and storage cost measurement parameters, and constructing a cloud storage objective function QI;

[0072] Explanation, through the formula The cloud storage objective function is calculated; where ga_1, ga_2, ga_3 are the proportional coefficients of each item, and ga_1+ga_2+ga_3=1;

[0073] Step S14, randomly generate candidate solutions for the storage distribution position matrix; represent the location where the data replica is stored by integer coding;

[0074] Step S15: taking the value of the cloud storage objective function as the fitness value, and calculating the fitness value of each candidate solution;

[0075] Step S16, genetic operation: perform genetic operation based on the candidate solution and output a new generation of candidate solutions; the genetic operation includes: selection, crossover and mutation operations;

[0076] Explanation: The selection refers to the use of roulette selection and tournament selection methods to select excellent individuals for reproduction according to fitness values; the crossover refers to exchanging some genes, that is, the storage locations of some replica data blocks, between the selected candidate solutions to generate new candidate solutions; the mutation operation refers to: randomly changing the genes of the candidate solutions, that is, randomly changing the storage locations of some data replicas) to increase the diversity of the population;

[0077] Step S17, repeating step S15 and step S16 until a preset number of iterations is reached or the fitness value converges; after termination, selecting the candidate solution with the highest fitness as the optimal solution, that is, storing the distribution position matrix, and storing the distribution position matrix in the database.

[0078] What needs to be further explained in the embodiments of the present invention is that the access time deviation is obtained in the following manner: based on the access request, a replica data block with the shortest distance is selected from the database as the nearest replica, and the replica data block with the nearest distance is located; the transmission distance between the nearest replica and the data request location is obtained; based on the real-time network speed and the replica data block size, as well as the transmission distance, the predicted access consumption time is calculated; by monitoring the real-time measurement, the actual access consumption time is obtained; based on the difference between the actual access consumption time and the predicted access consumption time, the access time deviation is constructed.

[0079] It is necessary to further explain in the embodiment of the present invention that a computing task prediction model for the completion time of a computing service is built based on a machine learning algorithm, the request data of the computing service is input into the computing task prediction model, and the predicted computing service completion time is output. The building process of the computing task prediction model includes the following steps:

[0080] Step S21: Collect the request data of computing power service, including but not limited to the type, size, complexity and priority of the request; predict the computing power resources required to execute the computing power service request based on the historical data;

[0081] Step S22: Select features that have a significant impact on the predicted completion time from the request data of the computing service and the required computing resources;

[0082] Explain and clean the collected data to deal with missing values, outliers, and duplicate values; convert non-numeric features (such as request type) into numeric features so that the machine learning algorithm can process them; feature encoding: convert the selected non-numeric features into numeric features. Common encoding methods include independent encoding, label encoding, and ordinal encoding;

[0083] Step S23: Initialize the neural network model parameters as the initial model of the computing power task prediction model, input the computing power service request data into the computing power task prediction model, output the predicted computing power service completion time, use historical data to train the selected machine learning algorithm, and optimize the performance of the neural network model on the training set by adjusting the parameters; use the test set to evaluate the performance of the neural network model;

[0084] Explanation: The evaluation indicator is any one of the mean square error, root mean square error, and mean absolute error, or a combination of multiple ones. The evaluation indicator is used to reflect the difference between the computing service completion time predicted by the computing task prediction model and the actual computing service completion time;

[0085] Step S24, model optimization and deployment: optimize the computing power task prediction model according to the evaluation results, including adjusting the neural network parameters and adjusting the feature weights; output the trained computing power task prediction model; deploy the trained computing power task prediction model to the environment to perform real-time prediction of new computing power service requests.

[0086] In a further design, the storage time deviation described in the embodiment of the present invention refers to any numerical value that can reflect the change in the difference between the actual storage consumption time and the predicted storage consumption time. For example, the storage time deviation is the ratio of the difference between the actual storage consumption time and the predicted storage consumption time to the predicted storage consumption time; or, considering the cloud storage quality index, the storage time deviation is represented by the ratio of the difference between the actual storage consumption time and the predicted storage consumption time to the predicted storage consumption time, divided by the numerical value of the cloud storage quality index; the influence of constraints is eliminated by dividing by the cloud storage quality index.

[0087] The present invention does not limit the calculation method of storage time deviation. In further design, the cloud storage quality index is considered and the data storage quality evaluation index Y is used. a Indicates the storage time deviation; the data storage quality evaluation index Y a The way to obtain is:

[0088] Assume that s groups of databases to be stored are stored in the cloud database, and k represents the sequence number of the databases to be stored;

[0089] Assume that each database to be stored has p replica data blocks, and q represents the number of the replica data block;

[0090] The predicted storage consumption time of the qth replica data block is recorded as yct_q, and the actual storage consumption time of the qth replica data block is recorded as sct_q. Calculate the data storage quality assessment index .

[0091] In a further design, the access time deviation described in the embodiment of the present invention refers to any value that can reflect the change between the actual access time and the predicted access time; for example, the access time deviation is: the ratio of the difference between the actual access time and the predicted access time to the predicted access time; or, considering the standard deviation of the access time difference, the standard deviation of the access time is calculated using the average of several access time differences and each access time difference; the access time deviation is obtained based on the standard deviation and the average of the access time differences;

[0092] The present invention does not make a unique limitation on the calculation method of the access time deviation. In further design, the node load and the access time threshold are comprehensively considered; the data access quality evaluation index Y b It represents the access time deviation, and the access response time threshold is recorded as Tth;

[0093] Assume that R data accesses are performed, and t represents the sequence number of the data accesses;

[0094] The predicted access time of the t-th data access is recorded as y_t, and the actual access time of the t-th data access is recorded as y"_t;

[0095] The data access quality evaluation index Y is calculated by the following formula b

[0096] , where α and β represent weight factors, which are used to adjust the weights of time differences and abnormal accesses; 1[Tq′>Tth] represents the indicator function, which is 1 when y_t>Tth, otherwise it is 0; Nod_t is the load of the node corresponding to the tth data access, which is a value between 0 and 1, used to indicate the busyness of the node.

[0097] Embodiment 2: The difference between the embodiment of the present invention and embodiment 1 is that the big data processing system further includes:

[0098] The big data cloud service comprehensive management module receives the storage time deviation, access time deviation and computing power deviation; conducts joint analysis to obtain the cloud service value coefficient, and takes measures based on the cloud service value coefficient.

[0099] In the further design, the weight coefficients for storage time deviation, access time deviation and computing power deviation are matched, which are denoted as wqa, wqb and wqc respectively; through the formula The cloud service value coefficient YF is calculated, where Form (·) represents a linear normalization function, which is used to limit the value range of the value in brackets to the range of 0 to 1.

[0100] The cloud service value coefficient reflects the current status of the cloud service and also indicates the future potential and market competitiveness of the service. Based on the cloud service value coefficient, resources are intelligently screened and allocated to ensure that high-value services receive more support, thereby enhancing the value of the entire cloud service. For example, for cloud services with high cloud service value coefficients, market expansion is increased; for cloud services with low cloud service value coefficients, resource integration or optimization adjustments are considered.

[0101] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A cloud AI-driven big data processing system, characterized in that: include: A scenario division module, which is used to divide cloud tasks into cloud data storage tasks, cloud data access tasks and cloud computing service tasks according to usage scenarios; A data collection module is used to collect task completion related data of the cloud server, wherein the task completion related data at least includes: actual storage consumption time, actual access consumption time and actual computing service completion time; The data storage service analysis module analyzes the process data of the data to be stored in the cloud database node to obtain the predicted storage consumption time; analyzes the actual storage consumption time and the predicted storage consumption time, outputs the storage time deviation Pyc, and transmits the obtained storage time deviation to the judgment and warning module; The method for obtaining the predicted storage consumption time is as follows: generating a number of replica data blocks of the data to be stored, taking the data loss probability Pds, the access speed fv and the storage cost fc as constraints, solving to obtain the storage distribution position matrix corresponding to the data to be stored, and finding the target storage position of each replica data block from the storage distribution position matrix; based on the distance between each replica data block and the corresponding target storage position and the network transmission speed, calculating the predicted storage consumption time for each replica data block to arrive at the storage distribution position matrix; The acquisition process of the storage distribution position matrix includes the following steps: Step S11, determining the constraint conditions as: data loss probability Pds, access speed fv and storage cost fc; Step S12, preprocessing: normalize the access speed and record it as the access speed measurement parameter Nor_fv; obtain the maximum and minimum values ​​of the storage cost, and obtain the storage cost measurement parameter Nor_fc after linear normalization processing; Step S13, comprehensively considering the data loss probability, access speed measurement parameters and storage cost measurement parameters, and constructing a cloud storage objective function QI; Step S14, randomly generate candidate solutions for the storage distribution position matrix; represent the location where the data replica is stored by integer coding; Step S15: taking the value of the cloud storage objective function as the fitness value, and calculating the fitness value of each candidate solution; Step S16, genetic operation: perform genetic operation based on the candidate solution and output a new generation of candidate solutions; the genetic operation includes: selection, crossover and mutation operations; Step S17, repeating step S15 and step S16 until a preset number of iterations is reached or the fitness value converges; after termination, selecting the candidate solution with the highest fitness as the optimal solution, that is, storing the distribution position matrix, and storing the distribution position matrix in the database; The access service data analysis module analyzes the process data of users accessing the cloud database to obtain the predicted access consumption time; analyzes the actual access consumption time and the predicted access consumption time, outputs the access time deviation Pyf, and transmits the obtained access time deviation to the judgment and warning module; The computing power service analysis module inputs the computing power service process data into the computing power task prediction model and outputs the predicted computing power service completion time; analyzes the predicted computing power service completion time and the actual computing power service completion time, and outputs the computing power time deviation Pys; transmits the obtained computing power time deviation to the judgment and warning module; The judgment and warning module judges the relationship between the storage time deviation, access time deviation and computing time deviation and the corresponding thresholds, and takes measures based on the judgment results.

2. The cloud AI-driven big data processing system according to claim 1, characterized in that: The process of obtaining the predicted storage consumption time includes the following: Assume that each database to be stored has p replica data blocks, and q represents the number of the replica data block; The size of the qth replica data block is recorded as S q ; Find the target storage location of the qth replica data block from the storage distribution location matrix as L q , the network transmission speed of the qth replica data block to the corresponding target storage location is V q ; By formula The predicted storage consumption time yct_q is calculated, where Jf represents the storage node load influence coefficient; Δtq represents the transmission distance influence constant.

3. The cloud AI-driven big data processing system according to claim 1, characterized in that: The access time deviation is obtained by: selecting a replica data block with the shortest distance from the database as the nearest replica based on the access request; obtaining the transmission distance between the nearest replica and the data request location; calculating the predicted access consumption time based on the real-time network speed, the replica data block size, and the transmission distance; and obtaining the actual access consumption time by monitoring the real-time measurement; Based on the difference between the actual access time and the predicted access time, the access time deviation is constructed.

4. The cloud AI-driven big data processing system according to claim 1, characterized in that: A computing task prediction model for the completion time of a computing service is built based on a machine learning algorithm, the request data of the computing service is input into the computing task prediction model, and the predicted computing service completion time is output. The building process of the computing task prediction model includes the following steps: Step S21: Collect the request data of computing power service, including but not limited to the type, size, complexity and priority of the request; predict the computing power resources required to execute the computing power service request based on the historical data; Step S22: Select features that have an impact on the predicted completion time from the request data of the computing service and the required computing resources; Step S23: Initialize the neural network model parameters as the initial model of the computing power task prediction model, input the computing power service request data into the computing power task prediction model, output the predicted computing power service completion time, use historical data to train the selected machine learning algorithm, and optimize the performance of the neural network model on the training set by adjusting the parameters; use the test set to evaluate the performance of the neural network model; Step S24, model optimization and deployment: optimizing the computing task prediction model according to the evaluation results, including adjusting the neural network parameters and feature weights; Output the trained computing task prediction model; deploy the trained computing task prediction model to the environment to make real-time predictions for new computing service requests.

5. The cloud-based AI-driven big data processing system according to claim 1, characterized in that: The big data processing system also includes: The big data cloud service comprehensive management module receives the storage time deviation, access time deviation and computing time deviation; obtains the cloud service value coefficient through joint analysis, and takes measures based on the cloud service value coefficient; matches the weight coefficients for the storage time deviation, access time deviation and computing time deviation, which are recorded as wqa, wqb and wqc respectively; and uses the formula The cloud service value coefficient YF is calculated, where Form (·) represents a linear normalization function, which is used to limit the value range of the value in brackets to the range of 0 to 1.

6. The cloud-based AI-driven big data processing system according to claim 5, characterized in that: Use data storage quality assessment index Y a Indicates the storage time deviation; the data storage quality evaluation index Y a The way to obtain is: Assume that s groups of databases to be stored are stored in the cloud database, and k represents the sequence number of the databases to be stored; Assume that each database to be stored has p replica data blocks, and q represents the number of the replica data block; The predicted storage consumption time of the qth replica data block is recorded as yct_q, and the actual storage consumption time of the qth replica data block is recorded as sct_q. Calculate the data storage quality evaluation index Y a .

7. The cloud-based AI-driven big data processing system according to claim 5, characterized in that: Use data access quality evaluation index Y b It represents the access time deviation, and the access response time threshold is recorded as Tth; Assume that R data accesses are performed, and t represents the sequence number of the data accesses; The predicted access time of the t-th data access is recorded as y_t, and the actual access time of the t-th data access is recorded as y"_t; The data access quality evaluation index Y is calculated by the following formula b , , where α and β represent weight factors, which are used to adjust the weights of time difference and abnormal access; It represents the indicator function, which is 1 when y_t>Tth, otherwise it is 0; Nod_t is the load of the node corresponding to the tth data access, which is a value between 0 and 1 and is used to indicate the busyness of the node.