Information management system and method based on cloud computing and big data

Abstract

The invention discloses an information management system and method based on cloud computing and big data, particularly relates to the technical field of data analysis, and effectively improves the execution efficiency, success rate and optimizability of a big data operation task by integrating task monitoring, analysis, performance judgment and an intelligent prediction model, reduces the troubleshooting cost, and improves the operation efficiency. Powerful technical support is provided for the information management system under the cloud computing and big data environment, and the problem of high troubleshooting cost of the information management system in the prior art is solved; through the computing power supply quality analysis module, the computing power performance monitoring module and the network performance monitoring module, comprehensive monitoring and troubleshooting of computing power resources, computing power nodes and network performance are achieved, potential problems can be found and processed in time, and smooth proceeding of big data operation tasks is ensured.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and more specifically, to an information management system and method based on cloud computing and big data. Background Art

[0002] Big data jobs usually refer to a series of data processing tasks performed on a big data processing platform. These tasks may include data cleaning, conversion, aggregation, analysis and other operations, aiming to extract valuable information and insights from raw data. When executing big data jobs, you need to configure the corresponding resources, such as computing nodes, memory, disk space, etc. The configuration of these resources will directly affect the execution efficiency and performance of the job.

[0003] Cloud computing platforms can dynamically allocate and release computing resources based on the needs of big data jobs. This elastic scalability enables big data jobs to quickly obtain sufficient computing power when needed, and to release resources in a timely manner when resource requirements decrease, thus avoiding resource waste.

[0004] In big data processing, cloud platforms can improve the speed of data processing and analysis through the computing resources they provide (such as CPU, GPU, memory, etc.). At the same time, cloud platforms can also provide network resources, such as network bandwidth, to ensure that users can efficiently access and use various services on the cloud platform.

[0005] However, it still has many shortcomings in actual use, such as the high troubleshooting cost of existing information management systems. Specifically, when processing big data, cloud platforms generate a large amount of log and monitoring data. These data are not only huge in quantity, but also in various formats and types, making analysis and interpretation difficult. Effective troubleshooting often requires in-depth analysis and mining of these data. Summary of the invention

[0006] In order to overcome the above-mentioned defects of the prior art, the present invention provides an information management system and method based on cloud computing and big data to solve the problems raised in the above-mentioned background technology.

[0007] To achieve the above objectives, the present invention provides the following technical solutions: an information management system based on cloud computing and big data, comprising: Task execution process monitoring module, used to monitor the execution process data of big data job tasks; The task execution analysis module predicts the expected computing time and expected transmission time, and jointly analyzes the computing time difference, transmission time difference and success rate of big data job tasks to obtain the task execution optimizability index; The task execution performance judgment module is used to judge the relationship between the task execution optimizability index and the threshold. If the task execution optimizability index is lower than the threshold, it indicates that the big data job task execution is abnormal, and an abnormal analysis instruction is generated; If the task execution optimizability index is not lower than the threshold, it indicates that the big data job task is executed normally and no action is taken.

[0008] Preferably, the big data job task execution process includes the following steps: Step S11: Obtain the computing power requirements corresponding to the big data job tasks by parsing the big data job tasks, wherein the computing power requirements include the quantity, type, and performance requirements of CPU, GPU, and storage resources; Step S12: perform task scheduling based on computing power requirements, and divide the big data job task into several subtasks according to logical definitions; Step S13: Match a computing power node cluster for each subtask, link the computing power node clusters based on the logical definition, and form a big data operation network topology structure; Step S14: After all subtasks are completed, the output results of the subtasks are summarized to obtain the output results corresponding to the big data job task.

[0009] Preferably, the task execution optimizability index is obtained in the following manner: Assume that the big data job task includes subtasks, using Indicates the sequential number of the subtask. The subtask is recorded as ; The first The expected computing time and actual computing time of each subtask in the corresponding computing node cluster are recorded as , ; The first The expected transmission time and actual transmission time of the data of each subtask from the current location to the corresponding computing node cluster are recorded as , ; The optimizable parameters for task execution are calculated using the following formula ; in, Indicates the success rate of big data job tasks.

[0010] Preferably, the expected computing time is obtained by analyzing the execution time of historical similar tasks (i.e., tasks with similar task type, data scale, task goal, and computing complexity) on the same or similar computing power node cluster to estimate the expected computing time of the new task; The expected transmission time is obtained by calculating the expected transmission time based on the network bandwidth and data volume; analyzing the transmission time of the same or similar data under the same network conditions in the past to obtain an estimated value of the expected transmission time; or, before the actual transmission, performing a transmission test of a small-size data packet, and calculating the expected transmission time corresponding to the subtask data size based on the test results.

[0011] Preferably, the abnormal analysis instructions include one or more of computing resource supply troubleshooting, computing node troubleshooting, and network performance troubleshooting.

[0012] Preferably, the system further comprises a computing power supply quality analysis module, which is used to execute computing power resource supply fault troubleshooting instructions and take measures according to the analysis results; The method for troubleshooting the computing power resource supply fault is: Obtain computing power demand and supply information in big data tasks; The computing power resource supply quality index is calculated by the following formula ; , , in, Indicates the weight coefficient corresponding to the subtask; Indicates The computing power satisfaction of each subtask is Indicates The computing power supply of each subtask, Indicates The computing power required for each subtask, The function is used to get the minimum value of the values ​​in brackets; If the quality of computing power resource supply is lower than the threshold, it indicates that the computing power resource supply is abnormal and cannot meet the needs of big data operation tasks; dispatch more computing power resources to meet the demand and increase computing power redundancy.

[0013] Preferably, the system further includes a computing power performance monitoring module, which is used to execute computing power node troubleshooting instructions and take measures based on analysis results; The method of troubleshooting the computing power node is as follows: Obtain the abnormal log information of the computing power node from the execution data of the big data job task execution process, and obtain the running time, restart times, and crash times of the computing power node to evaluate the stability of the node; The running time, restart times, and crash times of the computing power node after dimension normalization are recorded as, , , ; The node stability index is calculated by the following formula , in, , is the weight coefficient of restart and crash, which is used to adjust the influence of restart and crash times on the node stability index; The state of computing power performance is judged according to the size and change trend of the node stability index. The higher the node stability index, the better the node stability; the lower the node stability index, the worse the node stability; the change trend of the node stability index refers to the change of the node stability index over time; if the node stability index shows a downward trend, it means that the stability of the node is gradually decreasing; if the index shows an upward trend, it means that the stability of the node is gradually increasing; by observing the change trend of the node stability index, it is judged whether the stability of the node is within the controllable range.

[0014] Preferably, the system further comprises a network performance monitoring module for executing network performance troubleshooting instructions and taking measures according to the analysis results; The network performance troubleshooting method is: Monitor network data during the execution of big data job tasks to obtain the average network delay, packet loss rate, and bandwidth utilization coefficient between node clusters; The average network delay, packet loss rate, and bandwidth utilization coefficient between node clusters are recorded as , , ; The network connectivity quality index is calculated by the following formula ; in, Indicates the maximum acceptable value of network delay. Indicates the preset bandwidth utilization factor; , , is the weight coefficient, which is used to adjust the influence of each parameter on the network connectivity quality index; The network performance status is judged based on the size and change trend of the network connectivity quality index. If the index exceeds the corresponding threshold or shows an upward trend, it indicates that there is a problem with the network performance and corresponding measures need to be taken to investigate and repair it.

[0015] Preferably, the bandwidth utilization factor The way to obtain is: Monitor the data access records in the network topology of the big data operation, obtain the network bandwidth and data size information corresponding to each access, and record the time consumed to complete the data access; the consumed time refers to the round-trip communication delay, that is, the time from sending a message to receiving the confirmation of the message; Set up Data access behavior, using The sequential number representing the access behavior, the network bandwidth and data size information, and the time consumed to complete the data access are recorded as , , ; The set of behaviors for data storage is recorded as , the behavior set of data calls is recorded as ; The bandwidth utilization coefficient is calculated by the following formula , in, Function represents a linear normalization function, which is used to perform linear normalization on the content in brackets so that the value is in the range of 0 to 1; Indicates the upper limit of the size of the transmitted data; , They represent the storage weight coefficient and the call weight coefficient respectively, which can be set based on actual conditions, such as calculating according to the proportion of the cumulative big data size involved in storage and call operations.

[0016] Preferably, the system further comprises an intelligent alarm module, which warns of other abnormalities if the computing power resource supply quality index, network connectivity quality index and node stability index are all normal; if the computing power resource supply quality index, network connectivity quality index and node stability index are lower than corresponding thresholds, it indicates that corresponding faults exist; The intelligent alarm module includes an operation health monitoring and early warning unit, which jointly analyzes the computing resource supply quality index, network connectivity quality index, and node stability index to obtain the operation health coefficient, and quantitatively evaluates the information management quality based on the operation health coefficient.

[0017] To achieve the above object, the present invention provides the following technical solution: an information management method based on cloud computing and big data, comprising the following steps: Step S001: Monitor the execution process data of the big data task Step S002: Jointly analyze the computing time difference, transmission time difference and success rate of the big data task to obtain the task execution optimizability index. Step S003: determine the relationship between the task execution optimizability index and the threshold. If the task execution optimizability index is lower than the threshold, generate an abnormal analysis instruction; if the task execution optimizability index is not lower than the threshold, do not take any measures.

[0018] Preferably, executing the abnormal analysis instruction obtains the computing power resource supply quality index, the network connectivity quality index, and the node stability index; The method for obtaining the computing power resource supply quality index is as follows: Get the weight coefficient corresponding to the subtask , No. Subtask computing power satisfaction , No. The computing power supply of each subtask , No. Computing power requirements of each subtask , through the formula , , get; The network connectivity quality index is obtained by: denoting the average network delay, packet loss rate, and bandwidth utilization coefficient between node clusters as , , ; Get the maximum acceptable value of network delay and the preset bandwidth utilization factor ; Through the formula Calculate the network connectivity quality index ;in, , , is the weight coefficient, which is used to adjust the influence of each parameter on the network connectivity quality index; The node stability index is obtained by recording the running time, restart times, and crash times of the computing power node after dimension normalization as, , , ; Through the formula Calculate the node stability index ,in, , is the weight coefficient of restart and crash, which is used to adjust the influence of restart and crash times on the node stability index; The operational health factor The calculation formula is ; The operational health coefficient is used to quantitatively evaluate the operational health of the big data operation process and assist decision making based on the operational health, which helps administrators or operation and maintenance personnel understand the current status of the system.

[0019] Technical effects and advantages of the present invention: The information management system based on cloud computing and big data provided by the present invention effectively improves the execution efficiency, success rate and optimizability of big data operation tasks by integrating task monitoring, analysis, performance judgment and intelligent prediction models, reduces the cost of troubleshooting, and provides strong technical support for information management systems in cloud computing and big data environments, solving the problem of high troubleshooting costs in information management systems in the prior art.

[0020] The information management system based on cloud computing and big data provided by the present invention realizes comprehensive monitoring and troubleshooting of computing power resources, computing power nodes and network performance through a computing power supply quality analysis module, a computing power performance monitoring module and a network performance monitoring module. By adding an intelligent alarm module and an operation health monitoring and early warning unit, comprehensive monitoring and intelligent alarms of computing power resources, network performance and computing power nodes are realized. By quantitatively evaluating the operation health, the system provides administrators with more intuitive and accurate operation status information, which helps to further improve the system's operation and maintenance efficiency and management level. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a structural block diagram of the information management system based on the task execution optimizability index of the present invention.

[0022] Figure 2 This is a flowchart of the big data operation task execution of the present invention. DETAILED DESCRIPTION

[0023] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0024] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0025] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present application, its application, or uses.

[0026] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0027] Example 1, see Figure 1The information management system structure diagram based on the task execution optimizability index is provided. Figure 1 The information management system based on cloud computing and big data shown includes: Task execution process monitoring module, used to monitor the execution process data of big data job tasks; The task execution analysis module uses machine learning to build a prediction model to obtain the expected computing time and expected transmission time; jointly analyzes the computing time difference, transmission time difference and success rate of big data job tasks to obtain the task execution optimizability index; the computing time difference refers to the difference between the expected computing time and the actual computing time, and the transmission time difference refers to the difference between the expected transmission time and the actual transmission time; The task execution performance judgment module is used to judge the relationship between the task execution optimizability index and the threshold. If the task execution optimizability index is lower than the threshold, it indicates that the big data job task execution is abnormal, and an abnormal analysis instruction is generated; If the task execution optimizability index is not lower than the threshold, it indicates that the big data job task is executed normally and no action is taken.

[0028] See also Figure 2 The big data job task execution flow chart includes the following steps: Step S11: Obtain the computing power requirements corresponding to the big data job tasks by parsing the big data job tasks, wherein the computing power requirements include the quantity, type, and performance requirements of CPU, GPU, and storage resources; Step S12: perform task scheduling based on computing power requirements, and divide the big data job task into several subtasks according to logical definitions; Step S13: Match a computing power node cluster for each subtask, link the computing power node clusters based on the logical definition, and form a big data operation network topology structure; Step S14: After all subtasks are completed, the output results of the subtasks are summarized to obtain the output results corresponding to the big data job task.

[0029] Explanation: A computing node cluster usually consists of multiple high-performance computing nodes, each of which has a certain computing power; Explanation: The process of linking the computing power node cluster includes: Optimize network topology to ensure data transmission paths between nodes are as short as possible with the lowest latency; Select data transfer protocols to ensure the integrity and security of data during transmission; Implement cybersecurity measures to prevent data breaches and security attacks.

[0030] Explain that the aggregation process may involve merging, deduplication, sorting, and analysis and visualization of data; generating a complete output result based on the requirements of the task for use by users or subsequent processes.

[0031] It needs to be further explained in the embodiment of the present invention that the task execution optimizability index is obtained in the following manner: Assume that the big data job task includes subtasks, using Indicates the sequential number of the subtask. The subtask is recorded as ; The first The expected computing time and actual computing time of each subtask in the corresponding computing node cluster are recorded as , ; The first The expected transmission time and actual transmission time of the data of each subtask from the current location to the corresponding computing node cluster are recorded as , ; The optimizable parameters for task execution are calculated using the following formula ; in, Indicates the success rate of big data job tasks.

[0032] It needs to be further explained in the embodiment of the present invention that the expected computing time is obtained by analyzing the execution time of historical similar tasks (i.e., tasks with similar task types, data scales, task objectives, and computing complexities) on the same or similar computing power node clusters to estimate the expected computing time of the new task; The expected transmission time is obtained by calculating the expected transmission time based on the network bandwidth and data volume; analyzing the transmission time of the same or similar data under the same network conditions in the past to obtain an estimated value of the expected transmission time; or, before the actual transmission, performing a transmission test of a small-size data packet, and calculating the expected transmission time corresponding to the subtask data size based on the test results.

[0033] What needs to be further explained in the embodiments of the present invention is that the abnormal analysis instructions include one or more of computing resource supply troubleshooting, computing node troubleshooting, and network performance troubleshooting.

[0034] In a possible embodiment, a prediction model is built using machine learning to obtain expected computing time and expected transmission time, wherein the operation of the expected computing time prediction model includes: Data preparation: Collect a large amount of data on historically similar tasks, extract features such as task type, data size, and computational complexity, and perform preprocessing; Model building: Select appropriate machine learning models based on data characteristics, such as linear regression, neural network, etc., and use historical data sets for training; Model evaluation: Evaluate model performance through cross-validation method and adjust parameters to improve prediction accuracy; Prediction application: extract the features of new tasks, input them into the trained model, predict the expected computing time, and reasonably arrange computing resources and task scheduling; The operation of the expected transmission time prediction model includes: Data preparation: Collect the transmission time data of the same or similar data under the same network conditions in the past, extract the network bandwidth and data volume characteristics, and perform preprocessing; Model building: Select appropriate machine learning models, such as linear regression and support vector machines, based on the characteristics of transmission time and data distribution, and conduct training; Prediction application: Extract the transmission characteristics of new tasks, input them into the trained model, predict the expected transmission time, and reasonably arrange data transmission and task scheduling.

[0035] Summary: The embodiments of the present invention effectively improve the execution efficiency, success rate and optimizability of big data job tasks and reduce the cost of troubleshooting by integrating task monitoring, analysis, performance judgment and intelligent prediction models. It provides strong technical support for information management systems in cloud computing and big data environments and solves the problem of high troubleshooting costs in information management systems in the prior art.

[0036] Embodiment 2 is different from Embodiment 1 in that the information management system based on cloud computing and big data may include: The computing power supply quality analysis module is used to execute computing power resource supply troubleshooting instructions and take measures based on the analysis results; The method for troubleshooting the computing power resource supply fault is: Obtain computing power demand and supply information in big data tasks; The computing power resource supply quality index is calculated by the following formula ; , , in, Indicates the weight coefficient corresponding to the subtask; Indicates The computing power satisfaction of each subtask is Indicates The computing power supply of each subtask, Indicates The computing power required for each subtask, The function is used to get the minimum value of the values ​​in brackets; If the quality of computing power resource supply is lower than the threshold, it indicates that the computing power resource supply is abnormal and cannot meet the needs of big data operation tasks; dispatch more computing power resources to meet the demand and increase computing power redundancy.

[0037] In a possible embodiment, the system includes a computing power performance monitoring module, which is used to execute computing power node fault troubleshooting instructions and take measures according to analysis results; The method of troubleshooting the computing power node is as follows: Obtain the abnormal log information of the computing power node from the execution data of the big data job task execution process, and obtain the running time, restart times, and crash times of the computing power node to evaluate the stability of the node; The running time, restart times, and crash times of the computing power node after dimension normalization are recorded as, , , ; The node stability index is calculated by the following formula , in, , is the weight coefficient of restart and crash, which is used to adjust the influence of restart and crash times on the node stability index; The state of computing power performance is judged according to the size and change trend of the node stability index. The higher the node stability index, the better the node stability; the lower the node stability index, the worse the node stability; the change trend of the node stability index refers to the change of the node stability index over time; if the node stability index shows a downward trend, it means that the stability of the node is gradually decreasing; if the index shows an upward trend, it means that the stability of the node is gradually increasing; by observing the change trend of the node stability index, it is judged whether the stability of the node is within the controllable range.

[0038] The explanation shows that the three key indicators of the computing power node are taken into consideration: the running time, number of restarts and number of crashes. When the number of restarts and crashes is fixed, the longer the running time, the longer the node works in a stable state. The fewer the number of restarts and crashes, the faster the node can recover or less frequently encounter problems. By combining these factors, a comprehensive stability assessment can be given.

[0039] In a possible embodiment, the system includes a network performance monitoring module for executing network performance troubleshooting instructions and taking measures based on analysis results; The network performance troubleshooting method is: Monitor network data during the execution of big data job tasks to obtain the average network delay, packet loss rate, and bandwidth utilization coefficient between node clusters; The average network delay, packet loss rate, and bandwidth utilization coefficient between node clusters are recorded as , , ; The network connectivity quality index is calculated by the following formula ; in, Indicates the maximum acceptable value of network delay. Indicates the preset bandwidth utilization factor; , , is the weight coefficient, which is used to adjust the influence of each parameter on the network connectivity quality index; The network performance status is judged based on the size and change trend of the network connectivity quality index. If the index exceeds the corresponding threshold or shows an upward trend, it indicates that there is a problem with the network performance and corresponding measures need to be taken to investigate and repair it. Specific measures may include adjusting the network configuration, optimizing the data transmission strategy, and increasing the network bandwidth.

[0040] It is necessary to further explain in the embodiment of the present invention that the bandwidth utilization coefficient The way to obtain is: Monitor the data access records in the network topology of the big data operation, obtain the network bandwidth and data size information corresponding to each access, and record the time consumed to complete the data access; the consumed time refers to the round-trip communication delay, that is, the time from sending a message to receiving the confirmation of the message; Set up Data access behavior, using The sequential number representing the access behavior, the network bandwidth and data size information, and the time consumed to complete the data access are recorded as , , ; The set of behaviors for data storage is recorded as , the behavior set of data calls is recorded as ; The bandwidth utilization coefficient is calculated by the following formula , in, Function represents a linear normalization function, which is used to perform linear normalization on the content in brackets so that the value is in the range of 0 to 1; Indicates the upper limit of the size of the transmitted data; , They represent the storage weight coefficient and the call weight coefficient respectively, which can be set based on actual conditions, such as calculating according to the proportion of the cumulative big data size involved in storage and call operations.

[0041] Summary: The embodiments of the present invention realize comprehensive monitoring and troubleshooting of computing resources, computing nodes and network performance through a computing power supply quality analysis module, a computing power performance monitoring module and a network performance monitoring module. Through specific troubleshooting methods, potential problems can be discovered and handled in a timely manner, ensuring the smooth progress of big data operations.

[0042] Embodiment 3, the intelligent alarm module is also included, if the computing power resource supply quality index, network connectivity quality index, and node stability index are all normal, an alarm is issued to indicate other abnormalities; If the computing power resource supply quality index, network connectivity quality index, and node stability index are lower than the corresponding thresholds, it indicates that the corresponding fault exists; The intelligent alarm module includes an operation health monitoring and early warning unit, which obtains the computing power resource supply quality index, network connectivity quality index, and node stability index by executing abnormal analysis instructions; The method for obtaining the computing power resource supply quality index is as follows: Get the weight coefficient corresponding to the subtask , No. Subtask computing power satisfaction , No. The computing power supply of each subtask , No. Computing power requirements of each subtask , through the formula , , get; The network connectivity quality index is obtained by: denoting the average network delay, packet loss rate, and bandwidth utilization coefficient between node clusters as , , ; Get the maximum acceptable value of network delay and the preset bandwidth utilization factor ; Through the formula Calculate the network connectivity quality index ;in, , , is the weight coefficient, which is used to adjust the influence of each parameter on the network connectivity quality index; The node stability index is obtained by recording the running time, restart times, and crash times of the computing power node after dimension normalization as, , , ; Through the formula Calculate the node stability index ,in, , is the weight coefficient of restart and crash, which is used to adjust the influence of restart and crash times on the node stability index; Jointly analyze the computing resource supply quality index, network connectivity quality index, and node stability index to obtain the operation health coefficient, and quantitatively evaluate the information management quality based on the operation health coefficient; The operational health factor The calculation formula is ; The operational health coefficient is used to quantitatively evaluate the operational health of the big data operation process and assist decision making based on the operational health, which helps administrators or operation and maintenance personnel understand the current status of the system.

[0043] Summary: The embodiment of the present invention realizes comprehensive monitoring and intelligent alarm of computing resources, network performance and computing nodes by adding intelligent alarm modules and operation health monitoring and early warning units. The system can timely discover and handle potential problems, ensure the smooth progress of big data operation tasks, and improve the reliability and stability of the system. In addition, by quantitatively evaluating the operation health, the system provides administrators with more intuitive and accurate operation status information, which helps to further improve the operation and maintenance efficiency and management level of the system.

[0044] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An information management system based on cloud computing and big data, characterized by: include: Task execution process monitoring module, used to monitor the execution process data of big data job tasks; The task execution analysis module predicts the expected computing time and expected transmission time, and jointly analyzes the computing time difference, transmission time difference and success rate of big data job tasks to obtain the task execution optimizability index; The task execution performance judgment module is used to judge the relationship between the task execution optimizability index and the threshold. If the task execution optimizability index is lower than the threshold, it indicates that the big data job task execution is abnormal, and an abnormal analysis instruction is generated; If the task execution optimizability index is not lower than the threshold, it indicates that the big data job task is executed normally and no action is taken.

2. The information management system based on cloud computing and big data according to claim 1, characterized in that: The big data job task execution process includes the following steps: Step S11: Obtain the computing power requirements corresponding to the big data job tasks by parsing the big data job tasks, where the computing power requirements include the number, type, and performance requirements of CPU, GPU, and storage resources; Step S12: perform task scheduling based on computing power requirements, and divide the big data job tasks into several subtasks according to logical definitions; Step S13: Match a computing power node cluster for each subtask, link the computing power node clusters based on the logical definition, and form a big data operation network topology structure; Step S14: After all subtasks are completed, the output results of the subtasks are summarized to obtain the output results corresponding to the big data job task.

3. The information management system based on cloud computing and big data according to claim 2, characterized in that: The task execution optimizability index is obtained in the following manner: Assume that the big data job task includes subtasks, using Indicates the sequential number of the subtask. The subtask is recorded as ; The first The expected computing time and actual computing time of each subtask in the corresponding computing node cluster are recorded as , ; The first The expected transmission time and actual transmission time of the data of each subtask from the current location to the corresponding computing node cluster are recorded as , ; The task execution optimizable parameters are calculated by the following formula ; in, Indicates the success rate of big data job tasks.

4. The information management system based on cloud computing and big data according to claim 1, characterized in that: The abnormal analysis instructions include one or more of computing resource supply troubleshooting, computing node troubleshooting, and network performance troubleshooting.

5. The information management system based on cloud computing and big data according to claim 4, characterized in that: It also includes a computing power supply quality analysis module for executing computing power resource supply troubleshooting instructions and taking measures based on the analysis results; The method for troubleshooting the computing power resource supply fault is: Obtain computing power demand information and computing power supply information in big data tasks; calculate the computing power resource supply quality index through the following formula ; , , in, Indicates the weight coefficient corresponding to the subtask; Indicates The computing power satisfaction of each subtask is Indicates The computing power supply of each subtask, Indicates The computing power required for each subtask, The function is used to get the minimum value of the values ​​in brackets; If the quality of computing power resource supply is lower than the threshold, it indicates that the computing power resource supply is abnormal and cannot meet the needs of big data operation tasks; dispatch more computing power resources to meet the demand and increase computing power redundancy.

6. The information management system based on cloud computing and big data according to claim 4, characterized in that: It also includes a computing power performance monitoring module, which is used to execute computing power node troubleshooting instructions and take measures based on the analysis results; The method of troubleshooting the computing power node is as follows: Obtain the abnormal log information of the computing power node from the execution data of the big data job task execution process, and obtain the running time, restart times, and crash times of the computing power node to evaluate the stability of the node; The running time, restart times, and crash times of the computing power node after dimension normalization are recorded as, , , ; The node stability index is calculated by the following formula , in, , is the weight coefficient of restart and crash, which is used to adjust the influence of restart and crash times on the node stability index; The state of computing power performance is judged according to the size and change trend of the node stability index. The higher the node stability index, the better the node stability; the lower the node stability index, the worse the node stability; by observing the change trend of the node stability index, it can be judged whether the stability of the node is within the controllable range.

7. The information management system based on cloud computing and big data according to claim 4, characterized in that: It also includes a network performance monitoring module for executing network performance troubleshooting instructions and taking measures based on the analysis results; The network performance troubleshooting method is: Monitor network data during the execution of big data job tasks to obtain the average network delay, packet loss rate, and bandwidth utilization coefficient between node clusters; The average network delay, packet loss rate, and bandwidth utilization coefficient between node clusters are respectively , , ; The network connectivity quality index is calculated by the following formula ; in, Indicates the maximum acceptable value of network delay. Indicates the preset bandwidth utilization factor; , , is the weight coefficient, which is used to adjust the influence of each parameter on the network connectivity quality index; The network performance status is judged based on the size and change trend of the network connectivity quality index. If the index exceeds the corresponding threshold or shows an upward trend, it indicates that there is a problem with the network performance and corresponding measures need to be taken to investigate and repair it.

8. The information management system based on cloud computing and big data according to claim 7, characterized in that: The bandwidth utilization factor The way to obtain is: Monitor the data access records in the network topology of the big data operation, obtain the network bandwidth and data size information corresponding to each access, and record the time consumed to complete the data access; the consumed time refers to the round-trip communication delay, that is, the time from sending a message to receiving the confirmation of the message; Set up Data access behavior, using The sequential number representing the access behavior, the network bandwidth and data size information, and the time consumed to complete the data access are recorded as , , ; The set of behaviors for data storage is recorded as , the behavior set of data calls is recorded as ; The bandwidth utilization coefficient is calculated by the following formula , in, Function represents a linear normalization function, which is used to perform linear normalization on the content in brackets so that the value is in the range of 0 to 1; Indicates the upper limit of the size of the transmitted data; , They represent the storage weight coefficient and the call weight coefficient respectively, which can be set based on actual conditions, such as calculating according to the proportion of the cumulative big data size involved in storage and call operations.

9. An information management method based on cloud computing and big data, characterized in that: The following steps are involved: Step S001: Monitor the execution process data of the big data task Step S002: Jointly analyze the computing time difference, transmission time difference and success rate of the big data task to obtain the task execution optimizability index. Step S003: determine the relationship between the task execution optimizability index and the threshold. If the task execution optimizability index is lower than the threshold, generate an abnormal analysis instruction; if the task execution optimizability index is not lower than the threshold, do not take any measures.

10. The information management method based on cloud computing and big data according to claim 9, characterized in that: Execute the abnormal analysis command to obtain the computing power resource supply quality index, network connectivity quality index, and node stability index; Joint analysis of computing power resource supply quality index , Network Connectivity Quality Index , Node Stability Index , get the operation health coefficient, and quantitatively evaluate the information management quality based on the operation health coefficient; The operational health factor The calculation formula is ; Quantitatively evaluate the operational health of the big data operation process based on the operational health coefficient.

Images