Business data analysis method and device based on big data, equipment and storage medium

Abstract

The invention relates to the technical field of data analysis, and discloses a business data analysis method, device and equipment based on big data and a storage medium, the method comprises the following steps: collecting and preprocessing original log data to obtain standardized log data, and performing adaptive multi-modal feature extraction to obtain comprehensive feature representation; inputting a heterogeneous graph neural network to carry out anomaly detection and multi-level root cause analysis to obtain a hierarchical anomaly event; executing task scheduling and resource allocation of reinforcement learning to obtain a task allocation scheme; performing deep learning enhanced root cause reasoning and knowledge graph construction on the task allocation scheme to obtain a system state diagram and a causal relationship; and generating a multi-dimensional task data analysis report. According to the method, comprehensive analysis, anomaly detection, root cause reasoning and intelligent scheduling of log data are realized. The reliability and performance of the system can be improved, and more intuitive and accurate decision support can be provided for operation and maintenance personnel.

Description

Technical Field

[0001] The present invention relates to the field of data analysis technology, and in particular to a business data analysis method, device, equipment and storage medium based on big data. Background Art

[0002] With the rapid development of information technology, the log data generated by various systems and applications has grown exponentially. These log data contain key information about system operation, user behavior, and potential problems, and are of great value for system maintenance, performance optimization, and security management. However, traditional log analysis methods face challenges such as large data volume, complex structure, and high real-time requirements, making it difficult to effectively and quickly extract valuable information from massive logs.

[0003] Existing log analysis technologies often use a single analysis model or a fixed set of rules, which are difficult to adapt to complex and changing system environments and diverse anomaly patterns. This leads to insufficient accuracy in anomaly detection and failure to detect potential system problems in a timely manner. In addition, traditional methods also have limitations in root cause analysis and resource scheduling, making it difficult to deeply explore the root causes of problems, and unable to perform intelligent task scheduling and resource allocation based on dynamically changing system states. Summary of the invention

[0004] The present invention provides a business data analysis method, device, equipment and storage medium based on big data, which is used to realize comprehensive analysis, anomaly detection, root cause reasoning and intelligent scheduling of log data. This can not only improve the reliability and performance of the system, but also provide more intuitive and accurate decision support for operation and maintenance personnel.

[0005] In a first aspect, the present invention provides a business data analysis method based on big data, the business data analysis method based on big data comprising: Collecting and preprocessing the original log data to obtain standardized log data, and performing adaptive multimodal feature extraction on the standardized log data to obtain a comprehensive feature representation; Inputting the comprehensive feature representation into a heterogeneous graph neural network for anomaly detection and multi-level root cause analysis to obtain hierarchical abnormal events; According to the hierarchical abnormal events, perform task scheduling and resource allocation of reinforcement learning to obtain a task allocation plan; Performing deep learning enhanced root cause reasoning and knowledge graph construction on the task allocation scheme to obtain a system state diagram and causal relationship; A multi-dimensional task data analysis report is generated based on the system state diagram and the causal relationship.

[0006] In a second aspect, the present invention provides a business data analysis device based on big data, the business data analysis device based on big data comprising: A collection module, used to collect and preprocess the original log data to obtain standardized log data, and perform adaptive multimodal feature extraction on the standardized log data to obtain a comprehensive feature representation; An analysis module, used for inputting the comprehensive feature representation into a heterogeneous graph neural network to perform anomaly detection and multi-level root cause analysis to obtain hierarchical abnormal events; An allocation module, configured to perform task scheduling and resource allocation of reinforcement learning according to the hierarchical abnormal events to obtain a task allocation plan; A construction module is used to perform deep learning enhanced root cause reasoning and knowledge graph construction on the task allocation scheme to obtain a system state diagram and causal relationship; A generation module is used to generate a multi-dimensional task data analysis report based on the system state diagram and the causal relationship.

[0007] The third aspect of the present invention provides a business data analysis device based on big data, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory so that the business data analysis device based on big data executes the above-mentioned business data analysis method based on big data.

[0008] A fourth aspect of the present invention provides a computer-readable storage medium, which stores instructions that, when executed on a computer, enable the computer to execute the above-mentioned business data analysis method based on big data.

[0009] In the technical solution provided by the present invention, by performing adaptive multimodal feature extraction on the original log data, the comprehensive utilization of text, time series and structural features is realized, and the richness and accuracy of feature representation are improved. The use of heterogeneous graph neural networks for anomaly detection and multi-level root cause analysis can effectively capture the complex relationship between log events, improve the accuracy of anomaly detection and the depth of root cause analysis. The introduction of reinforcement learning for task scheduling and resource allocation realizes adaptive decision-making based on dynamic system status and optimizes the efficiency of system resource utilization. Through deep learning-enhanced root cause reasoning and knowledge graph construction, a more comprehensive and explainable system state diagram and causal relationship are provided, which helps operation and maintenance personnel to quickly locate and solve problems. Full automation from data collection to report generation is achieved, which greatly improves the efficiency and real-time performance of log analysis. The present invention realizes comprehensive analysis, anomaly detection, root cause reasoning and intelligent scheduling of log data. This can not only improve the reliability and performance of the system, but also provide more intuitive and accurate decision support for operation and maintenance personnel.

[0010] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0011] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A schematic diagram of an embodiment of a business data analysis method based on big data in an embodiment of the present invention; Figure 2 A schematic diagram of an embodiment of a business data analysis device based on big data in an embodiment of the present invention; Figure 3 It is a schematic diagram of an embodiment of a business data analysis device based on big data in an embodiment of the present invention. DETAILED DESCRIPTION

[0013] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] The terms "including" and "having" and any variations thereof mentioned in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device end including a series of steps or units is not limited to the listed steps or units, but may optionally include other steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or device ends.

[0015] To facilitate understanding of this embodiment, a business data analysis method based on big data disclosed in an embodiment of the present invention is first described in detail. Figure 1 As shown, the method comprises the following steps: 101. Collect and preprocess the original log data to obtain standardized log data, and perform adaptive multimodal feature extraction on the standardized log data to obtain a comprehensive feature representation; It is understandable that the execution subject of the present invention may be a business data analysis device based on big data, or may be a terminal or a server, which is not specifically limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0016] Specifically, by deploying multiple data collection agents on different systems and services, multi-source log data is distributedly collected, and log data is collected in real time and sent to centralized storage. To ensure data integrity and accuracy, a batch or streaming transmission mechanism is adopted, and an appropriate buffer is set to cope with network fluctuations. Regular expression matching and keyword extraction are performed on the original log data set. The regular expression library in the programming language is used to parse the original log data to identify specific patterns and keywords, and generate structured log entries, including necessary fields (such as timestamps, event types, user IDs, etc.). The structured log entries are sorted and aggregated by timestamp and processed using a database or memory data structure (such as Pandas DataFrame). By aggregating events in the same time period, a time series log sequence is generated, and statistical analysis is performed to calculate the frequency and time distribution of events to extract basic statistical features. The predefined parsing rule library is applied to semantically parse the structured logs to extract the operation type and error code. The log content is analyzed using natural language processing technology and semantic annotation data is generated. For the generation of log text representation, the semantic annotation data is input into a word embedding model (such as Word2Vec or BERT) to convert it into a vector representation. When processing time series log sequences, the time intervals and event sequence patterns are extracted through sliding window analysis. The size of the window is set according to specific needs to balance the granularity of the analysis and computational efficiency. At the same time, the field association graph is constructed using a graph database or graph processing framework (such as Neo4j), with semantically annotated data as nodes and edges constructed to represent the associations between fields. The text representation, time series features, and structural features are input into the multimodal feature fusion network. The network is built through a deep learning framework (such as TensorFlow or PyTorch), and the attention mechanism is used to enhance the representation ability of features. Dynamic weight allocation is automatically adjusted through training models to enhance the effectiveness of comprehensive feature representation.

[0017] 102. Input the comprehensive feature representation into the heterogeneous graph neural network for anomaly detection and multi-level root cause analysis to obtain hierarchical abnormal events; Specifically, the comprehensive feature representation is converted into a heterogeneous graph structure, in which nodes represent log events and edges represent the relationship between events, forming an initial heterogeneous graph, so that the mutual relationship between different events can be effectively captured and analyzed. The initial heterogeneous graph is input into the graph attention layer to calculate the attention weights between nodes. By introducing the attention mechanism, the importance relationship between different nodes is determined to obtain a weighted heterogeneous graph. The weighted heterogeneous graph is input into the multi-head graph attention layer. Each attention head independently calculates node features to capture different relationships and patterns, and then the outputs of multiple heads are spliced ​​together to obtain an enhanced node representation. The enhanced node representation is input into the graph convolution layer to aggregate the information of neighboring nodes, thereby updating the node features and forming a local context-aware node representation. The local context-aware node representation is input into the graph pooling layer for hierarchical clustering and graph structure compression to obtain a multi-scale graph representation, which helps to capture information of different granularities and more comprehensively understand the relationship between nodes. The multi-scale graph representation is skipped and the graph representations of different scales are fused to form multi-granular features. The multi-granular features are input into the graph variational autoencoder for processing. In this stage, the model optimizes the reconstruction error and KL divergence to obtain the anomaly score of each node. By analyzing the anomaly score and based on the dynamic threshold algorithm, it can effectively identify abnormal nodes and their corresponding abnormal subgraphs and construct hierarchical abnormal events.

[0018] Among them, the weighted heterogeneous graph is input into the multi-head graph attention layer. For each node in the weighted heterogeneous graph, the attention coefficient of each node and all adjacent nodes is calculated, and the attention distribution matrix is ​​obtained by modeling the relationship between the nodes. This matrix effectively represents the importance of each node in its neighborhood. Based on the attention distribution matrix, the feature vector of each node is weighted and summed to obtain the single-head attention output. This output represents the feature integration of the node in its local context and reflects the influence of the adjacent nodes on the target node. The single-head attention output is repeated for a preset number of times, and different parameter matrices are used in each iteration to generate multiple sets of independent attention outputs. These independent outputs represent different feature learning perspectives and can capture diverse information. Multiple sets of independent attention outputs are spliced ​​to obtain a spliced ​​feature vector. The spliced ​​feature vector is input into a nonlinear activation function to perform feature transformation to obtain an enhanced node representation. This increases the expressive power of the model through nonlinear activation, enabling it to capture more complex feature patterns. The enhanced node representation is input into the edge-aware convolution kernel to extract the local structural information between the node and its neighboring nodes to form a local feature representation. The role of the edge-aware convolution kernel is to emphasize the structural relationship between nodes and extract local features that are helpful for subsequent analysis. The local feature representation is residually connected with the original node features to effectively fuse the newly obtained local features with the original features to generate fused features. The advantage of residual connection is that it retains the original feature information while introducing local structural information to enhance the representation ability of the node. The fused features are batch normalized to reduce internal covariate shift and improve the stability and efficiency of training. Multi-layer feature information fusion is performed on the batch-normalized fused features to improve the quality of node representation and ensure that the final local context-aware node representation can not only reflect the characteristics of the node itself, but also effectively integrate the information of the surrounding neighborhood.

[0019] 103. According to the hierarchical abnormal events, perform task scheduling and resource allocation of reinforcement learning to obtain a task allocation plan; Specifically, the hierarchical abnormal events are converted into state vectors, which are used as the input of the reinforcement learning model to form a representation of the current system state. By analyzing the hierarchical abnormal events, key features are extracted and a vector that can reflect the current state of the system is constructed. Based on the current system state representation, a set of candidate actions is generated using a policy network, where each action corresponds to a possible task allocation strategy, and the initial action space is obtained. The initial action space is subjected to action masking operations to exclude actions that do not meet the system constraints and obtain a valid action set. The valid action set is input into the value network, the expected value of each action is calculated, and the action value evaluation result is obtained, providing a quantitative evaluation index for each action. Based on the action value evaluation result, the ε-greedy strategy is used to select the optimal action. While ensuring exploration, the strategy can use the existing information to make the best decision and obtain the task allocation decision for the current time step. The system resource allocation state is updated according to the task allocation decision, and the corresponding task scheduling operation is performed to obtain a new system state. The update process enables the system to dynamically adjust resource configuration according to real-time decisions to cope with the ever-changing environment. The new system state is input into the reward function, the immediate reward value of the current decision is calculated, and the reward signal is obtained to reflect the effect of the current decision and provide feedback information for the model. Based on the reward signal and the new system state, the parameters of the value network are updated using the temporal difference learning method to obtain the optimized value estimate. The optimized value estimate is used to update the policy network, and this process will continue until the preset convergence condition or the maximum number of iterations is reached. In this process, the model continuously adjusts its own strategy to achieve the best effect in various task allocation scenarios, and finally obtains an effective task allocation solution.

[0020] 104. Perform deep learning-enhanced root cause reasoning and knowledge graph construction on the task allocation scheme to obtain the system state diagram and causal relationship; Specifically, the task allocation scheme is converted into a structured data representation, which is used as the input of root cause reasoning to form the initial reasoning data. The initial reasoning data is input into the deep neural network for feature extraction to generate a feature vector. The extracted feature vector is input into the attention mechanism module to calculate the importance weights of different features and obtain the weighted feature representation. Through this process, the key features are effectively highlighted, which helps to pay more attention to the features that have a greater impact on the relationship between system components in subsequent modeling. Based on the weighted feature representation, the relationship between system components is modeled to obtain a component relationship graph to show the interaction between each component. Causal discovery and causal relationship identification are performed on the component relationship graph to obtain a preliminary causal graph. In this process, by analyzing the interaction between components, potential causal relationships are identified and these relationships are integrated into a preliminary causal graph. The preliminary causal graph is enhanced by logical reasoning to verify and strengthen the identified causal relationships to obtain an enhanced causal graph. Based on the enhanced causal graph, the initial structure of the knowledge graph is constructed, and the system components, events and relationships are represented as entities and edges to form the original knowledge graph. The construction of the original knowledge graph enables various information in the system to be presented in a visual form, which is helpful for subsequent analysis. Perform graph embedding analysis on the original knowledge graph, learn low-dimensional representations of entities and relationships, and obtain embedding vectors. Use the obtained embedding vectors to perform knowledge graph completion tasks, predict potential missing relationships, and obtain the target knowledge graph. Based on the target knowledge graph, perform state diagram visualization and key causal path extraction, and finally obtain the system state diagram and causal relationship.

[0021] Among them, the embedding vector is tensor decomposed to calculate the similarity score between entities and relations. By analyzing the implicit structural information in the embedding vector, a candidate relationship set is generated. The candidate relationship is a potential missing relationship based on similarity. After the candidate relationship set is input into the relationship classifier, the authenticity of each candidate relationship is evaluated to obtain a relationship confidence score. Based on the relationship confidence score, a threshold screening method is used to select high-confidence relationships to form a supplementary relationship set. The supplementary relationship set is merged with the original knowledge graph, and the graph structure is updated to obtain an extended knowledge graph. After the extension is completed, the knowledge graph is checked for consistency to ensure that the information in the graph is accurate and the target knowledge graph is generated. Based on the target knowledge graph, the graph layout algorithm is executed to calculate the position of each node and the routing of the edge to generate an initial visual layout. The force-directed algorithm is applied to the initial visual layout to optimize the distribution of nodes and reduce the intersection of edges to obtain the optimized layout structure. The optimized layout structure is converted into a visual graph, and color coding and legend information are added to enhance the user experience. The system state diagram is obtained, which can effectively convey the relationship between the components in the system and their dynamic changes. At the same time, the path search algorithm is applied to the target knowledge graph to identify the shortest path between key entities, form a set of candidate causal paths, and reveal the key causal relationships in the system. By scoring the path importance of the candidate causal path set, the path with the greatest impact in a specific situation is identified, and then the key causal links are extracted. The key causal links reflect the relationship between different entities and reveal the potential causal mechanism in the system.

[0022] 105. Generate multi-dimensional task data analysis report based on system state diagram and causal relationship.

[0023] Specifically, the system state diagram and the identified causal relationships are comprehensively analyzed. The system state diagram provides an overall view of the task operation, showing the interconnection between each task and its current status, while the causal relationship reveals the dependency and influence of these tasks under different conditions. In order to generate a multi-dimensional analysis report, the attribute data of each node and edge in the system state diagram are collected and sorted, including information such as the execution time, resource consumption, and frequency of abnormal events of the task. Through statistical analysis of these attributes, the performance indicators and operation efficiency of different tasks are revealed. Data visualization technology is used to intuitively display the key data of each element in the state diagram, highlighting key tasks and potential risk points. In the visualization process, color coding and legend information are combined to identify high-risk tasks and abnormal situations that need attention. Add an analysis section based on causal relationships to the report. By parsing the causal chain between tasks and analyzing the impact of specific events on the system state, it helps to identify which factors are the root causes of performance degradation or abnormalities, and provide suggestions for subsequent optimization measures. For example, if a task frequently has abnormalities, the relationship between it and other tasks can be found through causal analysis, and then corresponding improvement plans can be proposed. Integrate the analysis results of each dimension to form a comprehensive and systematic multi-dimensional task data analysis report.

[0024] In the embodiment of the present invention, by performing adaptive multimodal feature extraction on the original log data, the comprehensive utilization of text, time series and structural features is realized, and the richness and accuracy of feature representation are improved. The use of heterogeneous graph neural networks for anomaly detection and multi-level root cause analysis can effectively capture the complex relationship between log events, improve the accuracy of anomaly detection and the depth of root cause analysis. The introduction of reinforcement learning for task scheduling and resource allocation realizes adaptive decision-making based on dynamic system status and optimizes the efficiency of system resource utilization. Through deep learning enhanced root cause reasoning and knowledge graph construction, a more comprehensive and explainable system state diagram and causal relationship are provided, which helps operation and maintenance personnel to quickly locate and solve problems. Full automation from data collection to report generation is realized, which greatly improves the efficiency and real-time performance of log analysis. The present invention realizes comprehensive analysis, anomaly detection, root cause reasoning and intelligent scheduling of log data. This can not only improve the reliability and performance of the system, but also provide more intuitive and accurate decision support for operation and maintenance personnel.

[0025] In a specific embodiment, the process of executing step 101 may specifically include the following steps: Distributed collection of multi-source log data to obtain the original log data set, and regular expression matching and keyword extraction are performed on the original log data set to obtain structured log entries; Sort and aggregate structured log entries by timestamp to obtain a time series log sequence, and perform statistical analysis on the time series log sequence to calculate event frequency and time distribution to obtain basic statistical features; Based on the predefined parsing rule library, the structured log entries are semantically parsed to extract the operation type and error code to obtain semantically annotated data. The semantically annotated data is then input into the word embedding model for text feature recognition to obtain the log text representation. Perform sliding window analysis on time series log sequences to extract time intervals and event sequence patterns to obtain time series features, and construct field association graphs based on semantically annotated data to obtain structural features; The text representation, temporal features and structural features are input into the multimodal feature fusion network to obtain the initial comprehensive features, and the initial comprehensive features are subjected to attention mechanism and dynamic weight allocation to obtain the comprehensive feature representation.

[0026] Specifically, establish an efficient distributed collection system to obtain raw log data from multiple sources simultaneously. By using a data flow framework such as Apache Kafka or Flume, different types of log data from different applications, servers, and services are collected in real-time mode. After the collection is completed, a raw log data set is formed, which contains various types of log information. Regular expression matching and keyword extraction are performed on the raw log data. By defining specific regular expressions to identify important information in the log, such as timestamps, operation types, and error codes, structured log entries are obtained. The structured log entries are sorted and aggregated by timestamp to form a time series log sequence. Data is aggregated by writing aggregation functions, for example, using the groupby function in Python's Pandas library. After sorting and aggregation, statistical analysis is performed on the time series log sequence to calculate the event frequency and time distribution to obtain basic statistical features. For the calculation of event frequency, the formula is used: ; in, represents the event frequency, is the total number of events in a specific time window, is the duration of the time window. Through statistical analysis, the distribution of log data is reflected, and high-frequency events and time periods are identified. Based on the predefined parsing rule base, the structured log entries are semantically parsed, the operation type and error code are extracted, and semantic annotation data is obtained. The semantic annotation data is input into the word embedding model for text feature recognition. The word embedding model captures the semantic information of text data by converting each word into a vector of fixed dimension, and obtains the representation of log text. On this basis, a sliding window analysis is performed on the time series log sequence to extract the time interval and event sequence pattern. By setting a time window and extracting all related events within the window, the time feature is obtained. For example, assuming that the time window is 5 minutes, all events and their timestamps within the time period are collected. Through sliding window analysis, the relative time relationship between events is identified, and more in-depth time series features are obtained. At the same time, based on the semantic annotation data, a field association graph is constructed to show the relationship between different fields, obtain structural features, and reveal the potential patterns and relationships in the log data. The text representation, time series features, and structural features are input into the multimodal feature fusion network to obtain the initial comprehensive features. The fusion network can effectively integrate data from different modalities and improve the expressiveness of features. The attention mechanism and dynamic weight allocation are performed on the initial comprehensive features to improve the performance of the model, so that the model can focus on the most relevant features during the training process and finally obtain a comprehensive feature representation.

[0027] In a specific embodiment, the process of executing step 102 may specifically include the following steps: Convert the comprehensive feature representation into a heterogeneous graph structure, where nodes represent log events and edges represent relationships between events, to obtain an initial heterogeneous graph; Input the initial heterogeneous graph into the graph attention layer to calculate the attention weights between nodes to obtain a weighted heterogeneous graph; The weighted heterogeneous graph is input into the multi-head graph attention layer. Each attention head calculates the node features independently, and then the multi-head outputs are concatenated to obtain the enhanced node representation. The enhanced node representation is input into the graph convolution layer to aggregate the neighbor node information, update the node features, and obtain the local context-aware node representation. The local context-aware node representation is input into the graph pooling layer for hierarchical clustering and graph structure compression to obtain a multi-scale graph representation. The multi-scale graph representation is skip-connected to fuse graph representations of different scales to obtain multi-granularity features. The multi-granularity features are input into the graph variational autoencoder to optimize the reconstruction error and KL divergence to obtain the anomaly score. Based on the anomaly score, a dynamic threshold algorithm is used to identify abnormal nodes and abnormal subgraph construction to obtain hierarchical abnormal events.

[0028] Specifically, the comprehensive feature representation is converted into a heterogeneous graph structure, where nodes represent log events and edges represent the relationship between events. Each log event is regarded as a node, and these nodes are connected by edges. The attributes of the edges represent the strength or type of the relationship between different events. After constructing the initial heterogeneous graph, the graph attention network is used to calculate the attention weights between nodes. The graph attention layer calculates the attention score between nodes through the given features as follows: ; in, Representation Node and nodes The attention score between is the weight vector used to calculate the attention, is the feature transformation matrix, and The nodes are and nodes The characteristic representation of Representation vector concatenation. By calculating the attention scores of all edges, a weighted heterogeneous graph is obtained to enhance the relationship representation between nodes. The weighted heterogeneous graph is input into the multi-head graph attention layer. Each attention head independently calculates the node features to form different attention outputs, which are then spliced ​​together to obtain the enhanced node representation. The enhanced node representation is input into the graph convolution layer. The graph convolution layer can aggregate the information of neighboring nodes to update the features of each node and obtain a local context-aware node representation. By aggregating the information of neighboring nodes, the node can better reflect its position and role in the graph. The local context-aware node representation is input into the graph pooling layer for hierarchical clustering and graph structure compression. The nodes are clustered according to their features and connection relationships to obtain a multi-scale graph representation. In this process, a graph-based aggregation method is used to compress nodes to retain the most important structural information. Skip connections are made to the multi-scale graph representation to fuse graph representations of different scales together to obtain multi-granular features. The multi-granular features are input into the graph variational autoencoder for reconstruction error and KL divergence optimization. The graph variational autoencoder reconstructs the multi-granularity features of the input and calculates the reconstruction error to ensure that the generated features retain the information of the original data as much as possible. At the same time, the KL divergence is used to measure the difference between the distribution generated by the model and the real data distribution. The optimization goal is expressed by the following formula: L ; Among them, L represents the total loss, is the generated latent variable distribution, is the reconstructed distribution, is the Kullback-Leibler divergence, which is used to measure the difference between two probability distributions. The obtained anomaly score can indicate which nodes behave abnormally in the graph. Based on the anomaly score, a dynamic threshold algorithm is used to identify abnormal nodes and abnormal subgraphs, and hierarchical abnormal events are constructed. For example, in log data, assuming that the anomaly scores of some nodes are higher than the set dynamic threshold, these nodes represent abnormal behaviors in the system, such as excessively high error rates or abnormal resource usage. By aggregating abnormal nodes, a larger range of abnormal subgraphs can be identified, helping the system to quickly locate and handle potential problems.

[0029] In a specific embodiment, the execution step inputs the weighted heterogeneous graph into the multi-head graph attention layer, each attention head independently calculates the node features, and then splices the multi-head outputs to obtain an enhanced node representation, and the enhanced node representation is input into the graph convolution layer, the neighbor node information is aggregated, the node features are updated, and the process of obtaining the local context-aware node representation can specifically include the following steps: For each node in the weighted heterogeneous graph, the attention coefficient of each node and all adjacent nodes is calculated to obtain the attention distribution matrix. Based on the attention distribution matrix, the feature vector of each node is weighted summed to obtain the single-head attention output. Repeat the single-head attention output for a preset number of times, using a different parameter matrix each time, to obtain multiple sets of independent attention outputs, and concatenate the multiple sets of independent attention outputs to obtain a concatenated feature vector; The concatenated feature vector is input into a nonlinear activation function to perform feature transformation to obtain an enhanced node representation, and the enhanced node representation is input into an edge-aware convolution kernel to extract the local structural information between the node and its neighboring nodes to obtain a local feature representation; The local feature representation is residually connected with the original node feature to obtain the fused feature, and the fused feature is batch normalized and multi-layer feature information is fused to obtain the local context-aware node representation.

[0030] Specifically, we define a feature vector for each node and calculate the attention coefficients of each node and all adjacent nodes based on the graph attention mechanism. The eigenvector of , and its adjacent nodes The eigenvector of , the attention coefficient is calculated by the following formula: ; in, Is a node For Node The attention coefficient, Representation Node The set of neighbor nodes of is the attention score between nodes, calculated by a neural network, such as the LeakyReLU method. By calculating the attention coefficients of all adjacent nodes, we get the attention distribution matrix, which provides the relative importance of each node to its neighboring nodes. Based on the attention distribution matrix, we perform a weighted summation of the feature vectors of each node to get the single-head attention output. , the calculation of single-head attention output is expressed as: ; In order to improve the expressiveness of the model, the single-head attention output is repeated a preset number of times, using a different parameter matrix each time to obtain multiple sets of independent attention outputs. A different weight matrix is ​​applied each time , the obtained independent attention output is expressed as: ; Concatenate multiple sets of independent attention outputs to obtain a concatenated feature vector: ; in, Indicates the number of attention heads. The concatenated feature vector is input into a nonlinear activation function, such as ReLU or Sigmoid, to perform feature transformation and obtain an enhanced node representation: ; in, is the activation function, is the weight matrix, is a bias term. The enhanced node representation captures the features of the node and the information of its neighboring nodes. The enhanced node representation is input into the edge-aware convolution kernel to extract the local structural information between the node and its neighboring nodes. The edge-aware convolution kernel can dynamically adjust the convolution operation according to the features of the edge and effectively extract the local feature representation. This process is expressed by the formula: ; in, It is for the edge This operation enables the local structural information of the node to be effectively captured, enhancing the richness of the features. The local feature representation is residually connected with the original node feature to obtain the fused feature: ; By combining the original features with the extracted local features, the integrity of the information can be better maintained. Batch normalization of the fused features can reduce internal covariate shift and improve the stability and speed of training. The feature representation after batch normalization is expressed as: ; in, and are the mean and variance of the batch, respectively. is a small constant used to prevent division by zero, and is a learnable parameter. In this way, the fusion features are fused with multiple layers of feature information to obtain local context-aware node representation.

[0031] In a specific embodiment, the process of executing step 103 may specifically include the following steps: Convert hierarchical abnormal events into state vectors as input to the reinforcement learning model to obtain the current system state representation; Based on the current system state representation, a policy network is used to generate a set of candidate actions. Each action corresponds to a possible task allocation strategy to obtain the initial action space. Perform action masking on the initial action space, exclude actions that do not meet the system constraints, obtain a valid action set, and input the valid action set into the value network to calculate the expected value of each action to obtain the action value evaluation result; Based on the action value evaluation results, the ε-greedy strategy is used to select the optimal action to obtain the task allocation decision for the current time step. According to the task allocation decision, the system resource allocation state is updated, and the corresponding task scheduling operation is performed to obtain the new system state. The new system state is input into the reward function, the immediate reward value of the current decision is calculated, and the reward signal is obtained. Based on the reward signal and the new system state, the value network parameters are updated using temporal difference learning to obtain the optimized value estimate. The optimized value estimate is used to update the policy network until the preset convergence condition or the maximum number of iterations is reached to obtain the task allocation plan.

[0032] Specifically, the abnormal events are encoded to generate state vectors. Each state vector contains the number, severity, and frequency of various abnormalities in the current system. These features are normalized to ensure that they are compared on the same scale. It is expressed as: ; in, Representative Characteristics of the abnormal event, such as its frequency of occurrence or impact, is the total number of abnormal events. The state vector is used as the input of the reinforcement learning model to help the model understand the current health of the system. Based on the current system state representation, a set of candidate actions is generated using the policy network, and each action corresponds to a possible task allocation strategy. By inputting the state vector To the policy network, output the initial action space A, including different resource allocation strategies, such as increasing the resources of a task, adjusting the task priority, or delaying a task. Perform action masking on the initial action space, exclude actions that do not meet the system constraints, and obtain a valid action set. Valid action set Contains all actions that meet the constraints. Input the valid action set into the value network, calculate the expected value of each action, and obtain the action value evaluation result. The output of the value network is expressed as: ; in, Indicates action The expected value of is the expectation of future rewards, is the current state. By evaluating the value of effective actions, a quantitative basis is provided for subsequent decision-making. Based on the action value evaluation results, use -The greedy strategy selects the best action to ensure a balance between exploration and exploitation. Randomly select an action The probability of , the probability of choosing the action with the highest current expected value is . This allows the model to effectively utilize existing knowledge and explore new possibilities. Once the optimal action is selected, the task allocation decision for the current time step is obtained, and then the system resource allocation state is updated according to the task allocation decision, and the corresponding task scheduling operation is performed to form a new system state Input the new system state into the reward function, calculate the immediate reward value of the current decision, and obtain the reward signal The reward function is designed based on the execution of the task, the efficiency of resource use, and the health of the system to ensure that the reward value can accurately reflect the effect of the current decision. The calculation of the reward signal is expressed as: ; in, is the reward function, is the current state, is the selected action. Based on the reward signal and the new system state, the value network parameters are updated using temporal difference learning to obtain the optimized value estimate. The update process uses the following update rule: ; in, are the parameters of the value network, is the learning rate, is a discount factor that weighs the importance of future rewards. The update process ensures that the model can gradually improve its estimate of the value of the state, thereby improving the effect of task allocation. The optimized value estimate is used to update the policy network to ensure that the output of the policy network reflects the latest knowledge and experience. This process continues until the preset convergence condition or the maximum number of iterations is reached, generating the final task allocation solution.

[0033] In a specific embodiment, the process of executing step 104 may specifically include the following steps: The task allocation scheme is converted into a structured data representation as the input of the root cause reasoning to obtain the initial reasoning data, and the initial reasoning data is input into the deep neural network for feature extraction to obtain the feature vector; Input the feature vector into the attention mechanism module, calculate the importance weights of different features, obtain the weighted feature representation, and model the relationship between system components based on the weighted feature representation to obtain the component relationship graph; Perform causal discovery and causal relationship identification on the component relationship diagram to obtain a preliminary causal diagram, and perform logical reasoning enhancement on the preliminary causal diagram to obtain an enhanced causal diagram; Based on the enhanced causal graph, the initial structure of the knowledge graph is constructed, and the system components, events, and relationships are represented as entities and edges to obtain the original knowledge graph. The original knowledge graph is then subjected to graph embedding analysis to learn the low-dimensional representation of entities and relationships and obtain the embedding vector. Using the embedded vectors, the knowledge graph completion task is performed to predict potential missing relationships and obtain the target knowledge graph. Based on the target knowledge graph, state diagram visualization and key causal path extraction are performed to obtain the system state diagram and causal relationships.

[0034] Specifically, the various task information, resource allocation, and related contextual data in the plan are organized into a standardized input format. Structured data includes task identifiers, allocated resources, expected completion time, etc. In this way, the original task allocation plan is converted into initial reasoning data that can be processed by the machine learning model. The initial reasoning data is input into a deep neural network for feature extraction, from which hidden patterns and features are learned to obtain a feature vector with strong expressive power. The feature vector is input into the attention mechanism module to calculate the importance weights of different features and obtain a weighted feature representation. Through the attention mechanism, a weight is assigned to each feature. , indicating the importance of the feature in the current task assignment, the formula is expressed as: in, is the weighted feature representation, is the first feature vector Features, is the total number of features. Based on the weighted feature representation, the relationship between system components is modeled to form a component relationship graph. The component relationship graph describes the interactions and dependencies between various components of the system (such as servers, databases, network devices, etc.). By analyzing the component relationship graph, potential causal relationships are identified and a causal graph is preliminarily constructed. In this process, the accuracy of causal relationship identification is improved by using technologies such as graph neural networks to obtain a preliminary causal graph. The preliminary causal graph is enhanced by logical reasoning to improve the expressive power of the causal graph. Through reasoning rules, such as the relationship based on cause and effect, the identified causal relationships are verified and enhanced to obtain an enhanced causal graph. Based on the enhanced causal graph, the initial structure of the knowledge graph is constructed, and the system components, events, and relationships are represented as entities and edges. Each entity is a system component, and the edge represents the relationship between components, such as dependency, influence, etc. After the construction is completed, the original knowledge graph is obtained. The original knowledge graph is subjected to graph embedding analysis to learn the low-dimensional representation of entities and relationships and obtain an embedding vector. Graph embedding technology compresses high-dimensional graph structure information into a low-dimensional space, and the low-dimensional representation can retain the semantic information in the graph structure. Use the embedding vector to perform the knowledge graph completion task, predict potential missing relationships, and obtain the target knowledge graph. Use the embedding vectors of known relationships and entities to identify existing but unrecorded relationships through machine learning algorithms. Based on the target knowledge graph, perform state diagram visualization and key causal path extraction to obtain the system state diagram and causal relationships. State diagram visualization uses graphical tools to present the entities and their relationships in the knowledge graph, helping operation and maintenance personnel to intuitively understand the system status and its changes. Key causal path extraction identifies key factors that affect system performance by analyzing and enhancing important paths in the causal graph.

[0035] In a specific embodiment, the execution step uses the embedding vector to perform the knowledge graph completion task, predict potential missing relationships, obtain the target knowledge graph, and perform state graph visualization and key causal path extraction based on the target knowledge graph. The process of obtaining the system state graph and causal relationship can specifically include the following steps: Perform tensor decomposition on the embedded vector, calculate the similarity score between entities and relations, obtain a set of candidate relations, and input the set of candidate relations into the relation classifier, evaluate the authenticity of each candidate relation, and obtain the relation confidence score; Based on the relationship confidence score, a threshold screening method is used to select high-confidence relationships to obtain a supplementary relationship set, and the supplementary relationship set is merged with the original knowledge graph to update the graph structure to obtain an extended knowledge graph. Perform graph consistency check on the extended knowledge graph to obtain the target knowledge graph, and execute the graph layout algorithm based on the target knowledge graph to calculate the node positions and edge routings to obtain the initial visual layout; Apply force-directed algorithm to the initial visualization layout to optimize node distribution and edge intersection to obtain the optimized layout structure, and convert the optimized layout structure into a visualization graph, add color coding and legend information to obtain the system status diagram; A path search algorithm is applied to the target knowledge graph to identify the shortest path between key entities and obtain a set of candidate causal paths. Path importance scoring and key causal links are extracted based on the candidate causal path set to obtain causal relationships.

[0036] Specifically, we perform tensor decomposition on the embedding vectors and decompose them into low-dimensional matrices to calculate the similarity scores between entities and relations. Suppose there is an embedding tensor , whose dimensions are ,in is the number of entities, is the embedding dimension. The result of tensor decomposition is expressed as ; in, is the entity matrix, is the relationship matrix. Using these matrices, we calculate the similarity score ; Through the formula, we get the candidate relationship set, calculate the similarity between each pair of entities, and find out the existing relationship. Input the candidate relationship set into the relationship classifier, evaluate the authenticity of each candidate relationship, and get the relationship confidence score. , which indicates the credibility of the candidate relationship. Based on the relationship confidence score, a threshold screening method is used to select high-confidence relationships to obtain a supplementary relationship set. , then the relationship and Add to the supplementary relationship set. Merge the supplementary relationship set with the original knowledge graph, update the graph structure, and obtain the extended knowledge graph. Perform a graph consistency check on the extended knowledge graph to ensure that the newly added relationships do not cause logical conflicts. Use the consistency check algorithm in graph theory to determine whether each node and edge conforms to the predefined rules and obtain the target knowledge graph. Based on the target knowledge graph, execute the graph layout algorithm to calculate the node position and edge routing to obtain the initial visual layout. The graph layout algorithm can reasonably distribute the nodes in the visual space, making the edges between the relationships clearly visible, which is helpful for analysis. After the initial layout is completed, the force-directed algorithm is applied to optimize the distribution of nodes and the intersection of edges. The force-directed algorithm simulates the action of physical forces to pull nodes that have relationships with each other closer, while unrelated nodes are pushed away, forming a layout with a reasonable structure. Convert the optimized layout structure into a visual graph, and add color coding and legend information so that users can intuitively understand the system status and the relationship between each component. Apply the path search algorithm to the target knowledge graph to identify the shortest path between key entities and obtain the candidate causal path set. Path search algorithms such as the Dijkstra algorithm or the A* algorithm can efficiently find the shortest path connecting key entities in the knowledge graph and help identify important causal relationships. Path importance scoring is performed based on the candidate causal path set to evaluate the importance of each path in the system, helping to determine which causal paths have a significant impact on system performance, thereby extracting key causal links and ultimately obtaining a clear causal network.

[0037] The above describes the business data analysis method based on big data in the embodiment of the present invention. The following describes the business data analysis device based on big data in the embodiment of the present invention. Figure 2 , an embodiment of a business data analysis device based on big data in an embodiment of the present invention includes: The collection module 201 is used to collect and preprocess the original log data to obtain standardized log data, and perform adaptive multimodal feature extraction on the standardized log data to obtain a comprehensive feature representation; An analysis module 202 is used to input the comprehensive feature representation into a heterogeneous graph neural network for anomaly detection and multi-level root cause analysis to obtain hierarchical abnormal events; Allocation module 203, used to perform task scheduling and resource allocation of reinforcement learning according to hierarchical abnormal events to obtain a task allocation plan; A construction module 204 is used to perform deep learning enhanced root cause reasoning and knowledge graph construction on the task allocation scheme to obtain a system state diagram and causal relationship; The generation module 205 is used to generate a multi-dimensional task data analysis report based on the system state diagram and the cause-effect relationship.

[0038] Through the collaborative cooperation of the above-mentioned components, adaptive multimodal feature extraction is performed on the original log data, and the comprehensive utilization of text, time series and structural features is realized, which improves the richness and accuracy of feature representation. The use of heterogeneous graph neural networks for anomaly detection and multi-level root cause analysis can effectively capture the complex relationship between log events, improve the accuracy of anomaly detection and the depth of root cause analysis. The introduction of reinforcement learning for task scheduling and resource allocation realizes adaptive decision-making based on dynamic system status and optimizes the efficiency of system resource utilization. Through deep learning-enhanced root cause reasoning and knowledge graph construction, a more comprehensive and explainable system state diagram and causal relationship are provided, which helps operation and maintenance personnel to quickly locate and solve problems. Full automation from data collection to report generation is achieved, which greatly improves the efficiency and real-time performance of log analysis. The present invention realizes comprehensive analysis, anomaly detection, root cause reasoning and intelligent scheduling of log data. This can not only improve the reliability and performance of the system, but also provide more intuitive and accurate decision support for operation and maintenance personnel.

[0039] above Figure 2 The business data analysis device based on big data in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The business data analysis device based on big data in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0040] Figure 3 3 is a schematic diagram of the structure of a business data analysis device based on big data provided by an embodiment of the present invention. The business data analysis device 300 based on big data may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 310 (for example, one or more processors) and a memory 320, and one or more storage media 330 (for example, one or more mass storage device terminals) storing application programs 333 or data 332. Among them, the memory 320 and the storage medium 330 may be short-term storage or permanent storage. The program stored in the storage medium 330 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the business data analysis device 300 based on big data. Furthermore, the processor 310 may be configured to communicate with the storage medium 330, and execute a series of instruction operations in the storage medium 330 on the business data analysis device 300 based on big data to implement the steps of the above-mentioned business data analysis method based on big data.

[0041] The big data-based business data analysis device 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input and output interfaces 360, and / or one or more operating systems 331, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Figure 3 The structure of the business data analysis device based on big data shown does not constitute a limitation on the business data analysis device based on big data provided by the present invention, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0042] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of the big data-based business data analysis method.

[0043] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0044] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the whole or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0045] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A business data analysis method based on big data, characterized in that: The method comprises: Collecting and preprocessing the original log data to obtain standardized log data, and performing adaptive multimodal feature extraction on the standardized log data to obtain a comprehensive feature representation; Inputting the comprehensive feature representation into a heterogeneous graph neural network for anomaly detection and multi-level root cause analysis to obtain hierarchical abnormal events; According to the hierarchical abnormal events, perform task scheduling and resource allocation of reinforcement learning to obtain a task allocation plan; Performing deep learning enhanced root cause reasoning and knowledge graph construction on the task allocation scheme to obtain a system state diagram and causal relationship; A multi-dimensional task data analysis report is generated based on the system state diagram and the causal relationship.

2. The business data analysis method based on big data according to claim 1 is characterized in that: The collecting and preprocessing of the original log data to obtain standardized log data, and the adaptive multimodal feature extraction of the standardized log data to obtain a comprehensive feature representation include: Perform distributed collection on multi-source log data to obtain an original log data set, and perform regular expression matching and keyword extraction on the original log data set to obtain structured log entries; Sorting and aggregating the structured log entries by timestamp to obtain a time series log sequence, and performing statistical analysis on the time series log sequence to calculate event frequency and time distribution to obtain basic statistical features; Based on a predefined parsing rule library, the structured log entry is semantically parsed to extract the operation type and error code to obtain semantic annotation data, and the semantic annotation data is input into a word embedding model to perform text feature recognition to obtain a log text representation; Performing a sliding window analysis on the time series log sequence, extracting time intervals and event sequence patterns to obtain time series features, and constructing a field association graph based on the semantically annotated data to obtain structural features; The text representation, the temporal features and the structural features are input into a multimodal feature fusion network to obtain initial comprehensive features, and the initial comprehensive features are subjected to an attention mechanism and dynamic weight allocation to obtain a comprehensive feature representation.

3. The business data analysis method based on big data according to claim 1 is characterized in that: The comprehensive feature representation is input into a heterogeneous graph neural network for anomaly detection and multi-level root cause analysis to obtain hierarchical abnormal events, including: Converting the comprehensive feature representation into a heterogeneous graph structure, wherein nodes represent log events and edges represent relationships between events, to obtain an initial heterogeneous graph; Input the initial heterogeneous graph into the graph attention layer to calculate the attention weights between nodes to obtain a weighted heterogeneous graph; The weighted heterogeneous graph is input into a multi-head graph attention layer, each attention head independently calculates node features, and then the multi-head outputs are concatenated to obtain an enhanced node representation, and the enhanced node representation is input into a graph convolution layer to aggregate neighbor node information, update node features, and obtain a local context-aware node representation; The local context-aware node representation is input into a graph pooling layer for hierarchical clustering and graph structure compression to obtain a multi-scale graph representation, and the multi-scale graph representation is skipped to fuse graph representations of different scales to obtain multi-granularity features; The multi-granularity features are input into a graph variational autoencoder to perform reconstruction error and KL divergence optimization to obtain anomaly scores, and based on the anomaly scores, a dynamic threshold algorithm is used to identify abnormal nodes and abnormal subgraph construction to obtain hierarchical abnormal events.

4. The business data analysis method based on big data according to claim 3 is characterized in that: The weighted heterogeneous graph is input into a multi-head graph attention layer, each attention head independently calculates node features, and then the multi-head outputs are spliced ​​to obtain an enhanced node representation, and the enhanced node representation is input into a graph convolution layer, neighbor node information is aggregated, node features are updated, and local context-aware node representation is obtained, including: For each node in the weighted heterogeneous graph, the attention coefficient of each node and all adjacent nodes is calculated to obtain an attention distribution matrix, and based on the attention distribution matrix, the feature vector of each node is weighted summed to obtain a single-head attention output; Repeat the single-head attention output for a preset number of times, using a different parameter matrix each time, to obtain multiple sets of independent attention outputs, and perform a splicing operation on the multiple sets of independent attention outputs to obtain a spliced ​​feature vector; Input the concatenated feature vector into a nonlinear activation function, perform feature transformation, obtain an enhanced node representation, and input the enhanced node representation into an edge-aware convolution kernel to extract local structural information between the node and its neighboring nodes to obtain a local feature representation; The local feature representation is residually connected with the original node feature to obtain a fused feature, and the fused feature is batch normalized and multi-layer feature information is fused to obtain a local context-aware node representation.

5. The business data analysis method based on big data according to claim 1 is characterized in that: The step of performing task scheduling and resource allocation of reinforcement learning according to the hierarchical abnormal events to obtain a task allocation plan includes: Convert the hierarchical abnormal events into a state vector as an input of a reinforcement learning model to obtain a current system state representation; Based on the current system state representation, a set of candidate actions is generated using a policy network, each action corresponds to a possible task allocation strategy, and an initial action space is obtained; Performing action masking on the initial action space, excluding actions that do not meet the system constraints, obtaining a valid action set, and inputting the valid action set into the value network, calculating the expected value of each action, and obtaining an action value evaluation result; Based on the action value evaluation result, the ε-greedy strategy is used to select the optimal action to obtain the task allocation decision of the current time step, and according to the task allocation decision, the system resource allocation state is updated, and the corresponding task scheduling operation is performed to obtain a new system state; Inputting the new system state into a reward function, calculating an instantaneous reward value of a current decision, obtaining a reward signal, and updating a value network parameter using temporal difference learning based on the reward signal and the new system state to obtain an optimized value estimate; The optimized value estimate is used to update the policy network until a preset convergence condition or a maximum number of iterations is reached, thereby obtaining a task allocation solution.

6. The business data analysis method based on big data according to claim 1 is characterized in that: The deep learning enhanced root cause reasoning and knowledge graph construction of the task allocation scheme to obtain a system state diagram and causal relationship includes: Convert the task allocation scheme into a structured data representation as an input for root cause reasoning to obtain initial reasoning data, and input the initial reasoning data into a deep neural network for feature extraction to obtain a feature vector; Input the feature vector into the attention mechanism module, calculate the importance weights of different features, obtain weighted feature representation, and model the relationship between system components based on the weighted feature representation to obtain a component relationship graph; Performing causal discovery and causal relationship identification on the component relationship diagram to obtain a preliminary causal diagram, and performing logical reasoning enhancement on the preliminary causal diagram to obtain an enhanced causal diagram; Based on the enhanced causal graph, an initial structure of a knowledge graph is constructed, system components, events, and relationships are represented as entities and edges to obtain an original knowledge graph, and graph embedding analysis is performed on the original knowledge graph to learn low-dimensional representations of entities and relationships to obtain an embedding vector; The embedding vector is used to perform a knowledge graph completion task, predict potential missing relationships, obtain a target knowledge graph, and perform state graph visualization and key causal path extraction based on the target knowledge graph to obtain a system state graph and causal relationships.

7. The business data analysis method based on big data according to claim 6 is characterized in that: The method uses the embedding vector to perform a knowledge graph completion task, predict potential missing relationships, obtain a target knowledge graph, and perform state graph visualization and key causal path extraction based on the target knowledge graph to obtain a system state graph and causal relationships, including: Performing tensor decomposition on the embedding vector, calculating similarity scores between entities and relations, obtaining a candidate relation set, and inputting the candidate relation set into a relation classifier, performing authenticity evaluation on each candidate relation, and obtaining a relation confidence score; Based on the relationship confidence scores, a threshold screening method is used to select high-confidence relationships to obtain a supplementary relationship set, and the supplementary relationship set is merged with the original knowledge graph to update the graph structure to obtain an extended knowledge graph; Performing a graph consistency check on the extended knowledge graph to obtain a target knowledge graph, and executing a graph layout algorithm based on the target knowledge graph to calculate node positions and edge routings to obtain an initial visual layout; Applying a force-directed algorithm to the initial visual layout to optimize node distribution and edge intersection to obtain an optimized layout structure, and converting the optimized layout structure into a visual graph, adding color coding and legend information to obtain a system state diagram; A path search algorithm is applied to the target knowledge graph to identify the shortest path between key entities, obtain a set of candidate causal paths, and perform path importance scoring and key causal link extraction based on the candidate causal path set to obtain causal relationships.

8. A business data analysis device based on big data, characterized in that: The device is used to perform the business data analysis method based on big data according to any one of claims 1 to 7, comprising: A collection module, used to collect and preprocess the original log data to obtain standardized log data, and perform adaptive multimodal feature extraction on the standardized log data to obtain a comprehensive feature representation; An analysis module, used for inputting the comprehensive feature representation into a heterogeneous graph neural network to perform anomaly detection and multi-level root cause analysis to obtain hierarchical abnormal events; An allocation module, configured to perform task scheduling and resource allocation of reinforcement learning according to the hierarchical abnormal events to obtain a task allocation plan; A construction module is used to perform deep learning enhanced root cause reasoning and knowledge graph construction on the task allocation scheme to obtain a system state diagram and causal relationship; A generation module is used to generate a multi-dimensional task data analysis report based on the system state diagram and the causal relationship.

9. A business data analysis device based on big data, characterized in that: The big data-based business data analysis device comprises: a memory and at least one processor, wherein instructions are stored in the memory; The at least one processor calls the instructions in the memory so that the big data-based business data analysis device executes the big data-based business data analysis method as described in any one of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the business data analysis method based on big data as described in any one of claims 1 to 7 is implemented.

Images