Big data integration system based on artificial intelligence

Abstract

The invention discloses a big data integration system based on artificial intelligence, and particularly relates to the technical field of data integration, by deploying an edge data analysis model at a big data acquisition end and combining cloud computing resources for collaborative optimization, efficiency abnormity in the data integration process is timely warned, computing resource allocation is dynamically adjusted, and the efficiency of data integration is improved. A data processing strategy is optimized, so that delay and resource waste in an integration process are effectively reduced; a data stream processing path is optimized cooperatively through edge end preprocessing and cloud deep analysis; through combined application of key node monitoring, key node performance association rule mining, adaptive performance bottleneck diagnosis and dynamic scheduling, real-time optimization and closed-loop control of a big data integration process are realized, and abnormity of an actual value of integration efficiency is avoided. The problems of low integration efficiency, unbalanced resources, integration efficiency prediction lag and insufficient resource optimization are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of data integration. More specifically, the present invention relates to a big data integration system based on artificial intelligence. Background Art

[0002] With the rapid development of information technology, the collection, storage, processing, and analysis of big data have become important means for various industries to achieve intelligent decision-making. Especially in large-scale distributed computing environments, the real-time integration and efficient processing of massive data have become important factors restricting system performance. Traditional centralized data processing methods, although having advantages in computing power, often lead to over-concentration or uneven distribution of computing resources due to the rapid growth of data volume and the complexity of computing tasks, affecting the integration efficiency and processing quality. For this reason, the combination of cloud computing and edge computing has been proposed as a solution to relieve the load pressure on the cloud and improve the overall efficiency by sinking some computing tasks to edge devices.

[0003] Existing big data integration systems face the following technical problems: First, the resource allocation and data distribution strategies in the data integration process are insufficient to cope with the dynamically changing load and the heterogeneity of computing nodes, resulting in resource overload or idleness of some computing nodes, thereby affecting the overall integration efficiency of the system; Second, the means for real-time prediction and optimization of integration efficiency are relatively lagging, lacking effective mechanisms to cope with sudden performance degradation. Especially in the face of complex data characteristics and heterogeneous environments, it is difficult to achieve efficient dynamic resource scheduling. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides a big data integration system based on artificial intelligence to solve the problems raised in the above background art.

[0005] To achieve the above object, the present invention provides the following technical solution: A big data integration system based on artificial intelligence, comprising:

[0006] An integration task division module, which obtains the big data to be integrated and divides the big data to be integrated into multiple time period data according to a set fixed time interval;

[0007] A data integration execution module, after completing the division of the time period data, allocates each time period data to a computing node for big data integration according to a pre-determined resource allocation strategy and the running status of each computing node;

[0008] An integration efficiency prediction model construction module, during the execution of the integration, obtains in real time the preset running information of the computing node, the resource allocation strategy, the characteristics of the data to be integrated, and the environmental context information as the input information of the integration efficiency prediction model, and constructs an integration efficiency prediction model;

[0009] An integrated risk warning module, which is used to warn of abnormal integration efficiency. By dynamically monitoring the completion time and the amount of processed data of a large amount of data to be integrated, the actual value E of the integration efficiency is calculated. act ; The predicted value E of the integration efficiency is output through an integration efficiency prediction model. pred ;

[0010] If the difference between the actual value E of the integration efficiency act and the theoretical value E of the integration efficiency pred exceeds the preset value, a warning is sent to the user, indicating that the big data integration process is abnormal.

[0011] Preferably, the data integration execution module includes an abnormal calculation node control unit, which is used to identify abnormal calculation nodes and dynamically adjust the distribution strategy based on the abnormal calculation nodes;

[0012] The process of identifying abnormal calculation nodes includes: continuously monitoring the operation indicators of each calculation node and comparing them with the historical baseline or health threshold; once an abnormal fluctuation or fault alarm of a calculation node is detected, it is marked as a fault / deterioration state;

[0013] The process of dynamically adjusting the distribution strategy based on the abnormal calculation nodes includes:

[0014] Temporarily isolate and transfer critical tasks: For the nodes marked as faulty / deteriorated, temporarily stop the distribution of new data segments; if there is important large data to be integrated that is being processed, the remaining tasks are transferred to other normal nodes for execution according to the integration progress and the amount of data;

[0015] Re-evaluate the resource configuration of the remaining available nodes: Dynamically adjust the priority of task transfer based on the remaining load of the calculation nodes, the urgency of the integration tasks, and the network topology;

[0016] Continuously monitor and recover the calculation nodes: Conduct fault troubleshooting or performance tuning for the faulty calculation nodes, and after restoring them to a healthy state, include them in the integration task distribution queue again.

[0017] Preferably, the process of building the integration efficiency prediction model includes the following steps:

[0018] Step S01, feature extraction: Extract features related to the integration efficiency from the input information as input features; the input information includes node preset operation information, resource allocation strategy parameters, data characteristics to be integrated, and environmental context information;

[0019] Step S02, prepare training data and labels: Collect historical data during the big data integration process, including input features and historical actual efficiency, and use the historical actual efficiency index as a label to construct a training data set and a validation data set;

[0020] Step S03, Select Model Architecture and Loss Function: Select a machine learning algorithm to build a model, initialize parameters, set the loss function, and output the selected initialized model;

[0021] Step S04, Model Training and Validation: Iteratively train the selected initialized model, adjust network parameters and hyperparameters, and evaluate the prediction accuracy through a validation set; until the loss function meets the preset requirements, and output the trained integrated efficiency prediction model;

[0022] Step S05, Model Deployment and Application: Deploy the trained integrated efficiency prediction model in a big data integration system, obtain the time period data obtained in real time, and output the integrated efficiency prediction value.

[0023] Preferably, the loss function is obtained as follows:

[0024] Conduct several tests on the integrated efficiency prediction model, and record the actual operation information of the nodes, the actual integrated efficiency value, and the theoretical integrated efficiency value for each test;

[0025] If the actual operation information of the node is the same as the preset operation information of the node, it is marked as a valid test;

[0026] Suppose there are N valid tests, and i is used to represent the sequence number of the valid test; obtain the actual integrated efficiency value E act_i of the i-th valid test and the theoretical integrated efficiency value E pred_i ;

[0027] Use the difference between the actual integrated efficiency value E act_i and the theoretical integrated efficiency value E pred_i to represent the loss function.

[0028] Preferably, the integrated efficiency prediction model satisfies the following formula:

[0029]

[0030] Among them, S represents the total scale or total duration interval of the data to be processed, x is the integration variable from 0 to S, which is used to depict the continuous evolution process of data or time, H(x) represents the information entropy on the data block or time segment x, and H(x) is obtained through the information entropy calculation formula, P(x) represents the resource scheduling function that changes with x; DO represents the data source diversity impact factor;

[0031] is used to convert the input feature into an interval value from 0 to 1; suppose there are m input features, and j is used to represent the sequence number of the input feature; obtain the influence intensity of the input feature through the following formula:

[0032]

[0033] Among them, α j is the weight of the j-th input feature, Tx j is the eigenvalue of the j-th input feature, and γ is the power exponent;

[0034] λsin(ω·S) is the periodic correction term, λ is the periodic intervention weight coefficient, and ω is the periodic frequency; Γ(H) is the Gamma function of the entropy value H, and H is the average of all H(x).

[0035] Preferably, the integrated task division module is connected to the cloud-edge collaborative optimization module, and the cloud-edge collaborative optimization module preprocesses the big data, screens out abnormal data, and improves the integrated efficiency prediction value, including the following steps:

[0036] Deploy the edge abnormal data analysis model and the cloud abnormal data analysis model. The edge abnormal data analysis model is used to detect abnormal features in the data stream in real time, mark the identified first abnormal data, and upload it to the cloud; the cloud abnormal data analysis model combines the analysis results at the edge side, performs abnormal analysis on the aggregated big data, identifies the second abnormal data, and generates high-quality time period data; preferably, the high-quality time period data is transmitted to the computing node to improve the integrated efficiency of the big data set.

[0037] Preferably, if the performance index (such as the recognition accuracy) of the edge abnormal data analysis model is lower than the requirement, an alarm is issued for the abnormality of the edge abnormal data analysis model, and the parameters of the edge abnormal data analysis model are optimized through the cloud abnormal data analysis model.

[0038] Preferably, the cloud-edge collaborative optimization module further includes a synchronous update management unit, and the synchronous update management unit includes:

[0039] Periodically align the parameters of the edge abnormal data analysis model with the cloud abnormal data analysis model, and the parameter alignment adopts the gradient aggregation or weight alignment method;

[0040] Establish a model version library to identify the abnormal data analysis model currently used by each edge node, and at the same time maintain a version parameter update log on the central side;

[0041] Dynamically determine the data transmission method according to the importance and real-time nature of the on-site data;

[0042] If the data abnormality level is high or in-depth analysis is required, immediately transmit the original data to the cloud abnormal data analysis model;

[0043] If the data importance is low, first quickly process it at the edge and generate a summary before transmitting it back to the cloud.

[0044] Preferably, if the actual integrated efficiency value E act and the theoretical integrated efficiency value Epred If the difference exceeds the preset value, an exception analysis instruction is generated; after receiving the exception analysis instruction, the exception cause analysis module performs the following operations:

[0045] By calculating the proportion of computing resource usage, determine whether there is insufficient computing resources;

[0046] By calculating network latency, bandwidth utilization, and transmission efficiency, determine whether there is insufficient transmission resources;

[0047] By checking the size and complexity of the data, calculate the data feature load, and determine whether there is an abnormal data feature;

[0048] Determine whether there is an uneven load distribution problem.

[0049] Preferably, the data integration execution module includes a bottleneck node control unit, and the bottleneck node control unit includes the following:

[0050] Key node monitoring: Obtain the real-time performance index data of key nodes during the big data integration process; perform multi-time scale decomposition on the real-time performance index data through fast Fourier transform or wavelet transform, and decompose it into short-term fluctuations and long-term trends; the short-term fluctuations are used to monitor real-time performance anomalies and trigger immediate alarms; the long-term trends are used to predict the future performance bottlenecks of key nodes;

[0051] Mining of key node performance association rules: By combining static association rule mining algorithms and deep learning models, construct a multi-level performance association rule set; the multi-level performance association rule set is used for performance bottleneck diagnosis and dynamic scheduling;

[0052] Adaptive performance bottleneck diagnosis: According to the real-time performance index data and the multi-level performance association rule set, dynamically adjust the diagnosis strategy, identify the bottleneck type, and generate a bottleneck diagnosis report; the bottleneck diagnosis report includes the bottleneck node number, the performance bottleneck type, and the impact range on the integration efficiency;

[0053] Dynamic scheduling, based on the bottleneck diagnosis report and the multi-level performance association rule set, uses a time series prediction model to predict the future performance index data of key nodes, and dynamically adjusts the data flow allocation strategy and resource expansion strategy to form a closed-loop optimization process.

[0054] The technical effects and advantages of the present invention:

[0055] (1) The big data integration system provided by the present invention improves the efficiency prediction and resource allocation accuracy in the data integration process by deploying an edge data analysis model at the big data collection end and combining it with cloud computing resources; it can dynamically adjust the computing resource allocation, optimize the data processing strategy, thereby effectively reducing the latency and resource waste in the integration process, and solving the problems of low integration efficiency and uneven resources.

[0056] (2) The big data integration system provided by the present invention monitors the integration efficiency in real time through the cloud-edge collaboration method and makes dynamic adjustments according to the actual situation, reducing system bottlenecks or performance degradation caused by inaccurate integration efficiency prediction; through the collaborative mechanism of edge-side preprocessing and cloud-depth analysis, it optimizes the processing path of the data stream, and solves the problems of lagging integration efficiency prediction and insufficient resource optimization.

[0057] (3) By introducing an adaptive resource scheduling strategy and a real-time integration efficiency feedback mechanism, it can dynamically adjust the resource allocation according to the load conditions of each computing node. The system monitors the usage of node resources in real time, ensures the efficient utilization of computing resources, avoids overloading or idling of node resources, thereby improving the integration efficiency, and solving the problems of uneven resource allocation and low integration efficiency in the process of large-scale data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 It is a flowchart of the integration risk warning method based on the integration efficiency prediction model of the present invention.

[0059] Figure 2 It is a block diagram of the big data integration system of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0060] Hereinafter, exemplary embodiments of the present disclosure will be described in more detail with reference to the drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be completely conveyed to those skilled in the art.

[0061] At the same time, it should be understood that, for the sake of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0062] The following description of at least one exemplary embodiment is merely illustrative in nature and in no way serves as a limitation on the present application and its application or use.

[0063] Technologies, methods, and devices known to those of ordinary skill in the relevant art may not be discussed in detail, but where appropriate, such technologies, methods, and devices shall be considered as part of the specification.

[0064] Example 1, refer to Figure 1 The flowchart of the integration risk early warning method based on the integration efficiency prediction model. The present invention provides a Figure 1 big data integration system based on artificial intelligence as shown in

[0065] An integration task division module, configured to obtain the big data to be integrated, divide the big data to be integrated into multiple time period data according to a set fixed time interval or business requirements, each time period segment corresponds to a unique number, and store it as distributable time period data;

[0066] A data integration execution module, after completing the division of the time period data, distributes each time period data to a computing node for big data integration according to a pre-determined resource allocation strategy and the running status of each computing node;

[0067] An integration efficiency prediction model construction module, during the execution of the integration, obtains the preset running information of the computing node, the resource allocation strategy, the characteristics of the data to be integrated, and the environmental context information in real time as the input information of the integration efficiency prediction model, and constructs the integration efficiency prediction model;

[0068] An integration risk early warning module, configured to warn of abnormal integration efficiency. By dynamically monitoring the completion time and the amount of data processed of the big data to be integrated, the actual value E of the integration efficiency is calculated act ; The predicted value E of the integration efficiency is output through the integration efficiency prediction model pred ;

[0069] If the difference between the actual value E of the integration efficiency act and the theoretical value E of the integration efficiency pred exceeds a preset value, a warning is sent to the user, indicating that the big data integration process is abnormal.

[0070] In the embodiments of the present invention, it needs to be further explained that the data integration execution module includes an abnormal computing node control unit, and the abnormal computing node control unit is configured to identify abnormal computing nodes and dynamically adjust the distribution strategy based on the abnormal computing nodes to ensure the normal progress of the big data integration process;

[0071] The process of identifying abnormal computing nodes includes: continuously monitoring the running metrics of each computing node (such as CPU utilization rate, memory occupancy, network bandwidth, etc.), and comparing with the historical baseline or health threshold; once an abnormal fluctuation or fault alarm of the computing node is detected, it is marked as a fault / deterioration state;

[0072] The process of dynamically adjusting the distribution strategy based on abnormal computing nodes includes:

[0073] Temporarily isolate and transfer critical tasks: For nodes marked as faulty / deteriorating, temporarily stop the distribution of new data segments; if there is important large data to be integrated that is being processed, transfer the remaining tasks to other normal nodes for execution according to the integration progress and data volume;

[0074] Re-evaluate the resource configuration of the remaining available nodes: Dynamically adjust the priority of task transfer based on the remaining load of the computing nodes, the urgency of the integration tasks, and the network topology;

[0075] Continuously monitor and recover nodes: Conduct fault troubleshooting or performance tuning for faulty nodes, and after they recover to a healthy state, include them in the integration task distribution queue to further improve the overall throughput capacity of the system.

[0076] In the embodiments of the present invention, it needs to be further explained that the process of building the integration efficiency prediction model includes the following steps:

[0077] Step S01, feature extraction: The input information includes node preset running information (such as CPU, memory, bandwidth, etc.), resource allocation strategy parameters (parallelism, load balancing factor, etc.), characteristics of data to be integrated (size, complexity, data type distribution), and environmental context information (such as network latency, node heat distribution). Extract features related to integration efficiency from the input information as input features;

[0078] Step S02, prepare training data and labels: Collect historical data during the big data integration process, including input features and historical actual efficiency, and use the historical actual efficiency index as the label to construct a training data set and a validation data set;

[0079] Step S03, select model architecture and loss function: Select a machine learning algorithm (such as a multi-layer neural network model, a time series model) to build a model, initialize the parameters, set the loss function, and output the selected initialized model;

[0080] Step S04, model training and validation: Conduct iterative training on the selected initialized model, adjust network parameters (such as weights, biases), hyperparameters (such as learning rate, regularization coefficient), and evaluate the prediction accuracy through the validation set; until the loss function meets the preset requirements, output the trained integration efficiency prediction model;

[0081] Step S05, model deployment and application: Deploy the trained integration efficiency prediction model in the big data integration system, obtain the time period data in real time, and output the integration efficiency prediction value.

[0082] In the embodiments of the present invention, it needs to be further explained that the acquisition method of the loss function is as follows:

[0083] Perform several tests on the integrated efficiency prediction model, and record the actual operation information of the nodes, the actual value of the integrated efficiency, and the theoretical value of the integrated efficiency for each test;

[0084] If the actual operation information of the node is the same as the preset operation information of the node, it is marked as a valid test;

[0085] Suppose there are N valid tests, and i represents the sequence number of the valid test; obtain the actual value E of the integrated efficiency for the i-th valid test act_i and the theoretical value E of the integrated efficiency pred_i ;

[0086] Use the difference between the actual value E of the integrated efficiency act_i and the theoretical value E of the integrated efficiency pred_i to represent the loss function;

[0087] Furthermore, the loss function is calculated through the following formula:

[0088]

[0089] In a possible embodiment, the loss function includes a regularization term for constraining the model complexity (such as L1, L2 regularization).

[0090] In the embodiments of the present invention, it needs to be further explained that the integrated efficiency prediction model satisfies the following formula:

[0091]

[0092] Among them, S represents the total scale or total duration interval of the data to be processed, x is the integration variable from 0 to S, used to describe the continuous evolution process of the data or time, H(x) represents the information entropy on the data block or time segment x, H(x) is obtained through the information entropy calculation formula, P(x) represents the resource scheduling function that changes with x. For example, network bandwidth, CPU utilization, and node load will change dynamically, and the dynamic nature of this resource scheduling is modeled through P(x); DO represents the influence factor of data source diversity. Multiple heterogeneous data sources may increase the integration difficulty. The closer DO is to 1, the higher the degree of data source difference, and the greater the processing cost; the closer DO is to 0, the lower the degree of data source difference, and the lower the processing cost;

[0093]

[0094] Transform the input features into interval values from 0 to 1 through non-linear mapping; assume there are m input features, and j represents the sequence number of the input features; α jis the weight of the j-th input feature, Tx j is the eigenvalue of the j-th input feature; γ is the power exponent used to adjust the intensity of the non-linear influence of the input feature;

[0095] λsin(ω·S) is the periodic correction term, representing the impact of periodic factors (such as business peaks or troughs) on efficiency. λ is the periodic intervention weight coefficient, and ω is the periodic frequency; Γ(H) is the Gamma function of the entropy value H. H is the global entropy value, which is the average of all H(x), reflecting the complexity of the overall data, and is the complexity correction with the increase of the entropy value H.

[0096] In the embodiments of the present invention, it needs to be further explained that the acquisition method of the data source diversity influence factor is as follows:

[0097] Obtain the types of big data data sources to be integrated and number them;

[0098] Extract the feature dimensions of each data source;

[0099] Calculate the difference in feature dimensions between every two data sources;

[0100] Comprehensively calculate the weighted cumulative value of all the differences in feature dimensions, and obtain the data source diversity influence factor through linear normalization processing.

[0101] In the embodiments of the present invention, it needs to be further explained that if the integrated efficiency prediction value is lower than the set threshold, an immediate warning will be given, and the big data will be preprocessed through the cloud-edge collaborative optimization module to screen out abnormal data and improve the integrated efficiency prediction value, including the following steps:

[0102] Deploy an edge abnormal data analysis model and a cloud abnormal data analysis model. The edge abnormal data analysis model is used to detect abnormal features in the data stream in real time, mark the first abnormal data identified. The first abnormal data is the abnormal data preliminarily marked by edge detection, which has real-time and locality, and is uploaded to the cloud; the cloud abnormal data analysis model combines the analysis results at the edge side to perform abnormal analysis on the aggregated big data, identify the second abnormal data. The second abnormal data is the global abnormal data identified in the in-depth analysis of the cloud, which has comprehensiveness, and generates high-quality time period data; preferentially transmit the high-quality time period data to the computing node to improve the integrated efficiency of the big data set.

[0103] The collaborative working mechanism of the cloud-edge collaborative optimization module is as follows: The edge anomaly data analysis model real-time marks the first anomaly data and uploads it to the cloud anomaly data analysis model; the cloud anomaly data analysis model combines the first anomaly data and comprehensively analyzes and optimizes the data based on the global data; the analysis results of the cloud anomaly data analysis model are used to optimize the edge anomaly data analysis model. If the performance indicators (such as recognition accuracy) of the edge anomaly data analysis model are lower than the requirements, the edge anomaly data analysis model is warned of anomalies, and the parameters of the edge anomaly data analysis model are optimized through the cloud anomaly data analysis model.

[0104] In a possible embodiment, based on the output result of the edge anomaly data analysis model, the calculation priority of the cloud anomaly data analysis model is adjusted for each data block, that is, the priority order is set for each data block based on the number and proportion of anomaly data in the data block. Specifically, according to the number and proportion of anomaly data, the severity of different anomaly features is evaluated, and the priority order is set for them. Anomaly features with higher priority will be preferentially processed by the cloud anomaly data analysis model to ensure that high-impact or high-risk anomalies can be quickly identified and resolved, thereby optimizing the overall anomaly handling efficiency and improving system stability.

[0105] In the embodiments of the present invention, it needs to be further explained that the cloud-edge collaborative optimization module further includes a synchronization and update management unit.

[0106] Foreground summary: In the big data integration monitoring solution based on artificial intelligence, a lightweight anomaly data analysis model is deployed on edge devices (such as edge servers, intelligent sensors) arranged at the big data collection site for real-time detection or monitoring; while a more complex and higher-precision anomaly data analysis model is deployed at the central node (cloud / server); the models at both ends often need to be consistent or refer to each other in terms of training version, parameter update, and inference results, but due to limitations such as network bandwidth, latency, and model size, synchronous update cannot be achieved; resulting in an offset between the feature distribution of the edge device and the training distribution of the cloud model during data backhaul and retraining; based on this, a synchronization and update management unit is provided; the synchronization and update management unit includes:

[0107] The edge anomaly data analysis model periodically aligns parameters with the cloud anomaly data analysis model, and the parameter alignment adopts gradient aggregation or weight alignment methods to ensure model synchronization.

[0108] A model version library is established to identify the anomaly data analysis model currently used by each edge node, and at the same time, a version parameter update log is maintained on the central side.

[0109] According to the importance and real-time nature of the on-site data, the data backhaul method is dynamically determined.

[0110] If the data anomaly level is high or requires in-depth analysis, the original data is immediately sent back to the cloud anomaly data analysis model;

[0111] If the data is of low importance, it is first quickly processed at the edge and a summary is generated before being transmitted back to the cloud.

[0112] In the embodiment of the present invention, it is necessary to further explain that if the actual value of the integrated efficiency E act The integrated efficiency theoretical value E pred The difference exceeds the preset value, and an abnormal analysis instruction is generated; after the abnormal cause analysis module receives the abnormal analysis instruction, it performs the following operations:

[0113] Determine whether there is insufficient computing resources by calculating the usage ratio of computing resources;

[0114] By calculating network delay, bandwidth utilization and transmission efficiency, determine whether there is insufficient transmission resources;

[0115] By checking the size and complexity of the data, calculating the data feature load, and determining whether there are data feature anomalies;

[0116] Determine whether there is an imbalance in load distribution.

[0117] Summary: Example 1 proposes an innovative big data integration risk warning method by introducing an integrated efficiency prediction model and a cloud-edge collaborative optimization mechanism. This method takes the division of integrated tasks as the starting point, combines dynamic distribution strategies and real-time execution monitoring, and accurately identifies performance anomalies in the big data integration process by comparing actual efficiency values with predicted efficiency values on the basis of predicting integration efficiency, and effectively solves problems such as unbalanced resource allocation, data anomalies, and node failures through risk warning and cloud-edge collaborative optimization. At the same time, the accuracy and reliability of efficiency prediction are improved through feature extraction, nonlinear mapping, and periodic correction of the integrated efficiency prediction model. This embodiment is significantly innovative in technology, especially in terms of dynamic distribution, fault isolation, and cloud-edge collaborative model optimization, providing an efficient and stable solution for big data integration.

[0118] Example 2, Prospect Summary: In Example 1, the actual value of integration efficiency E is calculated by dynamically monitoring the completion time and the amount of data processed by the large data to be integrated. act ; Output the integrated efficiency prediction value E through the integrated efficiency prediction model pred ; If the actual value of the integrated efficiency E act The integrated efficiency theoretical value E pred If the difference exceeds the preset value, an early warning is issued to the user, indicating that the big data integration process is abnormal; although the supervision of big data integration is achieved, it tends to be post-supervision and cannot identify anomalies from the real-time big data integration scenario, thereby avoiding the actual value of integration efficiency Eact Anomaly; Based on this, a big data integration process monitoring module, a key node performance correlation rule mining module, an adaptive performance bottleneck diagnosis module, and a dynamic scheduling module in the embodiments of the present invention are proposed. Through combination, the management and control of the big data integration process are realized, effectively avoiding the actual value E of the integration efficiency act Anomaly, which has practical significance;

[0119] Refer to Figure 2 For the structure block diagram of the big data integration system in, the difference between the embodiment of the present invention and Embodiment 1 is that the data integration execution module includes a bottleneck node control unit, including:

[0120] Key node monitoring: Obtain the real-time performance index data of key nodes (including data cleaning nodes, data conversion nodes, and data storage nodes) in the big data integration process. These indexes have a significant impact on the processing efficiency of the big data to be integrated. The performance index data includes data throughput, processing delay, CPU utilization rate, memory utilization rate, average execution time of specific tasks, and failure rate;

[0121] Perform multi-time scale decomposition on the real-time performance index data through fast Fourier transform or wavelet transform, and decompose it into short-term fluctuations and long-term trends; the short-term fluctuations are used to monitor real-time performance anomalies and trigger immediate alarms; the long-term trends are used to predict future performance bottlenecks of key nodes and provide data support for dynamic scheduling; the decomposed real-time performance index data is stored in a time series database for subsequent analysis and optimization;

[0122] Key node performance correlation rule mining: By combining static correlation rule mining algorithms and deep learning models, construct a multi-level performance correlation rule set; static rules are generated based on historical data and expert knowledge, such as "the processing delay value corresponding to a specific throughput range"; dynamic rules use deep learning models to real-time mine complex correlation relationships between performance indexes based on time series data; the multi-level performance correlation rule set is used for performance bottleneck diagnosis and dynamic scheduling. Static rules quickly identify bottlenecks, and dynamic rules adapt to real-time changes in bottlenecks, thereby improving the system response ability and adaptability;

[0123] Adaptive performance bottleneck diagnosis: According to the real-time performance index data and the multi-level performance correlation rule set, dynamically adjust the diagnosis strategy, identify the bottleneck type and generate a bottleneck diagnosis report. The bottleneck diagnosis combines short-term fluctuation data and the correlation rule set to distinguish instantaneous bottlenecks (such as task peaks) and long-term bottlenecks (such as resource shortages); the bottleneck diagnosis report includes the bottleneck node number, the performance bottleneck type, and the impact range on the integration efficiency (such as the duration of efficiency decline and the proportion of efficiency decline);

[0124] Dynamic scheduling, based on the bottleneck diagnosis report and the multi-level performance association rule set, uses a time series prediction model to predict the future performance index data of key nodes, dynamically adjusts the data flow allocation strategy and the resource expansion strategy, and forms a closed-loop optimization process; the data flow allocation strategy dynamically adjusts the priority according to the performance index of the key node by optimizing the task routing; when the predicted index is close to the bottleneck threshold, the task allocation is adjusted to avoid node overload; the resource expansion strategy is triggered when the predicted performance index exceeds the set threshold for a long time (such as CPU utilization > 90% for 10 minutes continuously), and the expansion logic includes adding distributed nodes or increasing the computing resources (such as memory, number of CPU cores) of the existing nodes. The entire optimization process monitors the adjustment effect in real time, uses the feedback data to update the time series model, and continuously improves the scheduling and resource management strategies.

[0125] Explanation: The connection and difference between the key node and the computing node

[0126] Connection: Both the key node and the computing node are core components of big data processing; the performance of the key node may be completed through the cooperation of one or more computing nodes, and the running state of the computing node directly affects the efficiency of the key node;

[0127] Difference: The key node represents the data processing link at the logical level (such as cleaning, transformation, storage), which belongs to the functional division of the data processing process; the computing node represents the running unit at the physical level (such as a server, a virtual machine), which is used to execute tasks; the key node usually involves the division of business logic and tasks, while the computing node involves the allocation and management of underlying computing resources.

[0128] Summary: Embodiment 2 improves the limitations of Embodiment 1, extends from ex post supervision to real-time monitoring and management, and proposes a more proactive and real-time big data integration management and control method. Through the combined application of key node monitoring, key node performance association rule mining, adaptive performance bottleneck diagnosis, and dynamic scheduling, the real-time optimization and closed-loop control of the big data integration process are realized, and the abnormality of the actual value of the integration efficiency is avoided.

[0129] Finally: The above are only the preferred embodiments of the present invention and are not used to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A big data integration system based on artificial intelligence, characterized in that, Including: An integrated task division module, which obtains the big data to be integrated and divides the big data to be integrated into multiple time period data according to a set fixed time interval; A data integration execution module, after completing the division of the time period data, distributes each time period data to the computing nodes for big data integration according to a pre-determined resource allocation strategy and the running status of each computing node; An integrated efficiency prediction model construction module, during the execution of the integration, obtains the preset running information of the computing nodes, the resource allocation strategy, the characteristics of the data to be integrated, and the environmental context information in real time as the input information of the integrated efficiency prediction model, and constructs the integrated efficiency prediction model; An integrated risk warning module is used to warn of abnormal integration efficiency. By dynamically monitoring the completion time and the amount of processed data of a large amount of data to be integrated, the actual value E of the integration efficiency is calculated. act The predicted value E of the integration efficiency is output through the integration efficiency prediction model. pred ; If the actual value E of the integration efficiency act and the theoretical value E of the integration efficiency pred have a difference exceeding the preset value, a warning is issued to the user, indicating that the big data integration process is abnormal.

2. The big data integration system based on artificial intelligence according to claim 1, characterized in that, The data integration execution module includes an abnormal computing node control unit, and the abnormal computing node control unit is used to identify abnormal computing nodes and dynamically adjust the distribution strategy based on the abnormal computing nodes; The process of identifying abnormal computing nodes includes: continuously monitoring the running metrics of each computing node and comparing with the historical baseline or health threshold; once an abnormal fluctuation or a fault alarm of a computing node is detected, it is marked as a fault / deterioration state; The process of dynamically adjusting the distribution strategy based on the abnormal computing nodes includes: Temporarily isolating and transferring critical tasks: for the nodes marked as faulty / deteriorated, temporarily stop the distribution of new data segments; if there is important big data to be integrated that it is processing, then transfer the remaining tasks to other normal nodes for execution according to the integration progress and the data volume; Re-evaluating the resource configuration of the remaining available nodes: dynamically adjusting the priority of task transfer based on the remaining load of the computing nodes, the urgency of the integration tasks, and the network topology; Continuously monitoring and restoring the computing nodes: performing fault troubleshooting or performance tuning on the faulty computing nodes, and after restoring to a healthy state, incorporating them into the integration task distribution queue.

3. The big data integration system based on artificial intelligence according to claim 1, wherein, The process of building the integrated efficiency prediction model includes the following steps: Step S01, feature extraction: extracting features related to the integration efficiency from the input information as input features; the input information includes the preset running information of the nodes, the resource allocation strategy parameters, the characteristics of the data to be integrated, and the environmental context information; Step S02, preparing training data and labels: collecting historical data during the big data integration process, including input features and historical actual efficiency, using the historical actual efficiency index as the label, and constructing a training data set and a validation data set; Step S03, selecting the model architecture and loss function: selecting a machine learning algorithm to build the model, initializing the parameters, setting the loss function, and outputting the selected initialized model; Step S04, model training and validation: performing iterative training on the selected initialized model, adjusting the network parameters and hyperparameters, and evaluating the prediction accuracy through the validation set; until the loss function meets the preset requirements, outputting the trained integrated efficiency prediction model; Step S05, model deployment and application: deploying the trained integrated efficiency prediction model in the big data integration system, obtaining the time period data in real time, and outputting the integrated efficiency prediction value.

4. An artificial intelligence-based big data integration system according to claim 3, characterized in that, The acquisition method of the loss function is: Performing several tests on the integrated efficiency prediction model, and recording the actual running information of the nodes, the actual value of the integrated efficiency, and the theoretical value of the integrated efficiency for each test; If the actual operating information of the node is the same as the preset operating information of the node, it is marked as a valid test; Suppose there are N valid tests, and let i represent the sequential number of the valid test; obtain the actual value E of the integration efficiency of the i-th valid test act_i and the theoretical value E of the integration efficiency pred_i ; Use the actual value E of the integration efficiency act_i and the theoretical value E of the integration efficiency pred_i The difference represents the loss function.

5. An artificial intelligence-based big data integration system according to claim 4, characterized in that, The integrated efficiency prediction model satisfies the following formula: where S represents the total scale or total duration interval of the data to be processed, x is the integration variable from 0 to S, used to depict the continuous evolution process of data or time, H(x) represents the information entropy on the data block or time segment x, H(x) is obtained through the information entropy calculation formula, P(x) represents the resource scheduling function that changes with x; DO represents the data source diversity impact factor; used to convert the input features into interval values from 0 to 1; assuming there are m input features, j is used to represent the sequential number of the input features; the influence intensity of the input features is obtained through the following formula: where α j is the weight of the j-th input feature, Tx j is the eigenvalue of the j-th input feature, and γ is the power exponent; λsin(ω·S) is the periodic correction term, λ is the periodic intervention weight coefficient, ω is the periodic frequency; Γ(H) is the Gamma function of the entropy value H, and H is the average of all H(x).

6. The big data integration system based on artificial intelligence according to claim 1, wherein The integrated task division module is connected to the cloud-edge collaborative optimization module, and the cloud-edge collaborative optimization module preprocesses the big data, screens out abnormal data, and improves the integrated efficiency prediction value, including the following steps: Deploy the edge abnormal data analysis model and the cloud abnormal data analysis model. The edge abnormal data analysis model is used to detect abnormal features in the data stream in real time, mark the identified first abnormal data, and upload it to the cloud; the cloud abnormal data analysis model combines the analysis results of the edge side to perform abnormal analysis on the aggregated big data, identify the second abnormal data, and generate high-quality time period data; preferentially transmit the high-quality time period data to the computing node to improve the integrated efficiency of the big data set.

7. An artificial intelligence-based big data integration system according to claim 6, characterized in that, If the performance index of the edge abnormal data analysis model is lower than the requirement, an alarm is issued for the abnormality of the edge abnormal data analysis model, and the parameters of the edge abnormal data analysis model are optimized through the cloud abnormal data analysis model.

8. An artificial intelligence-based big data integration system according to claim 6, characterized in that, The cloud-edge collaborative optimization module further includes a synchronization update management unit, and the synchronization update management unit includes: Periodically align the parameters of the edge abnormal data analysis model with the cloud abnormal data analysis model at the edge side, and the parameter alignment adopts the gradient aggregation or weight alignment method; Establish a model version library to identify the abnormal data analysis model currently used by each edge node, and at the same time maintain a version parameter update log on the central side; Dynamically determine the data transmission method according to the importance and real-time nature of the on-site data; If the data abnormality level is relatively high or in-depth analysis is required, immediately transmit the original data back to the cloud abnormal data analysis model; If the data importance is relatively low, it is first quickly processed at the edge and the summary is generated and then transmitted back to the cloud.

9. An artificial intelligence-based big data integration system according to claim 1, characterized in that, If the actual value E of the integration efficiency act and the theoretical value E of the integration efficiency pred have a difference exceeding the preset value, an abnormal analysis instruction is generated; after receiving the abnormal analysis instruction, the abnormal cause analysis module includes the following operations: Judge whether there is a shortage of computing resources by calculating the proportion of computing resource usage; Judge whether there is a shortage of transmission resources by calculating the network delay, bandwidth utilization rate and transmission efficiency; Calculate the data feature load by checking the size and complexity of the data, and judge whether there is an abnormal data feature; Judge whether there is an unbalanced load distribution problem.

10. A big data integration system based on artificial intelligence according to claim 1, characterized in that, The data integration execution module includes a bottleneck node control unit, and the bottleneck node control unit includes: Key node monitoring: Obtain real-time performance metric data of key nodes during the big data integration process; perform multi-time scale decomposition on the real-time performance metric data through fast Fourier transform or wavelet transform, decomposing it into short-term fluctuations and long-term trends; the short-term fluctuations are used to monitor real-time performance anomalies and trigger immediate alerts; the long-term trends are used to predict future performance bottlenecks of key nodes; Mining of key node performance association rules: Construct a multi-level performance association rule set by combining static association rule mining algorithms and deep learning models; the multi-level performance association rule set is used for performance bottleneck diagnosis and dynamic scheduling; Adaptive performance bottleneck diagnosis: Dynamically adjust the diagnosis strategy according to real-time performance metric data and the multi-level performance association rule set, identify the bottleneck type and generate a bottleneck diagnosis report; the bottleneck diagnosis report includes the bottleneck node number, the performance bottleneck type, and the impact range on the integration efficiency; Dynamic scheduling, based on the bottleneck diagnosis report and the multi-level performance association rule set, uses a time series prediction model to predict the future performance metric data of key nodes, dynamically adjusts the data flow allocation strategy and resource expansion strategy, and forms a closed-loop optimization process.

Images