Data change trend prediction method for big data analysis platform

Abstract

The invention relates to the technical field of prediction analysis, in particular to a data change trend prediction method for a big data analysis platform, which comprises the following steps: collecting data sets of differentiated sources, recording business event data of the big data analysis platform, classifying and sorting the data according to timestamps and event types of the data, and obtaining a data change trend prediction result. And obtaining the optimized sorted data frame. In the invention, through key variable recording and analysis, including event time, event intensity and node connection quantity, enhancement of visualization and understanding of association between data points, construction of a dependency graph structure, training by adopting a graph convolutional network, and enhancement of deep mining of data characteristics, especially application in the aspect of abnormal mode identification, an abnormal mode identification result is obtained. More accurate data change trend prediction is provided, the timeliness of prediction is improved through application of time sequence analysis, the adaptability of the model to future events is enhanced, and the quality and efficiency of decision making are improved in practical application.

Description

Technical Field

[0001] The present invention relates to the field of predictive analysis technology, and in particular to a method for predicting data change trends on a big data analysis platform. Background Art

[0002] Predictive analytics is a technical field that applies statistics, data mining, machine learning, and artificial intelligence techniques to analyze historical and real-time data to predict the development trend of future or unknown events. The core purpose of this field is to support the decision-making process by building predictive models to estimate the future behavior of data. In a business environment, predictive analytics helps companies identify opportunities and risks, improve resource allocation, optimize marketing activities, and improve customer service efficiency. Common applications include sales forecasting, customer behavior analysis, inventory management, financial market trend forecasting, etc.

[0003] Among them, the data change trend prediction method of the big data analysis platform refers to the use of big data technology to predict the trend of data changes, and is applied to platforms that need to process and analyze large-scale data sets. This method analyzes historical data sets and applies statistical analysis and machine learning algorithms to predict future data changes and trends. It can help organizations better understand the long-term trends of data and provide a scientific basis for business decisions, market analysis, and resource allocation.

[0004] Existing predictive analysis technologies face the problems of slow processing speed and insufficient accuracy when processing large-scale data sets. Especially in the context of diversified data sources and increasing data volumes, traditional methods are difficult to effectively extract useful information from complex data relationships, limiting the depth and breadth of data analysis. Existing technologies also have obvious deficiencies in the adaptability and flexibility of predictive models, making it difficult to adjust models in a timely manner to cope with rapidly changing data characteristics and external environments. This rigid analysis framework makes the prediction results lack pertinence and foresight, affecting the efficiency of enterprises in resource allocation and market strategy formulation. For example, the lack of effective anomaly detection mechanisms and dynamic learning capabilities will lead to slow responses when responding to sudden market changes, resulting in missed business opportunities or unnecessary risks. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a method for predicting data change trends on a big data analysis platform.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: a data change trend prediction method of a big data analysis platform, comprising the following steps: S1: Collect data sets from different sources, record business event data of the big data analysis platform, classify and sort the data according to the timestamp and event type of the data, and obtain an optimized sorted data frame; S2: Based on the optimized sorted data frame, select key variables, record the time of event occurrence, event intensity and the number of node connections, use the variables as node attributes, analyze the degree of correlation between data points, and construct a dependency graph structure; S3: Using the dependency graph structure, setting the weights between nodes and the connection strength of edges, using data characteristics to set parameters and train the graph convolutional network, analyzing the connection characteristics and propagation paths of the nodes, and obtaining abnormal pattern characteristics; S4: Detect data change trends through the abnormal pattern features, conduct dynamic learning in combination with historical data, use time series analysis to predict changes and abnormal developments in the next week, and build a trend prediction model; S5: Based on the trend prediction model, the model parameters and graph structure are adjusted in real time, and the consistency of the prediction model is checked according to the current data characteristics and environmental changes to form an optimized graph structure; S6: Using the optimized graph structure, continue to monitor changes in data flows in the big data analysis platform, verify the model's responsiveness to real-time situations and prediction range, and generate model performance evaluation results.

[0007] As a further solution of the present invention, the optimized sorting data frame includes timestamps, types, associated participating nodes and key information of multiple events, the dependency graph structure includes a network diagram of dependencies between nodes, event occurrence time, event intensity and number of node connections, the abnormal pattern features include identified unconventional node activities, abnormal signal propagation paths and potential data leakage points, the trend prediction model includes future trend predictions, potential risk points and predicted range of changes obtained based on historical data analysis, the optimized graph structure includes adjusted node attributes, reconfigured network connections and data processing capabilities, and the model performance evaluation results include data flow change response speed, prediction consistency and real-time data monitoring capabilities.

[0008] As a further solution of the present invention, the steps of collecting data sets from different sources, recording business event data of the big data analysis platform, classifying and sorting the data according to the timestamp and event type of the data, and obtaining an optimized sorted data frame are specifically as follows: S101: Automatically collect business event data from multiple data sources, import them into the big data analysis platform, adjust the data format uniformly, check the consistency of the data, and process the data to avoid incomplete and invalid records, and generate a processed data set; S102: traversing data items from the processed data set, extracting timestamps and event types from the data, sorting the data in ascending order using the timestamps, and grouping the data according to the event types to generate time series classification data; S103: Using the time series classification data, a decision tree algorithm is used to adjust the sorting rules within each group of data, and sorting optimization is performed according to the frequency and time of event occurrence, thereby optimizing the efficiency of data retrieval and generating an optimized sorting data frame.

[0009] As a further solution of the present invention, the formula of the decision tree algorithm is as follows: , Among them, S is the average ranking index, represents the frequency of occurrence of the ith event, represents the time when the i-th event occurs, and is the weight coefficient for adjusting frequency and time importance, and n is the total number of events.

[0010] As a further solution of the present invention, based on the optimized sorted data frame, key variables are selected, event occurrence time, event intensity and number of node connections are recorded, variables are used as node attributes, and the degree of association between data points is analyzed. The steps of constructing a dependency graph structure are specifically as follows: S201: Filter key variables from the optimized sorting data frame, including event occurrence time, event intensity, and number of node connections, use the variables to define the attributes of each data point, determine the basic characteristics of each event node, and generate a node attribute set; S202: Analyze the potential connections between the data points in the node attribute set, calculate the correlation between event time, intensity and number of connections through statistical analysis, determine the correlation strength between the data points, and generate a correlation analysis result; S203: Based on the correlation analysis result, each node is used to represent an event, and the edges between nodes reflect the correlation between events, so as to generate a dependency graph structure.

[0011] As a further solution of the present invention, the dependency graph structure is adopted to set the weights between nodes and the connection strength of edges, and the graph convolution network is parameterized and trained using data characteristics to analyze the connection characteristics and propagation paths of the nodes. The specific steps of obtaining abnormal pattern characteristics are as follows: S301: Based on the dependency graph structure, weights and strengths are set for each node and edge connection, weight values ​​are adjusted according to event strength and the number of node connections in the data, information fluidity and responsiveness are optimized, and a weight setting graph is generated; S302: Using the weight setting graph, setting the parameters of the graph convolutional network, including the learning rate, activation function type and number of network layers, adjusting the network structure according to the connection characteristics of the nodes and the propagation path, performing network training, and generating training parameter configuration; S303: Using the training parameter configuration, analyzing the connection characteristics and signal propagation paths between nodes, identifying and recording node behaviors and connection patterns that do not conform to normal patterns, and generating abnormal pattern characteristics.

[0012] As a further solution of the present invention, the data change trend is detected through the abnormal pattern characteristics, dynamic learning is performed in combination with historical data, and time series analysis is used to predict changes and abnormal developments within the next week. The steps of constructing a trend prediction model are specifically as follows: S401: Analyze the abnormal pattern characteristics, evaluate the data change trend of the differentiated nodes, dynamically learn and update the abnormal pattern in combination with the data records, capture the early signals of the changes, and generate a dynamic learning model; S402: using the dynamic learning model and applying a time series analysis method, predicting data changes and abnormal developments within the next week, adjusting the analysis frequency and parameter sensitivity, and generating predictive analysis data; S403: Based on the predictive analysis data, the response threshold and alarm mechanism of the model are set to respond to data anomalies and trend changes, and a trend prediction model is constructed.

[0013] As a further solution of the present invention, based on the trend prediction model, the model parameters and the graph structure are adjusted in real time, and the consistency of the prediction model is checked according to the current data characteristics and environmental changes. The steps of forming an optimized graph structure are specifically as follows: S501: using the trend prediction model, dynamically adjusting model parameters and graph structure according to real-time data and environmental changes, verifying the response speed of the model output, and generating adjusted model parameters; S502: Using the adjusted model parameters, by simulating differentiated data scenarios, testing the consistency and responsiveness of the trend prediction model, adjusting weights and connection strengths, evaluating and optimizing the prediction ability of the model, and generating consistency detection results; S503: Based on the consistency detection result, configure the layout and characteristics of nodes and edges, optimize the sensitivity of the model to future event prediction, and generate an optimized graph structure.

[0014] As a further solution of the present invention, the optimized graph structure is used to continue monitoring the changes in the data flow in the big data analysis platform, verify the model's responsiveness and prediction range to real-time situations, and generate the model performance evaluation results in the following steps: S601: Using the optimized graph structure and the sliding window technology, monitor the data flow changes in the big data analysis platform, track the data characteristics and evolution trends, and generate real-time monitoring records; S602: Based on the real-time monitoring records, verify the responsiveness of the trend prediction model to real-time data changes, evaluate the prediction range of the trend prediction model under differentiated data conditions and environmental changes, and generate responsiveness analysis results; S603: Evaluate the performance of the trend prediction model through the response capability analysis results, calibrate the deviation of the model prediction, calculate the error range and analyze the response time, adjust the model parameters and structure according to the evaluation results, and generate the model performance evaluation results.

[0015] As a further solution of the present invention, the formula of the sliding window technology is as follows: , in, is the new window size, Represents the number of edges in the current graph structure, and K is a constant preset based on performance. is the adjustment coefficient, Represents the maximum degree of the node in the current window, and is the weight coefficient, is the average number of connections of a node.

[0016] Compared with the prior art, the advantages and positive effects of the present invention are: In the present invention, by collecting data sets from differentiated sources and recording various types of business event data, the data is classified and sorted, and the accuracy and efficiency of data processing are effectively improved. The visualization and understanding of the relationship between data points are enhanced by recording and analyzing key variables, including event time, intensity, and number of node connections. A dependency graph structure is constructed and trained using a graph convolutional network to strengthen the in-depth mining of data characteristics, especially in the application of abnormal pattern recognition, to provide more accurate data change trend predictions. The application of time series analysis not only improves the timeliness of predictions, but also enhances the model's adaptability to future events. Through real-time adjustment of model parameters and graph structures, data characteristics and environmental changes can be effectively responded to, ensuring the consistency and accuracy of the prediction model. The optimized graph structure continues to monitor changes in data flows, enhances the model's ability to respond to real-time situations, and improves the quality and efficiency of decision-making in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of the main steps of the present invention; Figure 2 It is a schematic diagram of the refinement of S1 of the present invention; Figure 3 It is a schematic diagram of the refinement of S2 of the present invention; Figure 4 It is a schematic diagram of the refinement of S3 of the present invention; Figure 5It is a schematic diagram of the refinement of S4 of the present invention; Figure 6 It is a detailed schematic diagram of S5 of the present invention; Figure 7 It is a detailed schematic diagram of S6 of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0019] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, in the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined.

[0020] See also Figure 1 The present invention provides a technical solution, a method for predicting data change trends on a big data analysis platform, comprising the following steps: S1: Collect data sets from different sources, record business event data of the big data analysis platform, classify and sort the data according to the timestamp and event type of the data, and obtain an optimized sorted data frame; S2: Based on the optimized sorted data frame, select key variables, record the event time, event intensity and number of node connections, use the variables as node attributes, analyze the degree of correlation between data points, and build a dependency graph structure; S3: Use the dependency graph structure to set the weights between nodes and the connection strength of edges, use data characteristics to set parameters and train the graph convolutional network, analyze the connection characteristics and propagation paths of nodes, and obtain abnormal pattern characteristics; S4: Detect data change trends through abnormal pattern characteristics, combine historical data for dynamic learning, use time series analysis to predict changes and abnormal developments in the next week, and build a trend prediction model; S5: Based on the trend prediction model, the model parameters and graph structure are adjusted in real time. According to the current data characteristics and environmental changes, the consistency of the prediction model is checked to form an optimized graph structure. S6: Using the optimized graph structure, continue to monitor the changes in data flow in the big data analysis platform, verify the model's responsiveness and prediction range to real-time situations, and generate model performance evaluation results.

[0021] The optimized sorting data frame includes the timestamps, types, associated participating nodes and key information of multiple events. The dependency graph structure includes a network diagram of the dependencies between nodes, event occurrence time, event intensity and number of node connections. The abnormal pattern features include identified unconventional node activities, abnormal signal propagation paths and potential data leakage points. The trend prediction model includes future trend predictions, potential risk points and predicted range of changes based on historical data analysis. The optimized graph structure includes adjusted node attributes, reconfigured network connections and data processing capabilities. The model performance evaluation results include data flow change response speed, prediction consistency and real-time data monitoring capabilities.

[0022] See also Figure 2 , collect data sets from different sources, record business event data of the big data analysis platform, classify and sort the data according to the timestamp and event type of the data, and obtain the optimized sorted data frame in the following steps: S101: Automatically collect business event data from multiple data sources, import them into the big data analysis platform synchronously, adjust the data format uniformly to check the consistency of the data, and process the data to avoid incomplete and invalid records. The execution process of generating the processed data set is as follows; In the process of automatic collection of business event data, the collaborative work of multiple data sources is crucial. The information of the data source is synchronously transmitted to the big data analysis platform through a specific protocol. Each data item must undergo format standardization before importing to ensure that the data format is unified and meets the requirements of the analysis platform. The process includes checking and adjusting the data format. The consistency check of the data is carried out by comparing the differences in data formats from different sources and unifying them into a standard template. The data processing stage involves screening out incomplete or invalid records and generating a processed data set.

[0023] S102: traverse the data items from the processed data set, extract the timestamp and event type in the data, sort in ascending order using the timestamp, and group the data according to the event type to generate the execution flow of time series classification data as follows; In the processed data set, the data items are traversed, and the timestamps and event types in the data are extracted item by item. The extracted timestamps are used for time sorting of the data. All timestamps are arranged in ascending order to ensure that the data items are presented in chronological order. The data are grouped according to the extracted event types. The data of each event type forms an independent data sequence to ensure that similar events belong to the same sequence. It has the dual dimensions of time sequence and event category, providing a more detailed classification structure for subsequent analysis and generating time series classification data.

[0024] S103: By using the time series classification data, the decision tree algorithm is used to adjust the sorting rules within each group of data, and the sorting is optimized according to the frequency and time of event occurrence, so as to optimize the efficiency of data retrieval and generate the optimized sorting data frame. The execution process is as follows; The formula for the decision tree algorithm is as follows: , Among them, S is the average ranking index, represents the frequency of occurrence of the ith event, represents the time when the i-th event occurs, and is the weight coefficient for adjusting frequency and time importance, and n is the total number of events.

[0025] Explain the meaning and function of each parameter: Represents the occurrence frequency of the i-th event, reflecting the frequency of the event in the data set. Represents the timestamp of the i-th event, reflecting the specific time when the event occurred. is the frequency weight coefficient, which is used to adjust the importance of event frequency in the sorting. is the time weight coefficient, which is used to adjust the importance of event time in sorting. n represents the total number of events, which is used to calculate the weighted average.

[0026] Detailed explanation of the formula and the process of formula calculation and derivation: The event priority is determined by taking into account the frequency and time of the event through the weighted average method. The frequency of the event needs to be quantified. and , according to the weights set and Adjust the influence of the two and calculate the average ranking index S of all events.

[0027] Frequency of events It is obtained by dividing the number of occurrences of an event during the observation period by the total length of time. For example, if an event occurs 15 times during a 30-day observation period, then Times / day.

[0028] Event time is the time from the start of observation to the occurrence of the event, measured in days. If the event occurs on the 10th day of the observation period, then =10 days.

[0029] Weight and It should be set according to the nature of the event and the research needs. If the frequency of events is more important, you can increase If the accuracy of event time is more critical, increase . Set the weights to =0.6 and =0.4, indicating that frequency has a slightly greater influence than time in the calculation.

[0030] Specific example: There are three events set up, and the specific data are as follows: Event 1: Frequency =0.5, time =10 days; Event 2: Frequency = 0.3, time =15 days; Event 3: Frequency =0.2, time =20 days; Calculation process: , The results show that, considering the weighted influence of the frequency and time of event occurrence, the average ranking index is 6.2, which represents the comprehensive importance of the event under the set weight. The numerical value can be used to further analyze the priority of events, provide basic data for decision tree algorithms, and optimize the ranking of time series data.

[0031] See also Figure 3 ,Based on the optimized sorted data frame, we select key variables, record the event occurrence time, event intensity and the number of node connections, use the variables as node attributes, analyze the correlation between data points, and build the dependency graph structure in the following steps: S201: Filter key variables from the optimized sorting data frame, including event occurrence time, event intensity, and number of node connections, use variables to define the attributes of each data point, determine the basic characteristics of each event node, and generate the node attribute set. The execution process is as follows; In the process of optimizing the sorted data frame, key variables are screened, including event occurrence time, event intensity and number of node connections. The variables are used to define the attributes of each data point. The event occurrence time provides time positioning for the data point, the event intensity reflects the influence or importance of the event, and the number of node connections indicates the connection status of the data point in the network. Key variables are used to determine the basic characteristics of each event node and generate a set of node attributes.

[0032] S202: Analyze the potential connections between the data points in the node attribute set, calculate the correlation between the event time, intensity and the number of connections through statistical analysis, determine the correlation strength between the data points, and generate the execution flow of the correlation analysis result as follows; The correlation between event time, intensity and number of connections was calculated by statistical analysis, according to the formula , calculate the correlation strength between data points. In the formula, represents the strength of association between data points a and b, , and Represent the event time, event intensity and number of connections of data point a respectively. Formula explanation and formula calculation derivation process: Assume that the data set D contains elements , represents the event time, strength, and number of connections for each data point. It is obtained by calculating the ratio of the weighted sum of the three variables to the square root of the sum of their squares, which is similar to calculating cosine similarity. The specific calculation process is as follows: Calculate the sum of the squares of each variable, and the sum of the squares of the event time of all data points is , the sum of the squares of the event intensities is , the sum of the squares of the number of connections is , for each pair of data points a and b, calculate the weighted sum , calculated by the value .

[0033] S203: Based on the correlation analysis result, each node is used to represent an event, and the edges between nodes reflect the correlation between events. The execution flow of generating a dependency graph structure is as follows; Based on the results of correlation analysis, each node represents an independent event, and the edges between nodes reflect the degree of correlation between events. It not only shows the interdependence between events, but also provides a basic framework for further network optimization and path analysis. The data points in the node attribute set are linked according to the correlation strength, and the thickness or color of the edge indicates different degrees of correlation strength. The generation of each edge depends on the specific value of the correlation analysis result, which further reveals the complexity of the data structure and the interactive characteristics of the network, and generates a dependency graph structure.

[0034] See also Figure 4 , using the dependency graph structure, setting the weights between nodes and the connection strength of edges, using data characteristics to set parameters and train the graph convolutional network, analyzing the connection characteristics and propagation paths of nodes, and obtaining the abnormal pattern characteristics in the following steps: S301: Based on the dependency graph structure, weights and strengths are set for each node and edge connection, and weight values ​​are adjusted according to event strength and the number of node connections in the data to optimize information fluidity and responsiveness. The execution flow of generating a weight setting graph is as follows; Set the weight and strength of each node-edge connection according to the formula , calculate the weight setting diagram. In the formula, represents the connection weight between node d and node f, and Represent the adjustment coefficients of event intensity and number of connections, and Represent the event intensity and number of connections of node d respectively. Formula explanation and formula calculation derivation process: Assume that the data set D includes elements , represents the event intensity and number of connections of each node. Weight By considering the event intensity of each node and number of connections Calculation is performed, where the adjustment factor and Determined according to actual needs. The specific calculation process is as follows: For each pair of nodes d and f, calculate the linear combination of event intensity and number of connections . Determine the weight of each connection , affecting the fluidity of information and the responsiveness of the network.

[0035] S302: Using the weight setting graph, set the parameters of the graph convolutional network, including the learning rate, activation function type and number of network layers, adjust the network structure according to the connection characteristics of the nodes and the propagation path, perform network training, and generate the execution process of the training parameter configuration as follows; A weight setting graph is used to set the parameters of the graph convolutional network, including the configuration of the learning rate, activation function type and number of network layers. The connection characteristics and propagation paths of the nodes are dynamically adjusted according to the actual connection and historical behavior of the data points. The setting of the network structure depends on the strength and direction of the association between the nodes. The key to setting the learning rate is to ensure the stability and convergence speed during training. The choice of activation function affects the nonlinear processing capability of the network. The configuration of the number of network layers is directly related to the depth and complexity of the model. During the network training process, parameter optimization is achieved through iterative updates to adapt to the ever-changing network structure and data flow, and generate training parameter configurations.

[0036] S303: Using the training parameter configuration, analyzing the connection characteristics and signal propagation paths between nodes, identifying and recording node behaviors and connection patterns that do not conform to the normal pattern, and generating the execution flow of abnormal pattern features are as follows; After using the training parameter configuration, the connection characteristics and signal propagation paths between nodes are analyzed. The process includes identifying and recording node behaviors and connection patterns that do not conform to normal patterns. Through in-depth observation of network behavior, the interaction dynamics and transmission efficiency between nodes are analyzed. It relies on statistical analysis of node behavior and comparison of behavior patterns, including detection of abnormal signals and identification of behavioral deviations. Through pattern recognition technology, abnormal behaviors are extracted from batches of network interaction data. The potential causes and impacts behind the behaviors are further analyzed to provide reference and guidance for network security and performance optimization, and to generate abnormal pattern characteristics.

[0037] See also Figure 5, detect data change trends through abnormal pattern characteristics, combine historical data for dynamic learning, and use time series analysis to predict changes and abnormal developments in the next week. The specific steps of building a trend prediction model are: S401: Analyze the abnormal pattern characteristics, evaluate the data change trend of the differentiated nodes, combine the data records, dynamically learn and update the abnormal pattern, capture the early signals of the change, and generate the dynamic learning model. The execution process is as follows; In the process of analyzing abnormal pattern characteristics, evaluating the data change trend of differentiated nodes becomes the key. Combining data records to dynamically learn and update abnormal patterns is a crucial step. The process involves capturing early signals of changes and adjusting the dynamic learning model in real time. By continuously monitoring data changes, it can adapt to the changing environment and improve the accuracy and response speed of predictions. The update of the model depends on the rapid identification and learning of emerging data patterns and the re-evaluation of known patterns. The continuous learning process ensures that the model can adjust its behavior in time to cope with new or unforeseen data changes and generate a dynamic learning model.

[0038] S402: Using the dynamic learning model and applying the time series analysis method, predicting the data changes and abnormal developments within the next week, adjusting the analysis frequency and parameter sensitivity, and generating the forecast analysis data, the execution process is as follows; Applying the time series analysis method, according to the formula , predict data changes and abnormal developments in the next week. In the formula, Represents the predicted value at the next time point, Represents the predicted value at the current time point, represents the actual observation value at the current time point, Represents the smoothing coefficient. Detailed explanation of the formula and the process of formula calculation: Considering the exponential smoothing method commonly used in time series analysis, by adjusting the coefficient To balance the impact of historical data and the importance of recent observations. The specific calculation process is: Set the initial prediction , using actual observed values Update the forecast and calculate the forecast value at each step for times the predicted value at the previous time point plus times the current actual observation value. By gradually updating, the model can dynamically adjust to capture the latest changing trends in the data.

[0039] S403: Based on the prediction and analysis data, the response threshold and alarm mechanism of the model are set to respond to data anomalies and trend changes, and the execution process of building a trend prediction model is as follows; Based on predictive analysis data, the model's response threshold and alarm mechanism are set. The ability to respond to data anomalies and trend changes directly affects the effectiveness and practicality of the model. By setting appropriate response thresholds, the model can issue alarms in a timely manner when the data reaches a specific level of change. The setting of the alarm mechanism involves the identification of abnormal data and the optimization of the response process. Through continuous data analysis, the model can automatically adjust the threshold to adapt to the needs of different situations and improve the accuracy and timeliness of early warnings. It can not only predict future data trends, but also provide rapid response when the data is abnormal, and build a trend prediction model.

[0040] See also Figure 6 Based on the trend prediction model, the model parameters and graph structure are adjusted in real time. According to the current data characteristics and environmental changes, the consistency of the prediction model is checked. The specific steps to form the optimized graph structure are as follows: S501: Using the trend prediction model, dynamically adjusting the model parameters and graph structure according to real-time data and environmental changes, verifying the response speed of the model output, and generating the execution flow of the adjusted model parameters are as follows; Dynamically adjust model parameters and graph structure according to real-time data and environmental changes, according to the formula , verify the reaction speed of the model output. In the formula, represents the adjusted model parameters, represents the original model parameters, Represents parameter values ​​updated based on real-time data, Represents the adjustment coefficient. Detailed explanation of the formula and the process of formula calculation: Setting the initial model parameters , get updated data by real-time monitoring of environmental data changes During the model parameter adjustment process, the old parameters are calculated With real-time update parameters The weighted average of To control the fusion ratio of historical parameters and new data, update each parameter so that It is the weighted sum of the old parameters and the new parameters, which enables the model to respond sensitively and adapt to real-time data changes.

[0041] S502: Using the adjusted model parameters, by simulating differentiated data scenarios, testing the consistency and responsiveness of the trend prediction model, adjusting the weights and connection strengths, evaluating and optimizing the prediction capability of the model, and generating the execution flow of the consistency test results are as follows; Using the adjusted model parameters, the consistency and responsiveness of the trend prediction model are tested by simulating differentiated data scenarios. The purpose of adjusting the weights and connection strengths in the process is to evaluate and optimize the predictive ability of the model. The simulated data scenarios include a variety of environmental variables and data fluctuations. The model's response and prediction results in each case are recorded and analyzed in detail. The model's weights and connection strengths are adjusted based on the simulation results to ensure that the model can maintain efficient consistency and responsiveness in various data environments, providing an important reference for the performance of the model in practical applications and generating consistency test results.

[0042] S503: Based on the consistency detection results, the layout and characteristics of nodes and edges are configured to optimize the sensitivity of the model to the prediction of future events. The execution flow of generating the optimized graph structure is as follows; Based on the consistency test results, the layout and characteristics of nodes and edges are configured to optimize the model's sensitivity to predicting future events. By adjusting the characteristics of node layout and connections, the model can more effectively identify and respond to patterns and changes in the data. The optimization process takes into account the characteristics of each node and the functionality of the connection, ensuring that the position and connection of each node in the graph structure can maximize the prediction ability and data processing efficiency, which not only improves the operational efficiency of the model, but also enhances the predictive sensitivity to future data events, laying a solid foundation for the further application and development of the model, and generating an optimized graph structure.

[0043] See also Figure 7 ,Using the optimized graph structure, continue to monitor the changes in data flow in the big data analysis platform, verify the model's responsiveness and prediction range to real-time situations, and generate the model performance evaluation results in the following steps: S601: Using the optimized graph structure and the sliding window technology, the data flow changes in the big data analysis platform are monitored, and the data characteristics and evolution trends are tracked to generate the execution process of real-time monitoring records as follows; The formula for the sliding window technique is as follows: , in, is the new window size, Represents the number of edges in the current graph structure, and K is a constant preset based on performance. is the adjustment coefficient, Represents the maximum degree of the node in the current window, and is the weight coefficient, is the average number of connections of a node.

[0044] Detailed explanation of the formula and the process of formula calculation and derivation: To ensure that the calculation process is consistent with the actual monitoring data, the following values ​​and sources are set first: : represents the number of edges in the current graph structure; set in a specific monitoring period, detected by real-time data stream analysis tools, there are currently 500 edges; K: A constant preset for system performance. Its value is set based on the system's carrying capacity and expected processing speed. In practical applications, the value is 100, taking into account the actual needs of system performance and operation. : Adjustment coefficient, used to adjust the sensitivity of the window size. It is set to 0.5 based on past experience. The coefficient improves the window's ability to respond to rapid changes in data streams. : The maximum degree of the node in the current window, that is, the maximum number of connections of a node; through data flow analysis, the maximum degree is 50; : Weight coefficient, adjust the average node degree The impact of; Based on the sensitivity analysis of historical data and window size adjustment, it is set to 0.3; : Average node degree, that is, the average number of connections among all nodes in the graph; the average degree is 20 through monitoring; : Basic adjustment value, to prevent the window from being too small, set it to 10 to ensure the continuity of data monitoring.

[0045] Substitute into the formula to calculate: ; Calculate the intermediate steps: calculate , the result is ; Calculate the denominator ; Calculate the entire fraction ; Calculation results ; The results show that the new window size is 0.01694, which represents the dynamic adjustment coefficient of the window in practical applications. ,The platform can respond more flexibly to real-time changes in data streams and optimize the efficiency of data processing and feature tracking. The values ​​reflect that by adjusting the window size, key data changes can be effectively captured while keeping the system performance from being overloaded.

[0046] S602: Based on the real-time monitoring records, verify the responsiveness of the trend prediction model to the real-time data changes, evaluate the prediction range of the trend prediction model under the differentiated data conditions and environmental changes, and generate the responsiveness analysis results. The execution process is as follows; Based on real-time monitoring records, verifying the responsiveness of the trend prediction model to real-time data changes is a key step to ensure the adaptability and effectiveness of the model. In the process, the model is placed under various conditions of data fluctuations and environmental changes. By recording and analyzing the response speed and accuracy of the model under the conditions, the prediction range under different environments is evaluated. Segmented recording and real-time monitoring are used in the verification process to capture the model's response to sudden data changes, provide quantitative support for the model's stability and prediction reliability, and further ensure that the model can accurately track changes in real-time data and generate responsiveness analysis results.

[0047] S603: Evaluate the performance of the trend prediction model through the response capability analysis results, calibrate the deviation of the model prediction, calculate the error range and analyze the response time, adjust the model parameters and structure according to the evaluation results, and generate the model performance evaluation results. The execution process is as follows; Calibrate the deviation of model prediction according to the formula Calculate the error range and analyze the response time. Where E represents the mean absolute error of the model prediction, represents the predicted value of the model at time point t, represents the actual observed value, Represents the number of data points. Detailed explanation of the formula and the process of formula calculation: When calibrating the model prediction deviation, the mean absolute error formula is used to calculate the difference between the prediction result and the actual observation value. The predicted value at each time point t is The actual observed value Take the absolute value of the difference, add up the differences of all time points and divide by the total number of data points The average error E is obtained. The response time is calculated by the time difference from data input to predicted output of the model in real-time monitoring to calibrate the responsiveness of the model prediction.

[0048] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A method for predicting data change trends on a big data analysis platform, characterized in that: The following steps are involved: Collect data sets from different sources, record business event data of the big data analysis platform, classify and sort the data according to the timestamp and event type of the data, and obtain an optimized sorted data frame; Based on the optimized sorted data frame, key variables are selected, the event occurrence time, event intensity and number of node connections are recorded, the variables are used as node attributes, the degree of association between data points is analyzed, and a dependency graph structure is constructed; The dependency graph structure is adopted to set the weights between nodes and the connection strength of edges, and the graph convolution network is parameterized and trained using data characteristics to analyze the connection characteristics and propagation paths of the nodes and obtain abnormal pattern characteristics; Through the abnormal pattern characteristics, the data change trend is detected, and dynamic learning is carried out in combination with historical data. Time series analysis is used to predict changes and abnormal developments in the next week, and a trend prediction model is constructed; Based on the trend prediction model, the model parameters and graph structure are adjusted in real time, and the consistency of the prediction model is checked according to the current data characteristics and environmental changes to form an optimized graph structure; By using the optimized graph structure, the changes in data flow in the big data analysis platform are continuously monitored, the responsiveness and prediction range of the model to real-time situations are verified, and the model performance evaluation results are generated.

2. The data change trend prediction method of the big data analysis platform according to claim 1 is characterized in that: The optimized sorting data frame includes the timestamps, types, associated participating nodes and key information of multiple events, the dependency graph structure includes a network diagram of the dependency relationship between nodes, event occurrence time, event intensity and number of node connections, the abnormal pattern features include identified unconventional node activities, abnormal signal propagation paths and potential data leakage points, the trend prediction model includes future trend predictions, potential risk points and predicted range of changes obtained based on historical data analysis, the optimized graph structure includes adjusted node attributes, reconfigured network connections and data processing capabilities, and the model performance evaluation results include data flow change response speed, prediction consistency and real-time data monitoring capabilities.

3. The data change trend prediction method of the big data analysis platform according to claim 1 is characterized in that: Collect data sets from different sources, record business event data of the big data analysis platform, classify and sort the data according to the timestamp and event type of the data, and obtain the optimized sorted data frame in the following steps: Automatically collect business event data from multiple data sources, import them into the big data analysis platform, adjust the data format uniformly, check the consistency of the data, and process the data to avoid incomplete and invalid records, and generate processed data sets; Traversing data items from the processed data set, extracting timestamps and event types from the data, sorting the data in ascending order using the timestamps, and grouping the data according to event types to generate time series classification data; By using the time series classification data, a decision tree algorithm is used to adjust the sorting rules within each group of data, and the sorting is optimized according to the frequency and time of event occurrence, thereby optimizing the efficiency of data retrieval and generating an optimized sorting data frame.

4. The data change trend prediction method of the big data analysis platform according to claim 3 is characterized in that: The formula of the decision tree algorithm is as follows: ， Among them, S is the average ranking index, represents the frequency of occurrence of the i-th event, represents the time when the i-th event occurs, and is the weight coefficient for adjusting frequency and time importance, and n is the total number of events.

5. The data change trend prediction method of the big data analysis platform according to claim 1 is characterized in that: Based on the optimized sorted data frame, key variables are selected, the event occurrence time, event intensity and number of node connections are recorded, the variables are used as node attributes, the correlation between data points is analyzed, and the steps of constructing a dependency graph structure are as follows: Selecting key variables from the optimized sorted data frame, including event occurrence time, event intensity, and number of node connections, using the variables to define the attributes of each data point, determining the basic characteristics of each event node, and generating a node attribute set; Analyze the potential connections between the data points in the node attribute set, calculate the correlation between event time, intensity and number of connections through statistical analysis, determine the correlation strength between the data points, and generate a correlation analysis result; Based on the correlation analysis result, each node is used to represent an event, and the edges between nodes reflect the correlation degree between events, so as to generate a dependency graph structure.

6. The data change trend prediction method of the big data analysis platform according to claim 1 is characterized in that: The dependency graph structure is adopted to set the weights between nodes and the connection strength of edges, and the parameters of the graph convolutional network are set and trained using data characteristics. The connection characteristics and propagation paths of the nodes are analyzed, and the steps for obtaining abnormal pattern characteristics are as follows: Based on the dependency graph structure, weights and strengths are set for each node and edge connection, weight values ​​are adjusted according to event strength and the number of node connections in the data, information fluidity and responsiveness are optimized, and a weight setting graph is generated; Using the weight setting graph, set the parameters of the graph convolutional network, including the learning rate, activation function type, and number of network layers, adjust the network structure according to the connection characteristics of the nodes and the propagation path, perform network training, and generate training parameter configuration; Using the training parameter configuration, the connection characteristics and signal propagation paths between nodes are analyzed, node behaviors and connection patterns that do not conform to normal patterns are identified and recorded, and abnormal pattern characteristics are generated.

7. The data change trend prediction method of the big data analysis platform according to claim 1 is characterized in that: Through the abnormal pattern characteristics, the data change trend is detected, and dynamic learning is performed in combination with historical data. Time series analysis is used to predict changes and abnormal developments in the next week. The steps of building a trend prediction model are as follows: Analyze the abnormal pattern characteristics, evaluate the data change trend of the differentiated nodes, dynamically learn and update the abnormal pattern in combination with the data records, capture the early signals of the changes, and generate a dynamic learning model; Using the dynamic learning model, applying the time series analysis method, predicting data changes and abnormal developments within the next week, adjusting the analysis frequency and parameter sensitivity, and generating predictive analysis data; Based on the predictive analysis data, the response threshold and alarm mechanism of the model are set to respond to data anomalies and trend changes, and a trend prediction model is constructed.

8. The data change trend prediction method of the big data analysis platform according to claim 1 is characterized in that: Based on the trend prediction model, the model parameters and graph structure are adjusted in real time, and the consistency of the prediction model is checked according to the current data characteristics and environmental changes. The steps to form an optimized graph structure are as follows: Adopting the trend prediction model, dynamically adjusting the model parameters and graph structure according to real-time data and environmental changes, verifying the response speed of the model output, and generating adjusted model parameters; Using the adjusted model parameters, by simulating differentiated data scenarios, testing the consistency and responsiveness of the trend prediction model, adjusting weights and connection strengths, evaluating and optimizing the prediction ability of the model, and generating consistency test results; Based on the consistency detection result, the layout and characteristics of nodes and edges are configured, the sensitivity of the model to the prediction of future events is optimized, and an optimized graph structure is generated.

9. The data change trend prediction method of the big data analysis platform according to claim 1 is characterized in that: Using the optimized graph structure, continue to monitor the changes in data flow in the big data analysis platform, verify the model's responsiveness and prediction range to real-time situations, and generate model performance evaluation results in the following steps: Using the optimized graph structure and the sliding window technology, the data flow changes in the big data analysis platform are monitored, and the data characteristics and evolution trends are tracked to generate real-time monitoring records; Based on the real-time monitoring records, verify the responsiveness of the trend prediction model to real-time data changes, evaluate the prediction range of the trend prediction model under differentiated data conditions and environmental changes, and generate responsiveness analysis results; The performance of the trend prediction model is evaluated through the response capability analysis results, the deviation of the model prediction is calibrated, the error range is calculated and the response time is analyzed, and the model parameters and structure are adjusted according to the evaluation results to generate the model performance evaluation results.

10. The data change trend prediction method of the big data analysis platform according to claim 9 is characterized in that: The formula for the sliding window technique is as follows: ， in, is the new window size, Represents the number of edges in the current graph structure, and K is a constant preset based on performance. is the adjustment coefficient, Represents the maximum degree of the node in the current window, and is the weight coefficient, is the average number of connections of a node.

Images