An information service platform and method for intelligent fusion of big data

Abstract

The present invention relates to the field of data processing technology, specifically to an information service platform and method for intelligent fusion of big data, the platform comprising: a data sequence annotation module obtains sequence data from multiple sources, including the timestamp, position, content value and source information of data points, calculates the content change rate and time interval between adjacent data points, determines the significance of data point changes over time, marks key change points, and generates annotated sequence data. In the present invention, by sorting and ranking key data points of sequence data from multiple sources and analyzing the correlation between data points, it is possible to screen data point combinations with practical significance, optimize the depth of data analysis, enable the prediction model to make more accurate trend predictions based on real associations, and dynamically adjust the prediction model parameters according to changes in real-time data streams to ensure the accuracy of predictions. At the same time, resource allocation is optimized by predicting future data processing needs.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an information service platform and method for intelligent fusion of big data. Background Art

[0002] The field of data processing technology refers to the process and methods of collecting, storing, managing, analyzing and processing large-scale data. It covers the entire data life cycle from data collection, cleaning, storage to data analysis, modeling and visualization. Extracting valuable information from complex, heterogeneous and large amounts of data can make decisions or improve business efficiency in various application scenarios.

[0003] Among them, the information service platform provides users with an integrated and interactive information access interface. The platform usually integrates multiple data sources, provides real-time data processing and information retrieval functions, and enables users to easily obtain, analyze and understand data. Through back-end data processing technology and front-end user interface design, it ensures efficient information delivery and optimal user experience.

[0004] Although existing technologies can integrate multiple data sources and perform information retrieval when processing data, it is difficult to effectively distinguish important change points from ordinary data points when faced with massive amounts of data, which can easily lead to omissions of key data or information overload, affecting the accuracy and timeliness of data retrieval. At the same time, it is difficult to identify potential connections between data points during data analysis, and it is difficult to form high-quality related data combinations. In addition, existing technologies have also failed to achieve dynamic response to real-time data streams, resulting in large deviations between prediction results and actual results when data streams change dramatically. Summary of the invention

[0005] The purpose of the present invention is to solve the shortcomings of the prior art and to propose an information service platform and method for intelligent fusion of big data.

[0006] In order to achieve the above-mentioned purpose, the present invention adopts the following technical scheme: an information service platform for big data intelligent fusion includes:

[0007] The data sequence annotation module obtains sequence data from multiple sources, including the timestamp, location, content value and source information of the data points, calculates the content change rate and time interval between adjacent data points, determines the significance of data point changes over time, marks key data points, and generates annotated sequence data;

[0008] The data association analysis module sorts and ranks the content values ​​and timestamps of the key data points based on the annotated sequence data, calculates the rank difference of the key data point pairs, analyzes the correlation between the key data points according to the Spearman rank correlation coefficient, screens the key data point combinations with correlation, and generates a feature association set;

[0009] The data flow prediction module uses the key data points in the feature association set to extract the data point timestamps, content value levels and correlation scores of the key data points, establish a prediction model, input key data point information, perform trend prediction on the current changes of the key data points, perform deviation analysis by comparing the real-time data flow, adjust the prediction model parameters, and generate data flow prediction results;

[0010] The data processing optimization module analyzes future data processing requirements and resource efficiency of current data processing according to the data flow prediction results, adjusts resource allocation for key data, and obtains optimized data fusion processing results.

[0011] As a further solution of the present invention, the step of obtaining the annotated sequence data is specifically as follows:

[0012] Generate standardized sequence data by cleaning, formatting and standardizing the data based on timestamps, location information, content values ​​and source information obtained from multiple data sources;

[0013] According to the standardized sequence data, the content values ​​and timestamps of adjacent data points are extracted using the formula:

[0014]

[0015] and

[0016]

[0017] Calculate the rate of change of adjacent data points and time interval , generate change rate and time interval analysis results;

[0018] in, Used to measure the degree of change in content value. Indicates The content value of the data point, Indicates The content value of the data point, Represents the time interval between two adjacent data points. No. The timestamp of each data point, No. The timestamp of each data point;

[0019] According to the change rate and time interval analysis results, the set change rate and time interval thresholds are applied to perform significance judgment, key data points exceeding the thresholds are marked, and labeled sequence data are generated.

[0020] As a further solution of the present invention, the step of obtaining the level difference of the key data point pair is specifically as follows:

[0021] Based on the labeled sequence data, extract the content value and time stamp of the key data point, perform a sorting operation, assign a level to each data point, and generate a key data point sequence;

[0022] According to the key data point sequence, the content value level of each pair of data points is extracted, and the level difference between the data points is calculated. The difference is used to quantify the relative change of the content value level of the data points to obtain the level difference data of the data points.

[0023] As a further solution of the present invention, the step of obtaining the combination of key data points with correlation by screening is specifically as follows:

[0024] According to the level difference data of the data points, the formula is adopted:

[0025]

[0026] Calculate Spearman's rank correlation coefficient , get the data point correlation analysis results;

[0027] in, It is The weight value of the data point, Indicates The rank difference of the data points, is the total number of data points, is the noise adjustment term, is the dynamic influence coefficient of content value and timestamp;

[0028] According to the data point correlation analysis results, key data point combinations with correlation are screened, and the data are integrated to generate a feature correlation set.

[0029] As a further solution of the present invention, the step of obtaining trend prediction of changes in data streams is specifically as follows:

[0030] Extracting key data of the feature association set, arranging the timestamps, content value levels and relevance scores of the data points as input parameters of the prediction model, and establishing a trend prediction model;

[0031] According to the trend prediction model, the formula is adopted:

[0032]

[0033] Analyze the external factors affecting the current moment and calculate the time point The data flow prediction value under , get the prediction result at the current moment;

[0034] in, Indicates the basic level value of the data flow, is the number of lags, It is The coefficient of the lag period, It is lagged observations, indicating that forward The data point value at time, is the coefficient of the external influencing factor, It is an external influencing factor. is the error term, is the identifier of the current time point, is the identifier of the lag period, is the number of lags.

[0035] As a further solution of the present invention, the step of obtaining the adjustment prediction model parameters is specifically as follows:

[0036] Comparing the prediction result at the current moment with the current data stream value, calculating the deviation ratio and comparing it with the conventional standard to obtain the deviation analysis result;

[0037] Based on the deviation analysis results, the lag coefficients in the model are re-evaluated, the impact of external environmental changes on the forecast is considered, the error estimate is updated, and the data flow forecast results are generated.

[0038] As a further solution of the present invention, the step of adjusting the acquisition of resource configuration for key data is specifically:

[0039] According to the data flow prediction results, the future data processing requirements are analyzed using the formula:

[0040]

[0041] Calculate the processing power required by the platform , get the future predicted data flow information;

[0042] in, is the processing power required per TB of data, is the estimated processing time. is the number of parallel processing nodes, It is the basic processing resource;

[0043] According to the current data processing performance of the platform, by real-time monitoring of resource usage, the formula is used:

[0044]

[0045] Calculate the amount of data consumed per TB of data processed , obtain the resource efficiency information of current data processing;

[0046] in, is the total amount consumed by the current task processing, is the amount of data currently being processed;

[0047] According to the predicted future data flow information and the resource efficiency information of current data processing, resource allocation is dynamically adjusted, the processing flow of key data is optimized, and an optimized data fusion processing result is obtained.

[0048] A method for providing an information service based on intelligent fusion of big data, wherein the method is based on the information service platform for intelligent fusion of big data and comprises the following steps:

[0049] S1: Based on multi-source sequence data, collect data point information, calculate the changes between data points, and generate annotated sequence data;

[0050] S2: using the annotated sequence data, sorting and ranking the data points, analyzing the correlation between the data points, and generating a feature correlation set;

[0051] S3: Analyze the data pattern and identify the trend based on the feature association set, and establish a data flow prediction model;

[0052] S4: Use the data flow prediction model to predict the trend of current data, adjust the model, and obtain the data flow prediction result;

[0053] S5: Based on the data flow prediction results, analyze future data processing requirements and resource efficiency, adjust resource allocation, and generate optimized data fusion processing results.

[0054] Compared with the prior art, the advantages and positive effects of the present invention are:

[0055] In the present invention, by sorting and ranking key data points of sequence data from multiple sources and analyzing the correlation between data points, it is possible to screen data point combinations with practical significance, optimize the depth of data analysis, enable the prediction model to make more accurate trend predictions based on real correlations, and dynamically adjust the prediction model parameters according to changes in real-time data streams to ensure the accuracy of predictions. At the same time, by predicting future data processing needs, resource allocation is optimized, avoiding waste of resources and improving the efficiency and accuracy of data processing. The entire process combines multiple means such as sensitive capture of data changes, deep mining of correlation analysis, and optimal resource allocation, effectively improving the accuracy of data processing and resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 is a flow chart of the platform of the present invention;

[0057] Figure 2 A flow chart of the steps for obtaining the annotated sequence data of the present invention;

[0058] Figure 3 A flow chart of the steps for obtaining the level difference of key data point pairs calculated by the present invention;

[0059] Figure 4 A flow chart of the steps for obtaining a combination of key data points having relevance for screening in the present invention;

[0060] Figure 5 A flow chart of the steps for obtaining trend prediction of data stream changes according to the present invention;

[0061] Figure 6 A flow chart of the acquisition steps for adjusting the prediction model parameters of the present invention;

[0062] Figure 7 A flow chart of the acquisition steps for adjusting resource allocation for key data in the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0064] In the description of the present invention, it should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, in the description of the present invention, "multiple" means two or more, unless otherwise clearly and specifically defined.

[0065] See also Figure 1 The present invention provides a technical solution: an information service platform for big data intelligent fusion includes:

[0066] The data sequence annotation module obtains sequence data from multiple sources, including the timestamp, location, content value and source information of the data points, calculates the content change rate and time interval between adjacent data points, determines the significance of data point changes over time, marks key data points, and generates annotated sequence data;

[0067] The data association analysis module sorts and ranks the content values ​​and timestamps of key data points based on the annotated sequence data, calculates the rank difference of key data point pairs, analyzes the correlation between key data points based on the Spearman rank correlation coefficient, screens the key data point combinations with correlation, and generates a feature association set;

[0068] The data stream prediction module uses the key data points in the feature association set to extract the data point timestamp, content value level and correlation score of the key data points, establish a prediction model, input the key data point information, perform trend prediction on the current changes of the key data points, perform deviation analysis by comparing the real-time data stream, adjust the prediction model parameters, and generate data stream prediction results;

[0069] The data processing optimization module analyzes future data processing requirements and resource efficiency of current data processing based on data flow prediction results, adjusts resource allocation for key data, and obtains optimized data fusion processing results;

[0070] The annotated sequence data includes identifiers of key data points, the significant change rate of each point, and the position index of these change points in the entire data sequence. The feature association set includes a set of highly correlated data points, the internal data point ranking of each set, and the association strength evaluation value of these data point combinations. The data flow prediction results include the predicted data flow trend line, the predicted data points at key time nodes, and the prediction accuracy score. The optimized data fusion processing results include optimized storage space allocation and adjusted data processing resource configuration.

[0071] See also Figure 2 , the specific steps for obtaining the annotated sequence data are:

[0072] Generate standardized sequence data by cleaning, formatting and standardizing the data based on timestamps, location information, content values ​​and source information obtained from multiple data sources;

[0073] When obtaining sequence data from multiple sources, you first need to collect raw data from different databases, log files or network interfaces. The raw data usually includes timestamps, location information, content values ​​and source information of the data. The timestamp can be extracted from the recorded log information, usually in the format of a standard time string or an integer value of the timestamp. The location information can be marked according to the system where the data is generated or the logical location of the record, such as the location identifier of a specific module or node. The content value depends on the type of data content, which can be transaction data, user behavior records or system log information. The source information usually comes from the specific identifier of the data source, such as the database table name, interface name, etc. All of these data need to be cleaned and standardized after collection to ensure that data from different sources are compatible.

[0074] Based on the standardized sequence data, the content values ​​and timestamps of adjacent data points are extracted using the formula:

[0075]

[0076] and

[0077]

[0078] Calculate the rate of change of adjacent data points and time interval , generate change rate and time interval analysis results;

[0079] in, Used to measure the degree of change in content value. Indicates The content value of a data point, usually obtained from a database or log, Indicates The content value of each data point is also obtained by reading the data. Represents the time interval between two adjacent data points. No. The timestamp of each data point, usually extracted from the time field, No. The timestamp of each data point is obtained in the same way as same.

[0080] Taking a specific data point as an example, the content value of the first data point is , timestamp , the content value of the second data point is , timestamp , we can calculate:

[0081] Rate of change:

[0082] Time interval:

[0083] According to the change rate and time interval analysis results, the set change rate and time interval thresholds are applied to make significance judgments, key data points that exceed the thresholds are marked, and annotated sequence data are generated;

[0084] According to the rate of change and time interval , to determine the significance of the change of data points over time. The significance can be measured by setting a certain threshold. If the rate of change is , then it is judged that the data point has changed significantly, and the calculated change rate is 0.333, which is greater than the threshold value of 0.3. Therefore, it is judged that the content of the data point has changed significantly. Next, these significantly changed data points are marked. The marking method can be to mark these key data points with specific identifiers. In addition, it is also necessary to consider whether the time interval is too large or too small. If the threshold of the time interval is 2 seconds, the calculated time interval is 4 seconds, which is in line with the normal time interval range. Therefore, the change point is recorded and a mark is generated in the sequence data.

[0085] See also Figure 3 , the specific steps for calculating the level difference of key data point pairs are:

[0086] Based on the labeled sequence data, the content value and timestamp of the key data points are extracted, and sorting operations are performed to assign a level to each data point to generate a sequence of key data points;

[0087] based on and To analyze the results, first process the labeled sequence data and extract the content values ​​and timestamps of all key data points. The content value represents the actual data record of each data point, and the timestamp identifies the time information of the data point. Next, sort the content values ​​and timestamps of the key data points respectively. The content values ​​are arranged in ascending order, and the timestamps are arranged in chronological order. After the sorting is completed, a level is assigned to each data point. Smaller content values ​​correspond to lower levels, and earlier timestamps also correspond to lower levels. Each data point will have two levels: one content value level and another timestamp level.

[0088] According to the key data point sequence, the content value level of each pair of data points is extracted, and the level difference between the data points is calculated. The difference is used to quantify the relative change of the content value level of the data points to obtain the level difference data of the data points;

[0089] To calculate the level difference between data points, the formula is used:

[0090]

[0091] in, Indicates The level difference of the data points measures the difference in the content value level between the two data points. Indicates The content value level of each data point, Indicates Another data point to compare The content value level, and Indicates the index number of the data point, which is used to distinguish different pairs of data points.

[0092] First, extract the content value level of each pair of data points from the key data point sequence and .

[0093] For example, there are two data points: data point 1 has a content value level of 5, and data point 2 has a content value level of 3.

[0094] Calculate the grade difference:

[0095]

[0096] The results show that the difference between the two data points in terms of content value level is 2.

[0097] See also Figure 4 , the specific steps for obtaining the combination of key data points with correlation are:

[0098] According to the level difference data of the data points, the formula is used:

[0099]

[0100] Calculate Spearman's rank correlation coefficient , get the data point correlation analysis results;

[0101] in, Indicates the nonlinear correlation between variables. The value range is -1 to 1, indicating positive correlation, negative correlation, and no correlation between variables. is a constant representing the standardized coefficient in the Spearman rank correlation coefficient formula, which is used to adjust the influence of the sum of squared differences to a suitable range by increasing the number of data points. The relationship with the sum of squared differences is adjusted so that the final correlation coefficient falls within the range of -1 to 1 and is not affected by the number of data points. It is The weight value of the data point reflects the importance of the data point to the calculation result. It mainly considers the reliability of the source of the data point or the size of its influence, and is comprehensively evaluated by indicators such as data point collection quality and signal-to-noise ratio. It is The square of the level difference of the data points is mainly used to amplify the level difference between the data points. is the total number of data points, indicating the number of all data points involved in the calculation, obtained by statistically analyzing the current sequence data. It is a noise adjustment item used to control the impact of noise on the overall calculation. It is obtained through the analysis of past data noise and reflects the potential impact of noise level on the calculation results during data acquisition. It is the dynamic influence coefficient of content value and timestamp, which indicates the degree of dynamic change of data points in the time series. It is estimated by analyzing the change rate of data stream and content value fluctuation.

[0102] For each data point pair, the formula Calculating weights , determines the weight of each data point pair in the correlation calculation.

[0103] in, is the weight adjustment coefficient, used to adjust the weight Level with data point content value The relationship between The content value has a greater impact on the weight, which is adjusted according to the processing priority of different types of data or the needs of the application scenario.

[0104] Calculate the squared difference in rank for each pair of data points , and by adjusting the weights, we get the weighted sum of squares .

[0105] Substitute the weighted sum of squares and the adjustment coefficient into the improved formula to calculate the Spearman correlation coefficient .

[0106] For example, there are two data points:

[0107] The content value level of data point 1 is 4, and the content value level of data point 2 is 3. The difference between the two levels is , set the weight adjustment coefficient , noise adjustment term , dynamic influence coefficient .

[0108] Calculating weights :

[0109]

[0110]

[0111] Compute the weighted sum of squares:

[0112]

[0113] Substitute into the formula to calculate the correlation coefficient:

[0114] 8

[0115] According to the results of data point correlation analysis, select key data point combinations with correlation, integrate the data, and generate feature correlation sets;

[0116] Based on the calculation results of the Spearman rank correlation coefficient, after correlation analysis of all data points, key data point combinations with correlation can be screened out. The specific operation is to filter the correlation coefficient of each pair of data points and set a certain threshold. For example, the data point pairs with an absolute value of the correlation coefficient greater than 0.5 are regarded as key data point combinations with correlation. Then all eligible data point pairs are screened out, and these data points are combined into a new set. The time series and content values ​​of these data points are further clustered to ensure that there is a stronger correlation between them. In the process of data point screening, the features of the data points can be further screened according to domain knowledge, and finally a feature association set is formed.

[0117] See also Figure 5 , the specific steps for obtaining trend prediction of data flow changes are:

[0118] Extract key data of feature association sets, organize timestamps, content value levels and relevance scores of data points as input parameters of the prediction model, and establish a trend prediction model;

[0119] According to the key data of the feature association set, including the timestamp, content value level and relevance score of the data point, these key data are extracted as the input parameters of the prediction model. The timestamp is used to identify the time series information of the data point, the content value level reflects the relative importance of the data point, and the relevance score is used to measure the degree of association between the data point and other data points. The time series analysis algorithm is selected to establish the prediction model. In the process of establishing the prediction model, the data is first preprocessed, including standardizing the timestamp to ensure the accuracy of the time series information, and adjusting the content value level and relevance score to reflect the importance and mutual correlation of each data point. Select an appropriate time series analysis algorithm, such as ARIMA or LSTM, and adjust the model parameters according to the characteristics of the data, such as seasonality and trend. By training and optimizing these parameters on the data collected in the past, the model is established and its prediction accuracy is verified on an independent test set.

[0120] According to the trend forecasting model, the formula is adopted:

[0121]

[0122] Analyze the external factors affecting the current moment and calculate the time point The data flow prediction value under , get the prediction result at the current moment;

[0123] in, Represents the prediction of data flow changes at the current moment, which is the output of the prediction model. It is a constant term, which indicates the basic level value of the data flow. It is estimated by regression analysis of past data and reflects the basic value of the data flow without the influence of other variables. is the weighted sum of multiple lags, is the number of lag periods, indicating how many moments of data are taken into consideration, which is determined by the settings of the forecasting model. It is The coefficients of the lag period reflect the influence of historical data on the current forecast value. These coefficients are obtained through model training optimization. The optimization method is usually the least squares method or gradient descent algorithm, with the goal of minimizing the forecast error. It is The actual observed value of the lag period is expressed as forward The actual data point value at the moment is collected directly from the data stream, reflecting the data changes at the historical moment. It is the coefficient of external influencing factors, reflecting the impact of external variables on the prediction results. It is determined by analyzing the degree of influence of external variables on the data flow and is often calculated through correlation analysis and regression analysis. It is an external influencing factor, representing the impact of external variables on the current data flow. It is obtained from external data sources collected by the big data platform, such as economic indicators, weather data, etc., and directly input into the prediction model. is the error term, which represents the random error that cannot be explained by the model. It is estimated through residual analysis and model fitting process, and reflects the unpredictable fluctuation part in the data stream. It is the identifier of the current time point, indicating the state of the data stream at the current moment, and is used to mark the time information of the prediction moment. It is the identifier of the lag period, used in the summation process, indicating the index of each lagged data point, and its value changes with the summation range. It is the number of lag periods, which is set to the number of historical data points that affect the current forecast value. The specific value is determined by adjusting the model parameters.

[0124] Extract the previous observation from the data stream. , which is used to calculate the contribution of the lag period.

[0125] Through historical data regression analysis, the constant term and the coefficients of each lag period are calculated. , and calculate the external influence coefficient , and get the current external variable value .

[0126] Estimate the error term at the current moment through model residual analysis .

[0127] Substitute all parameters into the formula to calculate the predicted value:

[0128]

[0129]

[0130] The results show that the predicted value at the current moment is 15.6, which indicates the expected change of the data flow at the current moment under the influence of given parameter settings and historical observations. The prediction results can provide a basis for further monitoring and analysis of the data flow.

[0131] See also Figure 6 , the specific steps for adjusting the prediction model parameters are:

[0132] Compare the prediction result at the current moment with the current data stream value, calculate the deviation ratio and compare it with the conventional standard to obtain the deviation analysis result;

[0133] For calculating the deviation ratio, the formula is used:

[0134]

[0135] in, is the actual data flow value.

[0136] For example, if the actual data stream value , then the deviation is:

[0137]

[0138] In this model, the benchmark value judgment standard is to compare the deviation with the percentage of the actual data flow value. Generally, if the deviation is within 5% of the actual data value (i.e. | / |≤0.05), the model can be considered to have good prediction accuracy. The ratio of deviation to actual value is: , indicating that the deviation is about 8.2% of the actual value, which exceeds the normal range of 5%, indicating that the model needs further adjustment. This deviation analysis helps identify parameters in the model that may need adjustment, such as the lag coefficient Or external variable influence coefficient , and analyze whether the deviation is caused by the fluctuation of the data stream itself or the change of external variables. The results show that the deviation is 1.4, accounting for 8.2% of the actual value, indicating that the model has a certain error. By refining the analysis of the source of deviation, the specific factors affecting the accuracy of the prediction can be clarified, providing direction for the optimization of the model, thereby improving the prediction accuracy.

[0139] Based on the deviation analysis results, the lag coefficients in the model are re-evaluated, the impact of external environmental changes on the forecast is considered, the error estimate is updated, and the data flow forecast results are generated;

[0140] According to the deviation results, the parameters of the forecast model are adjusted, and statistical regression analysis is performed using the previously collected data to calculate the lag coefficient of the model. The specific operation includes re-fitting the model using the least squares method to determine the new lag coefficient, and adjusting the external influence coefficient according to the change of external environmental variables. , including the update of new external data into the analysis. In addition, the error term The estimate of also needs to be updated based on the latest data to more accurately reflect the model's prediction error. Through the iterative process, the model parameters that can minimize the prediction error are finally determined. After applying these new parameters, the model will generate a new round of data flow prediction results, which will be used for the actual data comparison of the next cycle to complete the optimization and accuracy improvement of the model.

[0141] See also Figure 7 , the specific steps for adjusting the resource allocation of key data are:

[0142] According to the data flow prediction results, analyze the future data processing needs and use the formula:

[0143]

[0144] Calculate the processing power required by the platform , get the future predicted data flow information;

[0145] in, It is the processing capacity required for each TB of data, which is obtained based on historical data processing performance statistics and recent processing efficiency analysis. is the estimated processing time, based on the statistical data of the average processing time per TB of data. is the number of parallel processing nodes, based on the current configuration of the platform and the recent resource allocation plan, It is the basic processing resource, which represents the basic processing capability of the platform and is set based on the platform's default configuration and past operation and maintenance data.

[0146] Set parameter data stream prediction value TB, the processing requirement for each TB of data is , estimated processing time is , the number of parallel processing nodes , basic processing resources ,

[0147] Substitute into the formula to calculate:

[0148]

[0149]

[0150] The results show that the future data processing demand is 35.72 processing capacity values, which means that in order to smoothly process the predicted data flow of 15.6TB, the system needs at least 35.72 units of computing resources. The processing capacity value is the amount of resources that the platform needs to configure to meet the future data flow, specifically the comprehensive demand for computing resources such as memory, storage space, and processing speed. If the current capacity of the platform is 30 units, the platform needs to enhance its capacity to process the predicted data volume to ensure that there will be no processing delays or system performance degradation due to insufficient resources.

[0151] According to the current data processing performance of the platform, by real-time monitoring of resource usage, the formula is used:

[0152]

[0153] Calculate the amount of data consumed per TB of data processed , obtain the resource efficiency information of current data processing;

[0154] in, is the total amount consumed by the current task processing, obtained based on the data provided by the real-time monitoring system. is the current amount of data processed (TB), obtained through the logs recorded by the data management system.

[0155] Setting parameters , TB.

[0156] Computational resource efficiency:

[0157] The results show that the current resource efficiency is 1.25, which indicates the processing power required for the platform to process 1TB of data. This efficiency indicator can be used to determine whether existing resources are sufficient or whether additional resources need to be invested in to maintain or increase data processing capabilities, thereby ensuring that the system can continue to operate stably and effectively cope with the expected increase in data. This assessment helps to develop more accurate resource allocation strategies and optimize system performance to adapt to future changes in demand.

[0158] According to the future predicted data flow information and the resource efficiency information of current data processing, the resource allocation is dynamically adjusted, the processing flow of key data is optimized, and the optimized data fusion processing results are obtained.

[0159] The platform will identify the key data parts that need to be processed first based on the importance of data processing and the impact on the system. Then, for these key data, the platform will reallocate resources, such as focusing more computing power on the processing of key data and reducing resource allocation for low-priority data. In this way, the platform can ensure that key data is processed quickly and reduce processing delays while optimizing the overall data processing process. In this process, the platform monitors the processing efficiency of key data in real time and dynamically adjusts resource allocation based on the data feedback during the processing process to ensure that the fusion processing of key data achieves the best effect.

[0160] A method for providing an information service based on intelligent fusion of big data, the method for providing an information service based on intelligent fusion of big data is implemented based on the above-mentioned information service platform for intelligent fusion of big data, and comprises the following steps:

[0161] S1: Based on multi-source sequence data, collect data point information, calculate the changes between data points, and generate annotated sequence data;

[0162] S2: Use the labeled sequence data to sort and rank the data points, analyze the correlation between the data points, and generate a feature association set;

[0163] S3: Analyze data patterns and identify trends based on feature association sets to build data stream prediction models;

[0164] S4: Use the data stream prediction model to predict the trend of the current data, adjust the model, and obtain the data stream prediction result;

[0165] S5: Based on the data flow prediction results, analyze future data processing requirements and resource efficiency, adjust resource allocation, and generate optimized data fusion processing results.

[0166] The above are only preferred embodiments of the present invention and are not intended to limit the present invention in other forms. Any technician familiar with the profession may use the technical contents disclosed above to change or modify them into equivalent embodiments with equivalent changes and apply them to other fields. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. An information service platform for intelligent fusion of big data, characterized in that: The platform includes: The data sequence annotation module obtains sequence data from multiple sources, including the timestamp, location, content value and source information of the data points, calculates the content change rate and time interval between adjacent data points, determines the significance of data point changes over time, marks key data points, and generates annotated sequence data; The steps for obtaining the labeled sequence data are specifically as follows: Generate standardized sequence data by cleaning, formatting and standardizing the data based on timestamps, location information, content values ​​and source information obtained from multiple data sources; According to the standardized sequence data, the content values ​​and timestamps of adjacent data points are extracted using the formula: and Calculate the rate of change of adjacent data points and time interval , generate change rate and time interval analysis results; in, Used to measure the degree of change in content value. Indicates The content value of the data point, Indicates The content value of the data point, Represents the time interval between two adjacent data points. No. The timestamp of each data point, No. The timestamp of each data point; According to the change rate and time interval analysis results, a significance judgment is performed using the set change rate and time interval thresholds, key data points exceeding the thresholds are marked, and marked sequence data are generated; The data association analysis module sorts and ranks the content values ​​and timestamps of the key data points based on the annotated sequence data, calculates the rank difference of the key data point pairs, analyzes the correlation between the key data points according to the Spearman rank correlation coefficient, screens the key data point combinations with correlation, and generates a feature association set; The steps for obtaining the level difference of the key data point pairs are specifically as follows: Based on the labeled sequence data, extract the content value and time stamp of the key data point, perform a sorting operation, assign a level to each data point, and generate a key data point sequence; According to the key data point sequence, extract the content value level of each pair of data points, calculate the level difference between the data points, use the difference to quantify the relative change of the content value level of the data points, and obtain the level difference data of the data points; To calculate the level difference between data points, the formula is used: in, Indicates The level difference of the data points measures the difference in the content value level between the two data points. Indicates The content value level of each data point, Indicates Another data point to compare The content value level, and Represents the index number of the data point, which is used to distinguish different pairs of data points; The data flow prediction module uses the key data points in the feature association set to extract key data point information, including data point timestamps, content value levels and relevance scores, establish a prediction model, and perform trend prediction on the current changes of key data points by inputting key data point information, perform deviation analysis by comparing the real-time data flow, adjust the prediction model parameters, and generate data flow prediction results; The data processing optimization module analyzes future data processing requirements and resource efficiency of current data processing according to the data flow prediction results, adjusts resource allocation for key data, and obtains optimized data fusion processing results.

2. The information service platform for intelligent fusion of big data according to claim 1 is characterized in that: The steps of obtaining the combination of key data points with correlation by screening are specifically as follows: According to the level difference data of the data points, the formula is adopted: Calculate Spearman's rank correlation coefficient , get the data point correlation analysis results; in, It is The weight value of the data point, Indicates The rank difference of the data points, is the total number of data points, is the noise adjustment term, is the dynamic influence coefficient of content value and timestamp; According to the data point correlation analysis results, key data point combinations with correlation are screened, and the data are integrated to generate a feature correlation set.

3. The information service platform for intelligent fusion of big data according to claim 2 is characterized in that: The steps for obtaining trend prediction of current changes of key data points are specifically as follows: Extracting key data of the feature association set, arranging the timestamps, content value levels and relevance scores of the data points as input parameters of the prediction model, and establishing a trend prediction model; According to the trend prediction model, the formula is adopted: Analyze the external factors affecting the current moment and calculate the time point The data flow prediction value under , get the prediction result at the current moment; in, Indicates the basic level value of the data flow, is the number of lags, It is The coefficient of the lag period, It is lagged observations, indicating that forward The data point value at the time, is the coefficient of the external influencing factor, It is an external influencing factor. is the error term, is the identifier of the current time point, is the identifier of the lag period, is the number of lags.

4. The information service platform for intelligent fusion of big data according to claim 3 is characterized in that: The steps for obtaining the parameters of the adjusted prediction model are specifically as follows: Comparing the prediction result at the current moment with the current data stream value, calculating the deviation ratio and comparing it with the conventional standard to obtain the deviation analysis result; Based on the deviation analysis results, the lag coefficients in the model are re-evaluated, the impact of external environmental changes on the prediction is considered, the error estimate is updated, and the data flow prediction results are generated.

5. The information service platform for intelligent fusion of big data according to claim 4 is characterized in that: The acquisition steps of adjusting resource allocation for key data are specifically as follows: According to the data flow prediction results, the future data processing requirements are analyzed using the formula: Calculate the processing power required by the platform , get the future predicted data flow information; in, is the processing power required per TB of data, is the estimated processing time. is the number of parallel processing nodes, It is the basic processing resource; According to the current data processing performance of the platform, by real-time monitoring of resource usage, the formula is used: Calculate the amount of data consumed per TB of data processed , obtain the resource efficiency information of current data processing; in, is the total amount consumed by the current task processing, is the amount of data currently being processed; According to the predicted future data flow information and the resource efficiency information of current data processing, resource allocation is dynamically adjusted, the processing flow of key data is optimized, and an optimized data fusion processing result is obtained.

6. An information service method for intelligent fusion of big data, characterized in that: The information service platform for intelligent fusion of big data according to any one of claims 1 to 5 is implemented, comprising the following steps: Based on multi-source sequence data, collect data point information, calculate the changes between data points, and generate annotated sequence data; Using the labeled sequence data, sorting and ranking the data points, analyzing the correlation between the data points, and generating a feature correlation set; Based on the feature association set, data patterns are analyzed and trends are identified to establish a data flow prediction model; Use the data flow prediction model to predict the trend of current data, adjust the model, and obtain data flow prediction results; Based on the data flow prediction results, future data processing requirements and resource efficiency are analyzed, resource allocation is adjusted, and optimized data fusion processing results are generated.

Images