Modeling method and system for multi-source heterogeneous data model

Abstract

The invention discloses a multi-source heterogeneous data model modeling method and system, and belongs to the field of data modeling. The method solves the problem of modeling mainly aiming at a single data source or isomorphic data in the prior art, realizes global real-time information acquisition by accessing multiple types of data sources, can dynamically adjust an acquisition strategy according to data source characteristics and business requirements, ensures high efficiency and adaptability of data acquisition, and is suitable for large-scale popularization and application. Through data extraction and conversion, corresponding extraction strategies can be formulated for different types of data, formats are effectively cleaned and unified, the data are fused, a high-quality data basis is provided for modeling, during model construction, a proper algorithm is screened according to data characteristics, a model reflecting data relations and characteristics is accurately constructed, and then through model verification and optimization, a high-quality data basis is provided for modeling. The performance and generalization ability of the model can be improved, so that the high-quality, stable and reliable model is constructed.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of data modeling, and particularly relates to a multi-source heterogeneous data modeling method and system. BACKGROUND

[0002] In the current digital era, data has become an important asset in many fields such as enterprises, industrial production and various scientific researches, and the data sources are extremely wide and diverse, including real-time monitoring data generated by various sensors, SCADA system data in industrial control systems, structured database data and unstructured file system data, and the like. The data usually has heterogeneity, that is, there are great differences in data formats, structures, semantics and the like. Traditional data modeling methods mainly model single data sources or homogeneous data, and cannot effectively process multi-source heterogeneous data. Therefore, the existing requirements cannot be met, and thus the application provides a multi-source heterogeneous data modeling method and system. SUMMARY

[0003] The application aims to provide a multi-source heterogeneous data modeling method and system, which collects real-time information in the whole domain, dynamically adjusts the collection strategy according to the data source characteristics and business requirements, guarantees the efficiency and adaptability of data collection, provides high-quality data basis for modeling through data extraction and conversion, selects appropriate algorithms according to data characteristics during model construction, accurately constructs a model reflecting data relationships and characteristics, and then performs model verification and optimization to improve the model performance and generalization ability, thereby solving the problems in the background technology.

[0004] To achieve the above-mentioned purpose, the application provides the following technical scheme: a multi-source heterogeneous data modeling method, which comprises the following steps:

[0005] Collecting real-time information in the whole domain:

[0006] Supporting access to various types of data sources, including but not limited to sensor data, SCADA system data, database data and file system data, and realizing comprehensive collection of multi-source heterogeneous data;

[0007] Among them, the collection frequency is adjusted according to the real-time requirements and priority of the data source, and the collection frequency corresponding to the target data source is adjusted by using the priority proportion and the calibration data volume parameters corresponding to the target data source;

[0008] Data extraction and conversion:

[0009] The collected multi-source heterogeneous data is extracted and converted, and data in different formats and structures is uniformly converted into an intermediate format suitable for modeling, and the conversion process includes data cleaning, format specification unification and data fusion operation;

[0010] Model construction:

[0011] Based on the converted multi-source heterogeneous data, a preset modeling algorithm and rule are used to construct a data model, and the constructed data model is used to reflect the relationship and characteristics between the multi-source heterogeneous data;

[0012] Model verification and optimization:

[0013] The constructed data model is verified and evaluated, and the data model is optimized and adjusted according to the verification result, including the optimization of model parameters and the improvement of model structure.

[0014] Further, the global real-time information collection includes:

[0015] Supporting access to multiple different types of data sources including sensor data, SCADA system data, database data and file system data;

[0016] Through a unified access interface, seamless connection to different data sources is realized, and multi-source heterogeneous data is comprehensively collected;

[0017] At the same time, according to the characteristics and business requirements of the data source, data collection strategy is adopted for data collection, and according to the network condition and storage resource, data collection strategy is dynamically adjusted, including adjusting the collection frequency and data volume.

[0018] Further, according to the characteristics and business requirements of the data source, data collection strategy is adopted, and according to the network condition and storage resource, data collection strategy is dynamically adjusted, including:

[0019] According to the business scenario, the real-time requirements of each data source are determined, and according to the business target, the priority of each data source is determined;

[0020] According to the determined real-time requirements and priority of the data source, data collection strategy is selected for each data source, wherein the data collection strategy includes determining the collection frequency, data volume, and allocating network bandwidth and storage resources according to the priority;

[0021] Real-time monitoring of network bandwidth usage, identification of network congestion and bandwidth bottleneck, and regular testing of network delay and stability, recording of network interruption and data packet loss, for evaluating the reliability of the network;

[0022] At the same time, the usage of the storage system is regularly checked, including the remaining space and the health status of the storage device;

[0023] According to the real-time status of the network bandwidth and storage resources, the collection frequency and data volume of each data source are dynamically adjusted.

[0024] Further, the collection frequency is adjusted according to the real-time requirements and priority of the data source, including:

[0025] a network bandwidth occupation ratio of the data source is collected in real time;

[0026] when the network bandwidth occupation ratio of the data source exceeds a preset ratio reference value, the data source is taken as a target data source;

[0027] a current single data collection data amount of the target data source and a priority weight coefficient corresponding to the target data source are called, and the ratio reference value ranges from 18% to 23%;

[0028] the current single data collection data amount of the target data source after normalization processing is taken as a calibration data amount parameter;

[0029] data sources with network bandwidth occupation ratios not exceeding the preset ratio reference value are called from all data sources as reference data sources;

[0030] the priority weight coefficients of the reference data sources are called, and priority weight coefficient sum values corresponding to all reference data sources are obtained according to the priority weight coefficients of the reference data sources;

[0031] a priority proportion is obtained by ratio processing of the priority weight coefficient corresponding to the target data source and the priority weight coefficient sum value;

[0032] the acquisition frequency corresponding to the target data source is adjusted by using the priority proportion corresponding to the target data source and the calibration data amount parameter.

[0033] Further, adjusting the acquisition frequency corresponding to the target data source by using the priority proportion corresponding to the target data source and the calibration data amount parameter comprises:

[0034] the priority proportion corresponding to the target data source and the calibration data amount parameter are called;

[0035] the network bandwidth occupation ratio of the target data source is called;

[0036] an actual data amount acquisition intensity corresponding to the target data source is obtained by using the calibration data amount parameter and the network bandwidth occupation ratio corresponding to the target data source;

[0037] an actual demand priority intensity is obtained by using the network bandwidth occupation ratio and the priority proportion corresponding to the target data source;

[0038] the acquisition frequency of the target data source is adjusted by using the actual data amount acquisition intensity and the actual demand priority intensity corresponding to the target data source.

[0039] Further, the data extraction and conversion comprises:

[0040] Data extraction:

[0041] Develop appropriate extraction strategies according to data source characteristics, including:

[0042] For sensor data with high real-time requirements, use polling mechanism to periodically extract data at certain time intervals;

[0043] For large amounts of database data, use batch extraction method to extract data changes at regular intervals or according to trigger conditions;

[0044] For file system data, set up file listening mechanism to trigger data extraction when file is updated;

[0045] Data conversion:

[0046] After data extraction, remove noise and outliers in multi-source heterogeneous data, delete records with missing values or fill them with interpolation method;

[0047] Then, convert multi-source heterogeneous data of different formats into unified data types required for modeling, and associate related data from multiple data sources to form a complete data record, integrating information from different data sources;

[0048] At the same time, for time series data, align and integrate according to timestamp, interpolate or sample multi-source heterogeneous data of different frequencies to have the same time interval.

[0049] Further, the model construction includes:

[0050] Statistical analysis of converted multi-source heterogeneous data to understand the concentration trend, dispersion degree and correlation of multi-source heterogeneous data;

[0051] Select suitable modeling algorithms, including for classification problems, choose logistic regression, decision tree and random forest, and for regression problems, choose linear regression;

[0052] Construct data model according to selected modeling algorithm, and initialize model structure and set initial parameters of model;

[0053] Collect a large amount of existing historical data as training data to train the constructed data model, and monitor the training progress and performance indicators of the data model, including loss value and accuracy, during the training process.

[0054] Further, the model verification and optimization includes:

[0055] Collect a large amount of existing historical data as validation set to verify the data model, which is used to evaluate the stability and generalization ability of the data model;

[0056] And according to the model type, select the appropriate evaluation index, wherein, for classification model, the evaluation index includes accuracy, recall and F1 value, for regression model, the evaluation index includes mean square error, root mean square error and mean absolute error;

[0057] Compare the data models trained by different modeling algorithms, and select the optimal data model according to the performance index evaluation results;

[0058] Through grid search, random search or Bayesian optimization method, the hyperparameters of the data model are optimized, and the structure of the data model is optimized according to the performance and complexity of the data model;

[0059] After the above operations, the data model is converted into a deployable format and encapsulated, and at the same time, the environment required for the deployment of the data model is prepared, and the encapsulated data model is deployed to the actual production environment, and the prediction result is generated by receiving the input data;

[0060] Continuously monitor the performance of the data model in actual application, including but not limited to prediction accuracy and response time;

[0061] And according to the performance, regularly update and maintain the data model, including retraining the model or adjusting the model parameters.

[0062] A multi-source heterogeneous data modeling system for implementing a multi-source heterogeneous data modeling method, comprising:

[0063] The data acquisition module is configured to configure a unified access interface to collect global real-time information from various data sources including sensor data, SCADA system data, database data and file system data, and to obtain multi-source heterogeneous data;

[0064] Among them, according to the characteristics and business requirements of the data source, data acquisition strategy is adopted for data acquisition, and the data acquisition strategy is dynamically adjusted according to the network status and storage resources;

[0065] The data processing module is configured to extract and convert the collected multi-source heterogeneous data, and to perform cleaning, format unification and data fusion operations during the conversion process;

[0066] The model construction module is configured to construct a data model based on the multi-source heterogeneous data processed by the data processing module, using a preset modeling algorithm and rule, and collect a large amount of historical data as training data to train the model;

[0067] The verification and optimization module is configured to verify and evaluate the constructed data model using the verification set, and to optimize and adjust the data model according to the verification results, including optimization of model parameters and improvement of model structure;

[0068] A model deployment module is configured to deploy the trained and verified and optimized data model into an actual production environment, and continuously monitor the performance of the data model in actual application, and regularly update and maintain the data model according to the performance.

[0069] Further, the data collection module comprises:

[0070] A requirement analysis module is configured to analyze and determine the real-time requirements of each data source according to the business scenario, and evaluate the importance of each data source and determine the priority of each data source according to the business target;

[0071] A strategy making module is configured to select a suitable data collection strategy for each data source in combination with the real-time requirements and priority of the data source;

[0072] A network monitoring module is configured to monitor the network bandwidth usage in real time, identify network congestion and bandwidth bottlenecks in time, and regularly test network delay and stability, record network interruption and data packet loss, and if data packet loss or high delay occurs continuously for multiple times, it is judged that the network stability is not good;

[0073] A storage monitoring module is configured to regularly check the usage of the storage system, including the remaining space and the health status of the storage device, for early detection of storage space shortage or device failure hazards;

[0074] A strategy adjustment module is configured to make dynamic adjustment decisions for the data collection strategy according to the real-time monitoring status of the network bandwidth and storage resources, and to adjust the data collection strategy according to the decisions.

[0075] Compared with the prior art, the beneficial effects of the present application are:

[0076] In the aspect of data collection, the present application can access multiple types of data sources to realize global real-time information collection, and dynamically adjust the collection strategy according to the characteristics of the data sources and business requirements, thereby ensuring the efficiency and adaptability of data collection. Through data extraction and conversion, appropriate extraction strategies can be formulated for different types of data, and the data can be effectively cleaned, unified in format and fused, providing a high-quality data basis for modeling. In the model building, appropriate algorithms are selected according to the characteristics of the data to accurately build a model reflecting the data relationship and characteristics. After model verification and optimization, the performance and generalization ability of the model can be improved, and a high-quality, stable and reliable model can be built, which is helpful for data-driven decision-making and application. BRIEF DESCRIPTION OF DRAWINGS

[0077] Fig. 1 The flowchart of the multi-source heterogeneous data model modeling method of the present application;

[0078] Fig. 2 The structural diagram of the multi-source heterogeneous data model modeling system of the present application. DETAILED DESCRIPTION

[0079] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0080] In order to solve the technical problem that the existing data modeling method mainly models single data source or homogeneous data, and it is difficult to effectively process multi-source heterogeneous data, please refer to Figs. 1-2 The technical solutions of the present embodiment are as follows:

[0081] A multi-source heterogeneous data model modeling method, the method comprising the following steps:

[0082] Global real-time information collection:

[0083] Supporting access to multiple different types of data sources, including but not limited to sensor data, SCADA system data, database data and file system data, realizing comprehensive collection of multi-source heterogeneous data;

[0084] Among them, according to the real-time requirement and priority of the data source, the collection frequency is adjusted, and then the collection frequency corresponding to the target data source is adjusted by using the priority proportion and the calibration data volume parameters corresponding to the target data source;

[0085] Data extraction and conversion:

[0086] The collected multi-source heterogeneous data is extracted and converted, and data in different formats and structures is uniformly converted into an intermediate format suitable for modeling, and the conversion process includes data cleaning, format specification unification and data fusion operation;

[0087] Model construction:

[0088] Based on the converted multi-source heterogeneous data, a preset modeling algorithm and rule are used to construct a data model, and the constructed data model is used to reflect the relationship and characteristics between the multi-source heterogeneous data;

[0089] Model verification and optimization:

[0090] The constructed data model is verified and evaluated, and the data model is optimized and adjusted according to the verification result, including optimization of model parameters and improvement of model structure.

[0091] The technical effects of the above technical solutions are that: supporting access to multiple different types of data sources, comprehensive collection of multi-source heterogeneous data can be realized, thereby providing a rich and comprehensive data basis for subsequent data processing and modeling, through data extraction and conversion processing, multi-source heterogeneous data of different formats and structures can be uniformly converted into an intermediate format suitable for modeling, and during the conversion process, data cleaning, format specification unification and data fusion operations are performed, which can effectively improve the quality and consistency of multi-source heterogeneous data, further providing a strong guarantee for building an accurate model, and based on the converted multi-source heterogeneous data, a preset modeling algorithm and rule are used to build a data model, which can accurately reflect the relationship and characteristics between multi-source heterogeneous data, at the same time, the built model is verified and evaluated, and the model is optimized and adjusted according to the verification result, thereby improving the performance and generalization ability of the model, making it more stable and reliable in actual application.

[0092] Global real-time information collection, including:

[0093] Supporting access to multiple different types of data sources including sensor data, SCADA system data, database data and file system data;

[0094] Through a unified access interface, seamless connection to different data sources is realized, and multi-source heterogeneous data is comprehensively collected;

[0095] At the same time, according to the characteristics and business requirements of the data source, data collection strategies are adopted, and the data collection strategy is dynamically adjusted according to the network status and storage resources, including adjusting the collection frequency and data volume.

[0096] The technical effects of the above technical solutions are that: supporting access to multiple different types of data sources including sensor data, SCADA system data, database data and file system data, comprehensive collection of multi-source heterogeneous data can be realized, through a unified access interface, seamless connection to different data sources is realized, simplifying the data collection process, improving the efficiency and reliability of data collection, ensuring the integrity and accuracy of multi-source heterogeneous data, and according to the characteristics and business requirements of the data source, data collection strategies are adopted, and the data collection strategy is dynamically adjusted according to the network status and storage resources, through flexible collection strategies and dynamic adjustment mechanism, complex network environment and limited storage resources can be effectively coped with, ensuring the stability and sustainability of data collection.

[0097] According to the characteristics and business requirements of the data source, data collection strategies are adopted, and the data collection strategy is dynamically adjusted according to the network status and storage resources, including:

[0098] According to the business scenario, the real-time requirements of each data source are determined, and according to the business target, the priority of each data source is determined;

[0099] According to the determined data source real-time requirement and priority, a data collection strategy is selected for each data source, wherein the data collection strategy includes determining the collection frequency, data volume, and allocating network bandwidth and storage resources according to the priority;

[0100] Real-time monitoring of network bandwidth usage, identification of network congestion and bandwidth bottlenecks, and periodic testing of network latency and stability, recording network interruptions and data packet loss for evaluating network reliability;

[0101] At the same time, the usage of the storage system is regularly checked, including the remaining space and the health status of the storage devices;

[0102] According to the real-time status of the network bandwidth and storage resources, the collection frequency and data volume of each data source are dynamically adjusted.

[0103] The technical effects of the above technical solution are: according to the real-time requirement of each data source determined by the business scenario, the priority is determined according to the business target, so as to match the most suitable collection strategy for each data source, improve the accuracy and pertinence of data collection, determine the collection frequency, data volume and reasonably allocate network bandwidth and storage resources according to the priority of the data source, which can prioritize the data collection task of the key data source, and through real-time monitoring of the status of network bandwidth and storage system, the collection frequency and data volume can be dynamically adjusted according to real-time information. The dynamic adjustment mechanism can effectively deal with network congestion, bandwidth bottleneck, insufficient storage space and other problems, so as to avoid data loss or interruption in data collection, and ensure the continuity and stability of data collection.

[0104] Specifically, the collection frequency is adjusted according to the real-time requirement and priority of the data source, including:

[0105] Real-time collection of network bandwidth occupancy ratio of the data source;

[0106] When the network bandwidth occupancy ratio of the data source exceeds the preset ratio reference value, the data source is taken as the target data source;

[0107] The data volume of the current single data collection of the target data source and the priority weight coefficient corresponding to the target data source are retrieved; wherein the value range of the ratio reference value is 18%-23%;

[0108] The data volume of the current single data collection of the target data source is normalized to obtain the data volume of the current single data collection of the target data source after normalization as the calibration data volume parameter;

[0109] From all data sources, the data source whose network bandwidth occupancy ratio does not exceed the preset ratio reference value is taken as the reference data source;

[0110] Call the priority weight coefficient of the reference data source, and obtain the priority weight coefficient and value corresponding to all reference data sources according to the priority weight coefficient of the reference data source;

[0111] The priority weight coefficient corresponding to the target data source is used for ratio processing with the priority weight coefficient sum value to obtain a priority proportion;

[0112] The priority proportion corresponding to the target data source is used to adjust the acquisition frequency corresponding to the target data source.

[0113] The technical effects of the above technical solutions are: by comparing the real-time bandwidth occupation ratio with the preset threshold (18%-23%), the data source (target data source) with excessively high bandwidth occupation can be located in time, avoiding network congestion caused by excessive bandwidth occupation of a single data source. For the target data source, the weight of the target data source is compared with the weight of other normal data sources (reference data sources) in combination with the data amount (after normalization) and the priority weight, and "on-demand adjustment" is realized. By calculating the priority proportion (ratio of target weight to reference data source weight sum) of the target data source, the abstract priority is converted into a quantifiable adjustment basis, ensuring that the high-priority data source can still maintain a reasonable acquisition frequency when the bandwidth is tight, avoiding data acquisition delay or loss caused by insufficient bandwidth. After normalizing the data amount, the frequency is adjusted in combination with the priority proportion, which can avoid excessive resource occupation of the data source with large data amount but low priority, while ensuring that the data source with small data amount but high priority can obtain sufficient acquisition resources, achieving the balance between timeliness and resource consumption. Based on the real-time bandwidth occupation data, the acquisition frequency is adjusted, so that the system can adapt to the fluctuation of network bandwidth, avoid waste or insufficient bandwidth resources caused by fixed frequency acquisition, and improve the robustness of the system in different network environments. The data amount is normalized to eliminate the influence of the data amount difference of different data sources, so that the adjustment of the acquisition frequency is more comparable and accurate, avoiding adjustment deviation caused by different data amount scales. By threshold judgment, potential bandwidth occupation abnormal sources are identified in advance, and the acquisition frequency is actively adjusted, rather than passively coping with network congestion, thereby reducing the risk of system paralysis caused by excessive bandwidth.

[0114] Specifically, adjusting the acquisition frequency corresponding to the target data source by using the priority proportion corresponding to the target data source and the calibrated data amount parameter comprises:

[0115] Call the priority proportion corresponding to the target data source and the calibrated data amount parameter;

[0116] Call the network bandwidth occupation ratio of the target data source;

[0117] Obtain the actual data amount acquisition intensity corresponding to the target data source by using the calibrated data amount parameter and the network bandwidth occupation ratio corresponding to the target data source;

[0118] Wherein, the actual data volume acquisition intensity corresponding to the target data source is obtained by the following formula:

[0119] E 01 = e-b*|1-P*(1+k)|

[0120] Wherein, E 01 represents the actual data volume acquisition intensity corresponding to the target data source; b represents the calibration data volume parameter corresponding to the target data source; P represents the network bandwidth occupation ratio corresponding to the target data source; k represents the standard deviation of the network bandwidth occupation ratio in the historical operation record of the target data source; specifically, |1-P*(1+k)| reflects the "real-time bandwidth tension + historical fluctuation amplification" (when the bandwidth occupation is high and the fluctuation is large, P□(1+k) tends to 1); e -b*|1-P*(1+k)| The smaller the value is, the more the actual data volume that can be stably collected will be reduced due to the limitation of network resources;

[0121] The actual demand priority intensity corresponding to the target data source is obtained by the following formula:

[0122] Wherein, the actual demand priority intensity corresponding to the target data source is obtained by the following formula:

[0123]

[0124] Wherein, E 02 represents the actual demand priority intensity corresponding to the target data source; w represents the normalized priority ratio corresponding to the target data source; specifically, represents the "compensation ability of priority to resource competition"; represents the construction of "demand-resource conflict factor", and when the bandwidth occupation is high and the priority compensation is weak, the conflict is more intense; for converting "conflict" into "demand intensity attenuation", E 02 The larger the value is, the more the high-priority demand can be fully met due to network resource conflict.

[0125] The acquisition frequency of the target data source is adjusted by using the actual data volume acquisition intensity and the actual demand priority intensity corresponding to the target data source.

[0126] Wherein, the adjusted acquisition frequency of the target data source is obtained by the following formula:

[0127] F new =F old *[1+max(E 01 ,E 02 )]

[0128] Wherein, F newdenotes the adjusted target data source collection frequency; F old denotes the target data source collection frequency before adjustment.

[0129] The technical effects of the above technical solutions are: through E 01 Quantify the actual load pressure during data collection by combining the calibration data volume, real-time bandwidth occupation, and historical bandwidth fluctuation (standard deviation k), and reflect the collection intensity under the dual constraints of "data volume + bandwidth fluctuation".E 02 Correlate bandwidth occupation and priority proportion, convert priority to actual demand intensity, and reflect the driving of business demand on collection frequency. Based on E 01 (real-time data side) and E 02 (historical data side), dynamically adjust the frequency to adapt the changes in network load and business priority, and avoid "bandwidth waste" or "insufficient collection" caused by fixed frequency. By selecting the key driving factors through max(E 01 ,E 02 ), and combining with the original frequency F old , not only respect the most urgent collection demand (data pressure or business priority), but also retain the continuity of frequency adjustment, balance resource consumption and business value. Introducing historical bandwidth standard deviation k, normalized priority w, etc. to make frequency adjustment cover "real-time state + historical fluctuation + business weight", improve the stability of the system in dealing with complex network and diverse demand, and reduce the impact of extreme situations (such as bandwidth surge, high-priority tasks) on collection.

[0130] Data extraction and conversion, including:

[0131] Data extraction:

[0132] According to the characteristics of the data source, develop appropriate extraction strategies, including,

[0133] For sensor data with high real-time requirements, use a polling mechanism to extract data periodically at a certain time interval;

[0134] For large batches of database data, use batch extraction, extract data changes at regular intervals or according to trigger conditions;

[0135] For file system data, set up a file listening mechanism to trigger data extraction when the file is updated;

[0136] Data conversion:

[0137] After data extraction, remove noise and outliers from multi-source heterogeneous data, delete records with missing values or fill them with interpolation method;

[0138] The multi-source heterogeneous data in different formats is uniformly converted into a unified data type required for modeling, and relevant data from multiple data sources are associated to form a complete data record, and information of different data sources is integrated.

[0139] Meanwhile, for time series data, the multi-source heterogeneous data in different frequencies are interpolated or sampled to have the same time interval according to time stamp alignment and integration.

[0140] The technical effects of the above technical solutions are as follows: according to the characteristics of different data sources, the high real-time sensor data is extracted by polling, the changed part of the large batch of database data is extracted by batch, and the file system update data is extracted by listening, so that efficient and accurate data extraction can be realized to meet the data acquisition requirements of different data sources; then, by removing noise and outliers, deleting missing value records or using interpolation filling, the purity and integrity of multi-source heterogeneous data can be effectively improved, so that the accuracy and stability of the model can be guaranteed; and the multi-source heterogeneous data in different formats is uniformly converted into the required type for modeling, and the multi-source heterogeneous data is associated to form a complete record, and various information resources are integrated, which helps to reveal the potential association and characteristics between data, improves the comprehensiveness and insight of the data model, and aligns and integrates the time series data according to the time stamp, so that the different frequency data has the same time interval, which can ensure the synchronization and consistency of the multi-source heterogeneous data in the time dimension.

[0141] Model construction, comprising:

[0142] Statistical analysis is performed on the converted multi-source heterogeneous data to understand the concentration trend, dispersion degree and correlation of the multi-source heterogeneous data.

[0143] Suitable modeling algorithms are selected, wherein for classification problems, logistic regression, decision tree and random forest are selected, and for regression problems, linear regression is selected.

[0144] A data model is constructed according to the selected modeling algorithm, and the model structure is initialized and the initial parameters of the model are set.

[0145] A large amount of existing historical data is collected as training data to train the constructed data model, and during the training process, the training progress and performance indicators of the data model are monitored, including loss value and accuracy.

[0146] The technical effects of the above technical solutions are: by statistically analyzing the converted multi-source heterogeneous data, the concentration trend, dispersion degree and correlation of the data can be accurately grasped, key basis for model construction is provided, suitable modeling algorithms are selected according to different problem types, the pertinence and effectiveness of model construction can be improved, a large amount of historical data is used to fully train the data model, and the training progress and performance indicators are monitored in real time, problems occurring in the training process can be found and solved in time, the model is ensured to converge to an optimal state, and a high-quality and high-precision data model is output, from data statistical analysis to algorithm selection, to model initialization, training and monitoring, a systematic model construction process is formed, which helps to construct a data model that meets actual needs and has excellent performance.

[0147] Model verification and optimization, including:

[0148] A large amount of existing historical data is collected as a validation set to verify the data model, which is used to evaluate the stability and generalization ability of the data model;

[0149] And according to the model type, select the appropriate evaluation index, wherein for the classification model, the evaluation index includes accuracy, recall rate and F1 value, for the regression model, the evaluation index includes mean square error, root mean square error and mean absolute error;

[0150] Compare the data models trained by different modeling algorithms, and select the optimal data model according to the performance index evaluation result;

[0151] Through grid search, random search or Bayesian optimization method, the hyperparameters of the data model are optimized, and the structure of the data model is optimized according to the performance and complexity of the data model;

[0152] After the above operations, the data model is converted into a deployable format and encapsulated, at the same time, the environment required for data model deployment is prepared, and the encapsulated data model is deployed to the actual production environment, by receiving input data and generating prediction results;

[0153] Continuously monitor the performance of the data model in actual application, including but not limited to prediction accuracy and response time;

[0154] And according to the performance, regularly update and maintain the data model, including retraining the model or adjusting the model parameters.

[0155] The technical effects of the above technical solutions are as follows: a large amount of historical data is used as a verification set, combined with classification evaluation indexes such as accuracy, recall rate, and F1 value, and regression evaluation indexes such as mean square error, root mean square error, and mean absolute error, to comprehensively and objectively evaluate the performance of data models trained by different modeling algorithms, to provide a reliable basis for selecting the optimal data model, to finely tune the data model hyperparameters through methods such as grid search, random search, or Bayesian optimization, to optimize the data model structure according to the performance and complexity of the data model, to further improve the prediction accuracy and efficiency of the data model, to convert the optimized data model into a deployable format and encapsulate it, to prepare a deployment environment, to ensure that the data model can be quickly and stably deployed to a production environment to provide data support and decision-making basis for enterprises or applications in a timely manner, to continuously monitor the performance of the data model in actual application, and to regularly update and maintain the data model according to performance changes, to ensure that the data model always maintains good performance and adaptability during long-term operation.

[0156] Specifically, the embodiment also provides a multi-source heterogeneous data modeling system for implementing a multi-source heterogeneous data modeling method, including:

[0157] A data acquisition module is configured to configure a unified access interface to perform global real-time information acquisition from various data sources including sensor data, SCADA system data, database data, and file system data, and to acquire multi-source heterogeneous data.

[0158] Among them, according to the characteristics and business requirements of the data source, a data acquisition strategy is used for data acquisition, and the data acquisition strategy is dynamically adjusted according to the network status and storage resources.

[0159] A data processing module is configured to extract and convert the acquired multi-source heterogeneous data, and to perform cleaning, format unification, and data fusion operations during the conversion process.

[0160] A model construction module is configured to construct a data model based on the multi-source heterogeneous data processed by the data processing module using a preset modeling algorithm and rule, and to collect a large amount of existing historical data as training data to train the model.

[0161] A verification and optimization module is configured to verify and evaluate the constructed data model using a verification set, and to optimize and adjust the data model according to the verification result, including tuning of model parameters and improvement of model structure.

[0162] A model deployment module is configured to deploy the trained and verified and optimized data model to an actual production environment, and to continuously monitor the performance of the data model in actual application, and to regularly update and maintain the data model according to the performance.

[0163] The technical effects of the above technical solutions are: the data acquisition module supports the access of multiple data sources and can dynamically adjust the acquisition strategy according to the data source characteristics and business needs, ensuring the comprehensiveness, accuracy and efficiency of data acquisition, while adapting to different network and storage environments. The data processing module converts multi-source heterogeneous data into high-quality data suitable for modeling through extraction, cleaning, format unification and data fusion operations, thereby laying a solid foundation for subsequent model construction. The model construction module trains based on appropriate algorithms and a large amount of historical data, and combines the evaluation and optimization functions of the verification and optimization module to build a data model with high precision, high stability and strong generalization ability. The model deployment module ensures that the constructed data model can be smoothly deployed to the production environment and continuously monitors its performance, updates and maintains it in a timely manner, and ensures the long-term effectiveness of the data model in actual application. The multi-source heterogeneous data modeling system has a complete process from data acquisition, processing, model construction, verification and optimization to model deployment, and the modules work together to efficiently process and utilize multi-source heterogeneous data, providing comprehensive support for data-driven decision-making.

[0164] The data acquisition module comprises:

[0165] The requirement analysis module is configured to analyze and determine the real-time requirements of each data source according to the business scenario, and evaluate the importance of each data source and determine the priority of each data source according to the business target;

[0166] The strategy formulation module is configured to select appropriate data acquisition strategies for each data source based on the real-time requirements and priorities of the data sources;

[0167] The network monitoring module is configured to monitor network bandwidth usage in real time, identify network congestion and bandwidth bottlenecks in a timely manner, and periodically test network latency and stability, record network interruptions and data packet loss, and if data packet loss or high latency occurs continuously for multiple times, it is determined that the network stability is poor;

[0168] The storage monitoring module is configured to periodically check the usage of the storage system, including the remaining space and the health status of the storage devices, to detect storage space shortages or device failure hazards in advance;

[0169] The strategy adjustment module is configured to make dynamic adjustment decisions for the data acquisition strategy based on the real-time monitoring of the network bandwidth and storage resources, and adjust the data acquisition strategy according to the actual execution of the decisions.

[0170] The technical effects of the above technical scheme are: through the demand analysis module, the real-time requirements and priority of each data source are accurately determined, the strategy formulation module can customize the most suitable collection strategy for each data source, thereby improving the relevance and effectiveness of data collection, the network monitoring module monitors the network status in real time, and network problems can be found and warned in time, and in combination with the periodic inspection of the storage monitoring module on the storage resources, comprehensive environmental information can be provided for the strategy adjustment module, so that the strategy adjustment module dynamically optimizes the collection strategy, thereby effectively dealing with network fluctuations and storage limitations, and further ensuring the stability, continuity and data integrity of data collection.

[0171] Working principle: by accessing multiple different types of data sources, comprehensive collection of multi-source heterogeneous data can be achieved, and through data extraction and conversion processing, multi-source heterogeneous data of different formats and structures can be uniformly converted into an intermediate format suitable for modeling, which can effectively improve the quality and consistency of data and provide a strong guarantee for building accurate models. Based on the converted data, a preset modeling algorithm and rule are used to build a data model, which can accurately reflect the relationship and characteristics between multi-source heterogeneous data. At the same time, the built data model is verified and evaluated, and the data model is optimized and adjusted according to the verification result, thereby improving the performance and generalization ability of the data model. In this way, a high-quality, stable and reliable data model is built.

[0172] It should be noted that in this paper, relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also other elements not explicitly listed or inherent to such process, method, article or equipment.

[0173] Although embodiments of the present application have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and variations can be made to the embodiments without departing from the principles and spirit of the present application.

Claims

1. A multi-source heterogeneous data model modeling method, characterized in that, The method comprises the following steps: Global real-time information collection: Support access to a variety of different types of data sources, including but not limited to sensor data, SCADA system data, database data and file system data, to achieve comprehensive collection of multi-source heterogeneous data; Among them, according to the real-time requirement and priority of the data source, the collection frequency is adjusted, and then the collection frequency corresponding to the target data source is adjusted by using the priority proportion and the calibration data volume parameters corresponding to the target data source; Data extraction and conversion: Extract and convert the collected multi-source heterogeneous data, and uniformly convert the data in different formats and structures into an intermediate format suitable for modeling, including data cleaning, format specification, and data fusion operations; Model building: Based on the converted multi-source heterogeneous data, a preset modeling algorithm and rule are used to build a data model, and the built data model is used to reflect the relationship and characteristics between multi-source heterogeneous data; Model verification and optimization: Verify and evaluate the constructed data model, and optimize and adjust the data model according to the verification result, including model parameter tuning and model structure improvement.

2. The multi-source heterogeneous data model modeling method according to claim 1, characterized in that, The global real-time information collection comprises: Support access to a variety of different types of data sources, including sensor data, SCADA system data, database data and file system data; Through a unified access interface, seamless connection to different data sources is achieved, and multi-source heterogeneous data is comprehensively collected; At the same time, according to the characteristics and business requirements of the data source, data collection strategy is adopted, and the data collection strategy is dynamically adjusted according to the network status and storage resources, including adjusting the collection frequency and data volume.

3. The multi-source heterogeneous data model modeling method according to claim 2, characterized in that, According to the characteristics and business requirements of the data source, data collection strategy is adopted, and the data collection strategy is dynamically adjusted according to the network status and storage resources, including: According to the business scenario, determine the real-time requirement of each data source, and according to the business target, determine the priority of each data source; According to the determined real-time requirement and priority of the data source, select a data collection strategy for each data source, wherein the data collection strategy includes determining the collection frequency, data volume, and allocating network bandwidth and storage resources according to the priority; Real-time monitoring of network bandwidth usage, identifying network congestion and bandwidth bottlenecks, and regularly testing network delay and stability, recording network interruption and data packet loss for evaluating network reliability; At the same time, regularly check the usage of the storage system, including the remaining space and the health status of the storage device; According to the real-time status of the network bandwidth and storage resources, dynamically adjust the collection frequency and data volume of each data source.

4. The multi-source heterogeneous data model modeling method according to claim 3, characterized in that, According to the real-time requirement and priority of the data source, adjust the collection frequency, including: Real-time collection of network bandwidth occupancy ratio of the data source; When the network bandwidth occupancy ratio of the data source exceeds the preset ratio reference value, the data source is regarded as a target data source; Call the current single data collection data volume of the target data source and the priority weight coefficient corresponding to the target data source; wherein the value range of the ratio reference value is 18%-23%. Normalizing the data volume of the current single data collection of the target data source, obtaining the data volume of the current single data collection of the target data source after normalization as a calibration data volume parameter; Call data sources with network bandwidth occupation ratio not exceeding the preset ratio reference value from all data sources as reference data sources; Call the priority weight coefficient of the reference data source, and obtain the priority weight coefficient sum value corresponding to all reference data sources according to the priority weight coefficient of the reference data source; Perform ratio processing on the priority weight coefficient corresponding to the target data source and the priority weight coefficient sum value to obtain a priority proportion; Adjust the collection frequency corresponding to the target data source by using the priority proportion corresponding to the target data source and the calibration data volume parameter.

5. The multi-source heterogeneous data model modeling method according to claim 4, characterized in that, Adjusting the collection frequency corresponding to the target data source by using the priority proportion corresponding to the target data source and the calibration data volume parameter, comprising: Call the priority proportion corresponding to the target data source and the calibration data volume parameter; Call the network bandwidth occupation ratio of the target data source; Obtain the actual data volume collection intensity corresponding to the target data source by using the calibration data volume parameter and the network bandwidth occupation ratio corresponding to the target data source; Obtain the actual demand priority intensity by using the network bandwidth occupation ratio and the priority proportion corresponding to the target data source; Adjust the collection frequency of the target data source by using the actual data volume collection intensity and the actual demand priority intensity corresponding to the target data source.

6. The multi-source heterogeneous data model modeling method according to claim 1, characterized in that, The data extraction and conversion comprises: Data extraction: According to the characteristics of the data source, a suitable extraction strategy is formulated, wherein, For sensor data with high real-time requirements, a polling mechanism is adopted to periodically extract data at a certain time interval; For large batches of database data, a batch extraction method is adopted to extract data changes at a certain time or according to a trigger condition; For file system data, a file listening mechanism is set up to trigger data extraction when the file is updated; Data conversion: After data extraction, remove noise and outliers in multi-source heterogeneous data, delete records containing missing values or fill them in by interpolation; Then, unify the multi-source heterogeneous data of different formats into the unified data type required for modeling, and associate the related data from multiple data sources to form a complete data record and integrate the information of different data sources; At the same time, for time series data, align and integrate according to the timestamp, and interpolate or sample multi-source heterogeneous data of different frequencies to have the same time interval.

7. The multi-source heterogeneous data model modeling method according to claim 1, characterized in that, The model construction comprises: Statistical analysis of the converted multi-source heterogeneous data to understand the concentration trend, dispersion degree and correlation of the multi-source heterogeneous data; Screening suitable modeling algorithms, wherein, for classification problems, selecting logistic regression, decision tree and random forest, and for regression problems, selecting linear regression; Constructing a data model according to the selected modeling algorithm, and initializing the model structure and setting the initial parameters of the model; Collect a large amount of existing historical data as training data to train the constructed data model, and monitor the training progress and performance indicators of the data model during the training process, including loss value and accuracy.

8. The multi-source heterogeneous data model modeling method according to claim 1, characterized in that, The model verification and optimization comprises: A large amount of historical data is collected as a validation set to verify the data model, which is used to evaluate the stability and generalization ability of the data model; And according to the model type, select the appropriate evaluation index, wherein for the classification model, the evaluation index includes accuracy, recall rate and F1 value, for the regression model, the evaluation index includes mean square error, root mean square error and mean absolute error; Compare the data models trained by different modeling algorithms, and select the optimal data model according to the performance index evaluation result; Through grid search, random search or Bayesian optimization method, the hyperparameters of the data model are optimized, and the structure of the data model is optimized according to the performance and complexity of the data model; After the above operations, the data model is converted into a deployable format and encapsulated, and at the same time, the environment required for the deployment of the data model is prepared, and the encapsulated data model is deployed to the actual production environment, and the prediction result is generated by receiving the input data; Continuously monitor the performance of the data model in actual application, including but not limited to prediction accuracy, response time; And according to the performance, regularly update and maintain the data model, including retraining the model or adjusting the model parameters.

9. A multi-source heterogeneous data model modeling system for implementing the multi-source heterogeneous data model modeling method according to any one of claims 1-8, characterized in that, It includes: The data acquisition module is configured to configure a unified access interface to collect real-time information from various data sources including sensor data, SCADA system data, database data and file system data, and obtain multi-source heterogeneous data; According to the characteristics and business requirements of the data source, the data acquisition strategy is adopted for data acquisition, and the data acquisition strategy is dynamically adjusted according to the network status and storage resources; The data processing module is configured to extract and convert the collected multi-source heterogeneous data, and perform cleaning, format unification and data fusion operations during the conversion process; The model construction module is configured to construct a data model based on the multi-source heterogeneous data processed by the data processing module, using a preset modeling algorithm and rule, and collect a large amount of existing historical data as training data to train the model; The verification and optimization module is configured to verify and evaluate the constructed data model using the validation set, and optimize and adjust the data model according to the verification result, including model parameter optimization and model structure improvement; The model deployment module is configured to deploy the data model trained and verified and optimized to the actual production environment, and continuously monitor the performance of the data model in actual application, and regularly update and maintain the data model according to the performance.

10. The multi-source heterogeneous data model modeling system of claim 9, wherein, The data acquisition module includes: The requirement analysis module is configured to analyze and determine the real-time requirements of each data source according to the business scenario, and evaluate the importance of each data source according to the business target, and determine the priority of each data source; The strategy formulation module is configured to select appropriate data acquisition strategies for each data source according to the real-time requirements and priorities of the data sources. The network monitoring module is configured to monitor network bandwidth usage in real time, identify network congestion and bandwidth bottlenecks in a timely manner, test network delay and stability regularly, record network interruption and data packet loss, and determine that the network stability is poor if data packet loss or high delay occurs continuously for multiple times. The storage monitoring module is configured to check the usage of the storage system regularly, including the remaining space and the health status of the storage device, to find storage space shortage or device failure hidden danger in advance. The policy adjustment module is configured to make dynamic adjustment decisions for the data collection policy according to the real-time monitoring status of the network bandwidth and the storage resource, and to actually execute the adjustment operation of the data collection policy according to the decisions.

Images