Data development method and platform based on multi-source data integration

Abstract

The present invention belongs to the technical field of data processing, and discloses a data development method and platform based on multi-source data integration, which collects and pre-processes multi-source heterogeneous data to obtain multi-source initial data, extracts initial features of different data sources, and forms a preliminary feature set; establishes and optimizes an event tree based on the collected multi-source heterogeneous data and the corresponding preliminary feature set, mines the multi-source data collected in real time based on the optimized event tree, analyzes the multi-source data collected in real time based on an established risk assessment model, compares and analyzes the actual risk events obtained as risk prediction results, accesses and operates data through a user interface, realizes efficient integration, processing, analysis and application of data, and improves the overall level of data development and utilization.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a data development method and platform based on multi-source data integration. Background Art

[0002] With the rapid development of information technology and the popularization of the Internet, all walks of life have accumulated a huge amount of multi-source data. These data come from a wide range of sources and are highly heterogeneous, covering structured data, semi-structured data, and unstructured data. For example, sensor data, social media data, geospatial data, and business transaction data. These data have significant differences in format, structure, semantics, etc., making data integration and processing complex and challenging.

[0003] When faced with such a huge amount of multi-source heterogeneous data, traditional data processing and analysis methods have become incapable. Traditional data processing and analysis methods are based on structured data processing technologies, such as relational database management systems (RDBMS), data warehouses, and traditional data mining technologies. Traditional data processing and analysis methods rely on fixed data models and strict data structures, and are suitable for processing structured data, such as data in tabular form. However, when faced with multi-source heterogeneous data (including structured, semi-structured, and unstructured data), traditional methods have significant limitations in data integration, processing, and analysis. This includes the inability to effectively handle the diversity of data sources, format differences, and the need for cross-domain collaborative analysis. Therefore, when dealing with the integration and processing of multi-source heterogeneous data, traditional data processing and analysis methods cannot achieve cross-domain collaborative analysis of collected data, resulting in difficulties in integrating the collected multi-source heterogeneous data, which is not conducive to task scheduling and data analysis. Summary of the invention

[0004] To solve the above problems, the present invention provides a data development method and platform based on multi-source data integration, aiming to achieve efficient integration, processing, analysis and application of data through innovative steps and algorithms, improve the overall level of data development and utilization, and provide users with powerful decision-making support and risk management tools.

[0005] The data development method based on multi-source data integration described in the present invention comprises the following steps:

[0006] S10, collecting multi-source heterogeneous data and preprocessing them to obtain multi-source initial data;

[0007] S20, based on a unified identifier and data model, perform multimodal integration on the multi-source initial data, and extract initial features of different data sources from the multimodally integrated multi-source initial data to form a preliminary feature set;

[0008] S30, establishing an event tree based on the collected multi-source heterogeneous data and the corresponding preliminary feature set, wherein each node in the event tree is defined to represent different types of data features, and the path in the event tree is analyzed;

[0009] S40, eliminating redundant nodes and paths in the event tree, identifying target paths and nodes in the event tree, and obtaining an optimized event tree;

[0010] S50. Mining the multi-source data collected in real time based on the optimized event tree, analyzing the multi-source data collected in real time according to the established risk assessment model, comparing and analyzing the risk prediction results with the actual risk events, and accessing and operating the data through the user interface.

[0011] As a further solution of the present invention, S10 is specifically:

[0012] Collect multi-source heterogeneous data based on the determined data sources;

[0013] Convert the data to a standard normal distribution and change it to a unified time format;

[0014] The collected multi-source heterogeneous data are merged according to the time format and organized into a unified initial data set.

[0015] As a further solution of the present invention, in S20, multi-modal integration of multi-source initial data includes the following steps:

[0016] Determine an identifier that uniquely identifies each entity according to the time format of the initial data set, where the identifier includes an ID and a timestamp;

[0017] The unified identifiers of the initial data set are aligned according to the constructed unified data model, and the conflicting data are weighted according to the fusion rule of data averaging;

[0018] The aligned data are integrated according to a unified data model, and conflicting data are eliminated using the data averaging fusion rule to generate an integrated data set.

[0019] As a further solution of the present invention, weighted timing is performed on conflicting data according to a fusion rule of data averaging, including the following steps:

[0020] Traverse all data records and mark records with the same identifier but different values ​​as conflicting data based on the identifier of each entity;

[0021] Assign an initial weight to each data source based on its credibility, use weighted average to handle conflicting data, and calculate the weighted average for conflicting data with the same identifier;

[0022] Use a sliding window to adjust the weights based on the historical performance of the data source; the weighted average is:

[0023] ,

[0024] In the formula, is the data value after weighted average, For the The value of the data source, For data source The corresponding data source weight, is the number of data sources; the sum of the weights of each data source is 1, that is: .

[0025] As a further solution of the present invention, in S20, initial features of different data sources are extracted from the multi-source initial data of multi-modal integration to form a preliminary feature set, including the following steps:

[0026] Extract temporal features, statistical features, and frequency domain features from the integrated data set;

[0027] The extracted time features, statistical features and frequency domain features are transformed, the feature values ​​are converted into standard normal distribution, and the features extracted from different data sources are spliced ​​to form a preliminary feature set.

[0028] As a further solution of the present invention, when the extracted time features, statistical features and frequency domain features are subjected to feature transformation, it includes:

[0029] Transform the eigenvalues ​​to a standard normal distribution;

[0030] Features extracted from different data sources are concatenated into a long feature vector in a unified order;

[0031] in, data sources, and the feature vectors extracted from each are , then the concatenated feature vector is:

[0032] ,

[0033] In the formula, is the complete feature vector after concatenation, For data source Extracted feature vectors;

[0034] Among them, each feature When performing feature transformation:

[0035] ,

[0036] In the formula, is the mean of the feature, is the standard deviation of the feature, is the standardized eigenvalue, which is transformed into a standard normal distribution with a mean of 0 and a standard deviation of 1.

[0037] As a further solution of the present invention, S30 is specifically:

[0038] Define the root node, branches and nodes of the event tree based on multi-source heterogeneous data and the extracted preliminary feature set;

[0039] Starting from the root node, add nodes layer by layer, define the conditional relationships and transfer rules between nodes, and establish the relationships between nodes;

[0040] Define the data features of classification feature nodes, continuous feature nodes, time feature nodes and space feature nodes in the event tree, analyze and determine the path from the root node to the leaf node in the event tree, and construct the event tree.

[0041] As a further solution of the present invention, S40 is specifically:

[0042] Identify redundant nodes and paths by analyzing the frequency, feature similarity, and path contribution of nodes and paths;

[0043] Merge duplicate nodes through feature clustering and retain key paths through the shortest path;

[0044] Screen according to the Gini coefficient to identify target paths and nodes;

[0045] After removing redundant nodes and paths, the optimized event tree is rebuilt.

[0046] As a further solution of the present invention, when screening is performed according to the Gini coefficient, the path is calculated by the Gini coefficient, wherein:

[0047] ,

[0048] In the formula, For path The Gini coefficient of For path Middle The probability of a class.

[0049] As a further solution of the present invention, the multi-source data collected in real time is analyzed according to the established risk assessment model, and based on the optimized event tree, the pre-established risk assessment model is applied to analyze the real-time data to obtain the risk score and risk level, wherein the risk score calculation formula is:

[0050] ,

[0051] In the formula, Features The weights are determined based on model training; when actual risk events are obtained, the actual risk event data is collected as the data source for comparative analysis. The predicted results are compared with the actual risk events through statistical analysis to optimize the risk assessment model.

[0052] The present invention also provides a data development platform based on multi-source data integration, which is used to implement the above method, and the platform includes:

[0053] Data collection module: used to collect multi-source heterogeneous data from different sources and pre-process these data to form multi-source initial data;

[0054] Data integration module: Based on a unified identifier and data model, the pre-processed multi-source initial data is integrated in a multi-modal manner, and initial features of different data sources are extracted to form a preliminary feature set;

[0055] Event tree construction module: constructs an event tree based on the collected multi-source heterogeneous data and its preliminary feature set. Each node in the event tree represents different types of data features and is used to analyze the path in the event tree;

[0056] Event tree optimization module: used to eliminate redundant nodes and paths in the event tree, identify target paths and nodes, and thus obtain an optimized event tree;

[0057] Real-time data mining module: Based on the optimized event tree, it mines the multi-source data collected in real time, analyzes the data and predicts risks through the risk assessment model;

[0058] Risk assessment module: used to analyze multi-source data collected in real time, make risk predictions through the established risk assessment model, and compare and analyze the prediction results with actual risk events;

[0059] User interface module: provides a user interface for data access and operation, enabling users to view and operate data analysis results;

[0060] Data storage module: used to store multi-source heterogeneous data, preliminary feature sets, event trees and optimized event trees, real-time collected data, risk assessment models and analysis results.

[0061] The data development platform of the present invention integrates multi-source heterogeneous data, uses event trees to perform data analysis and risk assessment, provides real-time data mining and user interaction functions, and improves the accuracy and efficiency of data analysis.

[0062] Compared with the prior art, the data development method and platform based on multi-source data integration proposed in the present invention have the following beneficial effects:

[0063] 1. High data integration and processing efficiency: By collecting multi-source heterogeneous data and preprocessing them to form multi-source initial data, and using unified identifiers and data models, efficient integration of multimodal data is achieved, which not only improves the efficiency of data processing, but also ensures the consistency and integrity of the data;

[0064] 2. Accurate feature extraction: Extract initial features of different data sources from multi-source initial data to form a preliminary feature set, so that the data analysis process can focus on key features, reduce the interference of irrelevant data, and improve the accuracy of feature extraction;

[0065] 3. Event tree analysis is intuitive: By constructing an event tree, different types of data features are represented in the form of nodes, and the paths in the event tree are analyzed, making the data analysis process more intuitive; the event tree clearly shows the relationship and impact path between various data features, making it easier to understand and analyze;

[0066] 4. Redundancy elimination and optimization: The event tree optimization module can effectively eliminate redundant nodes and paths, identify target paths and nodes, and obtain an optimized event tree; this process improves the simplicity and logic of the event tree, reduces the interference of invalid information, and improves the efficiency and accuracy of data analysis;

[0067] 5. Real-time data mining and risk assessment: Based on the optimized event tree, the multi-source data collected in real time is mined and analyzed, and risk prediction is performed through the risk assessment model, realizing dynamic monitoring and rapid response to real-time data. This enables the platform to detect potential risks in a timely manner and take corresponding measures, thereby improving the effectiveness of risk management.

[0068] In summary, the present invention significantly improves the accuracy, efficiency and practicality of data analysis through efficient integration of multi-source data, accurate feature extraction, intuitive event tree analysis, redundant information elimination and optimization, real-time data mining and risk assessment, user-friendly interface design, and safe and reliable data storage, providing users with powerful decision-making support and risk management tools. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 A flowchart of a data development method based on multi-source data integration according to an embodiment of the present invention;

[0070] Figure 2 It is a flow chart of multi-modal integration of multi-source initial data in a data development method based on multi-source data integration according to an embodiment of the present invention;

[0071] Figure 3 A flowchart of establishing an event tree in a data development method based on multi-source data integration according to an embodiment of the present invention;

[0072] Figure 4 A flowchart of optimizing an event tree in a data development method based on multi-source data integration according to an embodiment of the present invention;

[0073] Figure 5 Schematic diagram of the test results of multi-source data integration efficiency, where (a) is the comparison of data integration time consumption, (b) is the trend of integration consistency change, (c) is the comparison of query response time, and (d) is the comparison of data throughput;

[0074] Figure 6 Schematic diagram of the feature extraction accuracy test results, where (a) is the feature matching comparison, (b) is the extraction error of different feature types, (c) is the impact of the similarity measurement method on the feature matching, and (d) is the impact of noise on the feature extraction error;

[0075] Figure 7 Schematic diagram of the event tree construction and optimization effect test results, where (a) is the comparison of event tree accuracy before and after optimization, (b) is the comparison of risk event identification before and after optimization, (c) is the comparison of the number of key path nodes and path length optimization, and (d) is the comparison of data processing time before and after optimization;

[0076] Figure 8 Schematic diagram of the test results of real-time data mining and risk assessment response speed, where (a) is the response time comparison between real-time data mining and batch processing methods, (b) is the update delay of real-time risk assessment results, (c) is the processing stability under the influence of data flow rate, and (d) is the response time comparison between ordinary events and high-risk events;

[0077] Fig. 9 The figure is a schematic diagram of the test results of the user interface operation convenience, where (a) is the average operation time distribution of users to complete tasks, (b) is the downward trend of user error rate with the usage stage, (c) is the user satisfaction survey score, and (d) is the distribution of user satisfaction feedback. DETAILED DESCRIPTION

[0078] The present invention is further described below in conjunction with the accompanying drawings and specific implementation methods. It should be noted that, under the premise of no conflict, the various embodiments or technical features described below can be arbitrarily combined to form a new embodiment.

[0079] Traditional data processing and analysis methods face many challenges in integrating and processing multi-source heterogeneous data, such as complex data collection preprocessing, difficult data integration, high storage management requirements, high requirements for analysis and mining technology, complex task scheduling, and difficult label management. The present invention proposes a data development method and platform based on multi-source data integration, which can effectively integrate and process multi-source heterogeneous data, improve data quality and analysis efficiency, and provide strong data support and decision-making basis for various industries.

[0080] See also Figure 1 As shown, an embodiment of the present invention provides a data development method based on multi-source data integration, the method comprising the following steps:

[0081] S10, collecting multi-source heterogeneous data and preprocessing them to obtain multi-source initial data.

[0082] In this step, multi-source heterogeneous data is collected and preprocessed to obtain multi-source initial data, including the following steps:

[0083] Collect multi-source heterogeneous data based on the determined data sources;

[0084] Convert the data to a standard normal distribution and change it to a unified time format;

[0085] The collected multi-source heterogeneous data are merged according to the time format and organized into a unified initial data set.

[0086] In this step, if a financial institution needs to collect customer information from multiple data sources (such as bank transaction records, credit card consumption data, social media behavior data, etc.), in order to integrate this data, it is necessary to pre-process it first. The collected multi-source heterogeneous data includes:

[0087] (1) Obtain transaction records from the bank (data source A);

[0088] (2) Obtain consumption data from credit card companies (data source B);

[0089] (3) Obtain behavioral data from social media platforms (data source C).

[0090] When converting data to a standard normal distribution and unifying the time format, the amount data, consumption frequency and other features in each data source are converted to a standard normal distribution to eliminate the impact of dimensions and unify the time formats of different data sources into "YYYY-MM-DD HH:MM:SS".

[0091] When merging multi-source data, align the data by timestamp and organize all the data into an initial dataset with a unified time format and standard normal distribution characteristics.

[0092] S20. Based on a unified identifier and data model, the multi-source initial data is multimodally integrated, and initial features of different data sources are extracted from the multimodally integrated multi-source initial data to form a preliminary feature set.

[0093] In this step, see Figure 2 As shown in FIG. 1 , multi-modal integration of multi-source initial data includes the following steps:

[0094] S201, determining an identifier that uniquely identifies each entity according to the time format of the initial data set, wherein the identifier includes an ID and a timestamp;

[0095] S202, aligning the unified identifiers of the initial data set according to the constructed unified data model, and performing weight calculation on the conflicting data according to a fusion rule of taking an average of the data;

[0096] S203: Integrate the aligned data according to a unified data model, use a data averaging fusion rule to eliminate conflicting data, and generate an integrated data set.

[0097] Among them, weighting and timing the conflicting data according to the fusion rule of averaging the data includes the following steps:

[0098] Traverse all data records and mark records with the same identifier but different values ​​as conflicting data based on the identifier of each entity;

[0099] Assign an initial weight to each data source based on its credibility, use weighted average to handle conflicting data, and calculate the weighted average for conflicting data with the same identifier;

[0100] Use a sliding window to adjust the weights based on the historical performance of the data source; the weighted average is:

[0101] ,

[0102] In the formula, is the data value after weighted average, For the The value of the data source, For data source The corresponding data source weight, is the number of data sources; the sum of the weights of each data source is 1, that is: .

[0103] In this embodiment, extracting initial features of different data sources from multi-source initial data of multi-modal integration to form a preliminary feature set includes the following steps:

[0104] Extract temporal features, statistical features, and frequency domain features from the integrated data set;

[0105] The extracted time features, statistical features and frequency domain features are transformed, the feature values ​​are converted into standard normal distribution, and the features extracted from different data sources are spliced ​​to form a preliminary feature set.

[0106] Among them, when the extracted time features, statistical features and frequency domain features are subjected to feature transformation, it includes:

[0107] Transform the eigenvalues ​​to a standard normal distribution;

[0108] Features extracted from different data sources are concatenated into a long feature vector in a unified order;

[0109] in, data sources, and the feature vectors extracted from each are , then the concatenated feature vector is:

[0110] ,

[0111] In the formula, is the complete feature vector after concatenation, For data source Extracted feature vectors.

[0112] Among them, each feature When performing feature transformation:

[0113] ,

[0114] In the formula, is the mean of the feature, is the standard deviation of the feature, is the standardized eigenvalue, which is transformed into a standard normal distribution with a mean of 0 and a standard deviation of 1.

[0115] For example, a financial institution wants to assess the credit risk of a customer by integrating data from different sources. When determining the unique identifier, the customer ID and timestamp are used as the unique identifier, and then the unified identifier is aligned, and the records of the same customer from different data sources are aligned, and the weighted average fusion rule is used to handle conflicting data. For example, assuming that a customer has transaction records in both data sources A and B, but the amounts are different, the weighted average is calculated based on the credibility of the data source. Then the data set is integrated, the aligned data is integrated according to the unified data model, and the conflicting data is eliminated to generate the final data set.

[0116] S30, establishing an event tree based on the collected multi-source heterogeneous data and the corresponding preliminary feature set, wherein each node in the event tree is defined to represent different types of data features, and the path in the event tree is analyzed.

[0117] In this step, see Figure 3 As shown, establishing an event tree includes the following steps:

[0118] S301, defining the root node, branches and nodes of the event tree according to multi-source heterogeneous data and the extracted preliminary feature set;

[0119] S302, starting from the root node, adding nodes layer by layer, defining conditional relationships and transfer rules between nodes, and establishing relationships between nodes;

[0120] S303, defining data features of classification feature nodes, continuous feature nodes, time feature nodes, and space feature nodes in the event tree, analyzing and determining the path from the root node to the leaf node in the event tree, and constructing an event tree.

[0121] For example, if a financial institution wants to analyze the transaction behavior patterns of customers through an event tree to identify potential credit risks, when defining the event tree structure, the root node represents the customer transaction behavior, and the branches are classified according to different transaction characteristics (such as transaction amount, consumption category, etc.). Then, add nodes, starting from the root node, and add nodes layer by layer. For example, the root node is the customer transaction behavior, the next layer of nodes can be the transaction amount, and then further subdivided into consumption categories. Finally, define the feature nodes, where:

[0122] Classification feature node: consumption category;

[0123] Continuous feature nodes: transaction amount;

[0124] Time feature node: transaction time;

[0125] Spatial feature node: transaction location.

[0126] S40, eliminating redundant nodes and paths in the event tree, identifying target paths and nodes in the event tree, and obtaining an optimized event tree.

[0127] In this step, see Figure 4 As shown, when obtaining the optimized event tree, the following steps are included:

[0128] S401, identifying redundant nodes and paths by analyzing the frequency, feature similarity, and path contribution of nodes and paths;

[0129] S402, merging duplicate nodes through feature clustering, and retaining key paths through the shortest paths;

[0130] S403, screening according to the Gini coefficient to identify the target path and node;

[0131] S404: After removing redundant nodes and paths, rebuild the optimized event tree.

[0132] When screening based on the Gini coefficient, the path is calculated using the Gini coefficient, where:

[0133] ,

[0134] In the formula, For path The Gini coefficient of For path Middle The probability of a class.

[0135] In this embodiment, by optimizing the event tree, financial institutions can improve the accuracy of credit risk assessment, wherein when identifying redundant nodes and paths, redundant nodes and paths are identified and marked based on the frequency, feature similarity, and path contribution of nodes and paths. When merging duplicate nodes and retaining key paths, feature clustering methods are used to merge duplicate nodes, and key paths are retained through the shortest path algorithm. When performing Gini coefficient screening, the paths and nodes with the highest discrimination are screened out by calculating the Gini coefficient of the path. When reconstructing the optimized event tree, redundant nodes and paths are removed to reconstruct the optimized event tree.

[0136] S50. Mining the multi-source data collected in real time based on the optimized event tree, analyzing the multi-source data collected in real time according to the established risk assessment model, comparing and analyzing the risk prediction results with the actual risk events, and accessing and operating the data through the user interface.

[0137] In this step, the multi-source data collected in real time is analyzed according to the established risk assessment model. Based on the optimized event tree, the pre-established risk assessment model is applied to analyze the real-time data to obtain the risk score and risk level. The risk score calculation formula is:

[0138] ,

[0139] In the formula, Features The weights are determined based on model training; when actual risk events are obtained, the actual risk event data is collected as the data source for comparative analysis. The predicted results are compared with the actual risk events through statistical analysis to optimize the risk assessment model.

[0140] In this embodiment, the financial institution monitors customer transaction behavior in real time and conducts risk assessment. During real-time data analysis, the customer transaction data collected in real time is analyzed based on the optimized event tree. During risk assessment, a pre-established risk assessment model is applied to calculate the risk score and risk level according to the risk scoring formula. During comparative analysis and model optimization, actual risk event data is collected and compared with the prediction results. The comparison results are statistically analyzed to optimize the risk assessment model and improve the prediction accuracy.

[0141] Through the above steps, financial institutions can effectively collect and integrate multi-source heterogeneous data, establish and optimize event tree models, and assess customers’ credit risks in real time. This method not only improves data processing efficiency, but also improves the accuracy of risk assessment, providing strong support for financial decision-making.

[0142] The present invention also provides a data development platform based on multi-source data integration, which is used to execute the steps of the data development method based on multi-source data integration in the above embodiment, including: a data collection module: used to collect multi-source heterogeneous data from different sources, and pre-process these data to form multi-source initial data;

[0143] Data integration module: Based on a unified identifier and data model, the pre-processed multi-source initial data is integrated in a multi-modal manner, and initial features of different data sources are extracted to form a preliminary feature set;

[0144] Event tree construction module: constructs an event tree based on the collected multi-source heterogeneous data and its preliminary feature set. Each node in the event tree represents different types of data features and is used to analyze the path in the event tree;

[0145] Event tree optimization module: used to eliminate redundant nodes and paths in the event tree, identify target paths and nodes, and thus obtain an optimized event tree;

[0146] Real-time data mining module: Based on the optimized event tree, it mines the multi-source data collected in real time, analyzes the data and predicts risks through the risk assessment model;

[0147] Risk assessment module: used to analyze multi-source data collected in real time, make risk predictions through the established risk assessment model, and compare and analyze the prediction results with actual risk events;

[0148] User interface module: provides a user interface for data access and operation, enabling users to view and operate data analysis results;

[0149] Data storage module: used to store multi-source heterogeneous data, preliminary feature sets, event trees and optimized event trees, real-time collected data, risk assessment models and analysis results.

[0150] In order to verify the performance of the method and platform described in the present invention, the following experiments were performed for comparison.

[0151] (1) Multi-source data integration efficiency test;

[0152] Experimental background: With the explosive growth of data volume and the diversification of data types, the integration efficiency of multi-source heterogeneous data has become a key factor restricting the development of data processing technology. In many fields such as finance, medical care, and transportation, how to efficiently integrate data from different sources and formats is of great significance for the further analysis and application of data. In this context, this experiment aims to verify the efficiency of a data development method based on multi-source data integration through experiments. This method collects multi-source heterogeneous data and performs preprocessing, and performs multimodal integration based on a unified identifier and data model, in order to achieve the purpose of improving data processing efficiency and accuracy.

[0153] Experimental Method: This experiment selected three representative data sources, including bank transaction records, social media data, and sensor data. These data sources cover structured, semi-structured, and unstructured data types. In the experiment, a total of 10 million data records were collected. These data records are not only huge in number, but also contain complex data relationships and diverse data formats. The method used in the experiment is based on a multimodal integration strategy of unified identifiers and data models to preprocess and integrate these data. First, the collected raw data is cleaned and standardized to eliminate noise and inconsistency in the data. Then, by building a unified data model, the data records in different data sources are mapped to a unified identifier, which is the key step of data integration. During the integration process, special attention is paid to the consistency and integrity of the data to ensure that no key information is lost during the integration process. In addition, a weighted average fusion rule is used to handle data conflicts. A weight is assigned to each data source based on the credibility of the data source, and the weighted average of the conflicting data is calculated to obtain a more accurate data integration result.

[0154] Result analysis: Figure 5 As shown, Figure 5(a) is a comparison of the time consumption for data integration. The integration process using the method of the present invention only takes 15 minutes, which is twice as efficient as the 30 minutes of the traditional method. This significant improvement is largely due to the efficient data preprocessing and multimodal integration strategy in the method. The consistency of the integrated data set reached 99.9%, indicating that there was almost no data loss or error during the integration process. In order to further verify the accuracy of the integration results, a detailed consistency test was performed. Specifically, it was tested whether the records in the integrated data set corresponded one-to-one with the records in the original data source, and whether the integrity and accuracy of the data were maintained. Figure 5 (b) in the figure shows the trend of integration consistency. The test results show that the matching rate between the records in the integrated data set and the original data source reached 99.87%, and the error rate was only 0.13%. Further statistical analysis shows that among the 10 million data records, only 12,900 records were erroneous or lost during the integration process. This result fully demonstrates the efficiency and accuracy of the method in processing large-scale heterogeneous data. In addition, the performance of the integrated data set was evaluated, including query response time and data throughput. Figure 5 (c) in the figure is the query response time comparison. Figure 5 (d) in the figure is a comparison of data throughput. In terms of query response time, the average response time is 0.15 seconds, which is much lower than the 0.3 seconds of the traditional method; in terms of data throughput, the method of the present invention can process 150,000 records per second, which is twice that of the traditional method. These results not only verify the advantages of the method of the present invention in data integration efficiency, but also demonstrate its potential and value in practical applications.

[0155] (2) Feature extraction accuracy test;

[0156] Experimental background: In the era of big data, feature extraction is a key step in data analysis, and its accuracy directly affects the effect of subsequent data mining and pattern recognition. The accuracy of feature extraction not only involves the reduction of data dimension, but also the retention of key information in the data. Therefore, evaluating the accuracy of feature extraction is crucial to ensure the reliability of data analysis results. This experiment aims to verify the consistency and accuracy between the feature set extracted from multi-source heterogeneous data sets and the known benchmark features through comparative analysis, so as to evaluate the accuracy of the feature extraction process.

[0157] Experimental Method: In the experiment, time features, statistical features, and frequency domain features were first extracted from the integrated data set. Time features include the timestamp and time interval of the data, statistical features cover the mean, variance, skewness and other statistics of the data, and frequency domain features include frequency domain analysis results such as Fourier transform. The extraction of these features is based on the preliminary feature set of the data set, which is achieved through a series of mathematical transformations and statistical analysis methods. The extracted feature set is then compared with the predetermined benchmark features, which are obtained through expert knowledge and historical data analysis and are considered to be the key and representative features in the data set. During the comparison process, a variety of similarity metrics are used, including cosine similarity, Euclidean distance, and Jaccard similarity coefficient, to comprehensively evaluate the accuracy of feature extraction.

[0158] Result analysis: The experimental results are as follows Figure 6 As shown, Figure 6 (a) in the figure is the feature matching comparison. Figure 6 (b) in the figure is the extraction error of different feature types. Figure 6 (c) in the figure shows the influence of similarity measurement method on feature matching. Figure 6 (d) in Fig. 1 is the effect of noise on feature extraction error. The matching degree between the feature set extracted from the integrated dataset and the benchmark features reached 95%, which is significantly higher than the expected matching degree of randomly selected features, verifying the efficiency and accuracy of the feature extraction process. Specifically, the matching degree of temporal features is 94.7%, the matching degree of statistical features is 95.3%, and the matching degree of frequency domain features is 94.9%. These results show that the proposed feature extraction method can effectively capture the key information in the data and extract it with high accuracy. The feature extraction error is within 5%, which is within the acceptable range. Considering the complexity and diversity of the data, this error level further proves the robustness of the feature extraction method. In order to further quantify the accuracy of feature extraction, a detailed error analysis is performed. In 1000 samples, the extraction error of temporal features is 4.8%, the extraction error of statistical features is 4.5%, and the extraction error of frequency domain features is 5.1%. These specific numerical results not only show the performance of the feature extraction method on different types of features, but also reveal the possible deviations and deficiencies in the feature extraction process. In addition, through comparative analysis, it was found that the main errors in the feature extraction process came from noise and outliers in the data, which were effectively controlled and reduced through subsequent data cleaning and preprocessing steps. Overall, the results of this experiment provide strong evidence for the effectiveness and reliability of the feature extraction method and lay a solid foundation for subsequent data analysis and application.

[0159] (3) Event tree construction and optimization effect testing;

[0160] Experimental background: In the field of risk management of complex systems, event trees, as a graphical risk analysis tool, are widely used in risk identification, assessment and decision support. Event trees start from the initial event and gradually expand possible subsequent events until the final result, thereby revealing the causal relationship between risk factors. However, the constructed event tree may have redundant paths and nodes, which increases the complexity and uncertainty of the analysis results. Therefore, the optimization of event trees is of great significance for improving the accuracy and efficiency of data analysis. This experiment aims to evaluate the impact of event tree construction and optimization modules on data analysis accuracy through comparative analysis, so as to verify the effectiveness of the optimization strategy.

[0161] Experimental method: In the experiment, we first conducted a risk assessment on 1,000 data records, which covered a variety of risk factors and event results. On this basis, we constructed an initial event tree, including the definition of root nodes, branches and nodes, as well as the conditional relationships and transfer rules between nodes. After the initial event tree was constructed, an optimization algorithm was used to optimize the event tree, including steps such as identifying and eliminating redundant nodes and paths, merging duplicate nodes, and retaining critical paths. The optimization algorithm is based on indicators such as the frequency of nodes and paths, feature similarity, and path contribution, and is implemented through methods such as feature clustering, shortest path algorithm, and Gini coefficient screening. The optimized event tree was used to conduct risk assessment on the same data set again to compare the risk assessment results before and after optimization.

[0162] Result analysis: The experimental results are as follows Figure 7 As shown, Figure 7 (a) is the comparison of event tree accuracy before and after optimization. Figure 7 (b) is the comparison of risk event identification before and after optimization. Figure 7 (c) in the figure is the optimization comparison between the number of critical path nodes and the path length. Figure 7(d) in the figure is a comparison of data processing time before and after optimization. The accuracy of the event tree in risk assessment before optimization was 80%, while the optimized event tree increased the accuracy to 85%, significantly improving the accuracy of risk assessment. Specifically, in the risk assessment of 1,000 data records, the event tree before optimization identified 800 risk events, of which 640 events were correctly identified and 160 events were incorrectly identified. After optimization, the event tree identified 765 risk events, of which 652 events were correctly identified and 113 events were incorrectly identified. This result shows that by optimizing the event tree, not only the accuracy of risk assessment is improved, but also the number of false positives and negatives is reduced. Further analysis found that the number of nodes on the critical path of the optimized event tree was reduced by 15% and the path length was shortened by 20%, which shows that the optimization strategy effectively removes redundant information and improves the simplicity and interpretability of the event tree. In addition, the optimized event tree reduced the analysis time by 25% when processing the same number of data records, from an average of 3.5 minutes to 2.6 minutes. This result verifies the effectiveness of the optimization strategy in improving data analysis efficiency. In summary, the results of this experiment provide strong evidence for the design and application of the event tree optimization module and provide an effective data analysis tool for the risk management field.

[0163] (4) Real-time data mining and risk assessment response speed testing;

[0164] Experimental background: In the era of big data, real-time data mining and risk assessment have become the focus of attention in many fields, especially in the fields of financial transaction monitoring, network security, and industrial process control. The ability to respond to data in real time is directly related to the effectiveness of risk management and the timeliness of decision-making. As a key component for processing streaming data, the response speed and real-time performance of the real-time data mining module are important indicators for measuring system performance. Therefore, this experiment aims to verify the response speed of the real-time data mining module to newly collected data and the real-time performance of risk assessment, so as to evaluate the performance of the system in practical applications.

[0165] Experimental method: In the experimental design, 10,000 data records collected in real time were selected as test samples. These data simulated the characteristics of streaming data in the real environment, including the randomness, timing and dynamic variability of data. In the experiment, the real-time data mining module was deployed on a high-performance computing platform to ensure sufficient computing resources. The input of the module is the real-time collected data stream, and the output is the risk assessment result. During the experiment, the timestamp recording method was used to accurately measure the module's response time to each piece of data, that is, the time interval from data input to processing result output. At the same time, the update delay of the risk assessment results was measured to evaluate the system's ability to update risk assessment results in real time. In the experiment, the risk assessment model was trained based on historical data and integrated into the real-time data mining module to achieve instant risk scoring of new data.

[0166] Result analysis: The experimental results are as follows Figure 8 As shown, Figure 8 (a) is the response time comparison between real-time data mining and batch processing methods. Figure 8 (b) in the figure is the update delay of the real-time risk assessment results. Figure 8 (c) in the figure is the processing stability under the influence of data flow rate. Figure 8 (d) in the figure is a comparison of the response time of common events and high-risk events. The average response time of the real-time data mining module is 2 seconds per data record, which is significantly better than the traditional batch processing method, which usually takes minutes or even longer to process the same amount of data. Specifically, in the test of 10,000 data records, the module was able to complete data mining and risk assessment within 2 seconds in 95% of cases, and the response time did not exceed 3 seconds in 99% of cases. In addition, the update delay test of the risk assessment results showed that the average delay of the result update was 3.5 seconds, and the update delay did not exceed 5 seconds in 90% of cases, meeting the needs of real-time monitoring. When the data flow rate is 10 records per second, the module can run stably without processing bottlenecks. Further statistical analysis shows that the module's response time when processing high-risk events increases slightly, averaging 2.5 seconds, but in risk management practice, this increased delay is acceptable because high-risk events require more complex analysis and verification processes. Through the analysis of the internal processing flow of the module, it is found that data preprocessing and feature extraction are the key steps that affect the response time. Optimizing these two steps can further improve the response speed of the system. In summary, the results of this experiment prove the superior performance of the real-time data mining module in terms of processing speed and real-time performance, providing strong technical support for real-time risk management.

[0167] (5) User interface operation convenience test;

[0168] Experimental background: In the field of information technology, user interface is an important bridge for human-computer interaction, and its design quality directly affects the user experience and work efficiency. An intuitive and easy-to-use user interface can significantly improve the user's operational convenience, reduce operational difficulty, and improve work efficiency. Especially in data-intensive applications, the design of the user interface is even more important, as it determines how users interact with the system and how to efficiently complete data access and operation tasks. In view of this, this experiment aims to evaluate the usability and ease of operation of the user interface module, and verify the effectiveness of the user interface design through quantitative and qualitative analysis.

[0169] Experimental method: In the experimental design, 20 test users were invited to participate in the user interface operation convenience test. These users included individuals with different backgrounds and experience levels to ensure the representativeness and breadth of the test results. The test users were required to complete a series of predefined data access and operation tasks through the user interface. These tasks covered common operations such as data query, editing, analysis and reporting. During the experiment, the average operation time required for each user to complete each task was recorded, and the error rate during the operation was counted. In addition, after the experiment, feedback on user satisfaction with the user interface was collected through a questionnaire survey, which included evaluations of the user interface's intuitiveness, ease of use, functionality and response speed.

[0170] Result analysis: Experimental results Fig. 9 As shown, Fig. 9 (a) is the average operation time distribution of users completing tasks. Fig. 9 (b) shows the decreasing trend of user error rate with the usage stage. Fig. 9 (c) in the figure is the user satisfaction survey score. Fig. 9(d) in the figure is the distribution of user satisfaction feedback. The average operation time for the test users to complete each task is 3 minutes / task, which is a relatively good performance in the field of user interface design. Specifically, among the 20 test users, 60% of the users completed the task within 2 minutes, 30% of the users completed the task within 3 minutes, and only 10% of the users took more than 3 minutes. This distribution shows the efficiency of the user interface in operation. In terms of error rate, the total number of errors made by all users during the operation was 36. Considering that a total of 400 operations (20 users × 20 tasks) were performed, the error rate was 9%, which is much lower than the industry average. Further analysis found that the errors were mainly concentrated in the first use of new users. As they became familiar with the user interface, the error rate dropped significantly. The user satisfaction survey results showed that 90% of users were satisfied with the overall design of the user interface, of which 75% of users believed that the user interface was intuitive and easy to use, 15% of users believed that the user interface was comprehensive, and only 5% of users made suggestions for improvement. In terms of specific values, the intuitiveness of the user interface scored 4.2 points (out of 5 points), the usability score was 4.1 points, the functionality score was 4.0 points, and the response speed score was 4.3 points. These scores are all higher than the industry average, indicating that the user interface has a high user satisfaction in design. Combining the operation time and error rate, as well as the feedback on user satisfaction, it can be concluded that the user interface is reasonably designed, easy to operate, and can meet the user's data access and operation needs. Future work can further optimize the user interface, especially in reducing new user operation errors and improving functionality, so as to further improve user satisfaction and operation efficiency.

[0171] The above description is only a preferred embodiment of the present invention and is not intended to be a further limitation of the present invention. All equivalent changes made using the contents of the present specification and drawings are within the protection scope of the present invention.

Claims

1. A data development method based on multi-source data integration, characterized in that: The following steps are involved: S10, collecting multi-source heterogeneous data and preprocessing them to obtain multi-source initial data; S20, based on a unified identifier and data model, perform multimodal integration on the multi-source initial data, and extract initial features of different data sources from the multimodally integrated multi-source initial data to form a preliminary feature set; specifically: Determine an identifier that uniquely identifies each entity according to the time format of the initial data set, wherein the identifier includes an ID and a timestamp; The unified identifiers of the initial data set are aligned according to the constructed unified data model, and the weights of the conflicting data are calculated according to the fusion rule of data averaging; Integrate the aligned data according to the unified data model, use the data averaging fusion rule to eliminate conflicting data, and generate an integrated data set; Extract temporal features, statistical features, and frequency domain features from the integrated data set; The extracted time features, statistical features and frequency domain features are transformed, the feature values ​​are converted into standard normal distribution, and the features extracted from different data sources are spliced ​​to form a preliminary feature set; S30, establishing an event tree based on the collected multi-source heterogeneous data and the corresponding preliminary feature set, wherein each node in the event tree is defined to represent different types of data features, and the path in the event tree is analyzed; S40, eliminating redundant nodes and paths in the event tree, identifying target paths and nodes in the event tree, and obtaining an optimized event tree; S50. Mining the multi-source data collected in real time based on the optimized event tree, analyzing the multi-source data collected in real time according to the established risk assessment model, comparing and analyzing the risk prediction results with the actual risk events, and accessing and operating the data through the user interface.

2. The data development method based on multi-source data integration according to claim 1, characterized in that: S10 is as follows: Collect multi-source heterogeneous data based on the determined data sources; Convert the data to a standard normal distribution and change it to a unified time format; The collected multi-source heterogeneous data are merged according to the time format and organized into a unified initial data set.

3. The data development method based on multi-source data integration according to claim 1, characterized in that: The conflicting data is weighted and timed according to the fusion rule of data averaging, including the following steps: Traverse all data records and mark records with the same identifier but different values ​​as conflicting data based on the identifier of each entity; Assign an initial weight to each data source based on its credibility, use weighted average to handle conflicting data, and calculate the weighted average for conflicting data with the same identifier; Use a sliding window to adjust the weights based on the historical performance of the data source; the weighted average is: ， In the formula, is the data value after weighted average, For the The value of the data source, For data source The corresponding data source weight, is the number of data sources; the sum of the weights of each data source is 1, that is: .

4. The data development method based on multi-source data integration according to claim 1, characterized in that: When the extracted time features, statistical features and frequency domain features are transformed, it includes: Transform the eigenvalues ​​to a standard normal distribution; Features extracted from different data sources are concatenated into a long feature vector in a unified order; in, data sources, and the feature vectors extracted from each are , then the concatenated feature vector is: ， In the formula, is the complete feature vector after concatenation, For data source Extracted feature vectors.

5. The data development method based on multi-source data integration according to claim 4 is characterized in that: S30 is specifically: Define the root node, branches and nodes of the event tree based on multi-source heterogeneous data and the extracted preliminary feature set; Starting from the root node, add nodes layer by layer, define the conditional relationships and transfer rules between nodes, and establish the relationships between nodes; Define the data features of classification feature nodes, continuous feature nodes, time feature nodes and space feature nodes in the event tree, analyze and determine the path from the root node to the leaf node in the event tree, and construct the event tree.

6. The data development method based on multi-source data integration according to claim 5 is characterized in that: S40 is specifically: Identify redundant nodes and paths by analyzing the frequency, feature similarity, and path contribution of nodes and paths; Merge duplicate nodes through feature clustering and retain key paths through the shortest path; Screen according to the Gini coefficient to identify target paths and nodes; After removing redundant nodes and paths, the optimized event tree is rebuilt.

7. The data development method based on multi-source data integration according to claim 6 is characterized in that: When screening based on the Gini coefficient, the path is calculated using the Gini coefficient, where: ， In the formula, For path The Gini coefficient of For path Middle The probability of a class.

8. Data development platform based on multi-source data integration, characterized by: The platform is used to execute the data development method based on multi-source data integration as described in any one of claims 1 to 7, and comprises: Data collection module: used to collect multi-source heterogeneous data from different sources and pre-process these data to form multi-source initial data; Data integration module: Based on a unified identifier and data model, the pre-processed multi-source initial data is integrated in a multi-modal manner, and initial features of different data sources are extracted to form a preliminary feature set; Event tree construction module: constructs an event tree based on the collected multi-source heterogeneous data and its preliminary feature set. Each node in the event tree represents different types of data features and is used to analyze the path in the event tree; Event tree optimization module: used to eliminate redundant nodes and paths in the event tree, identify target paths and nodes, and thus obtain an optimized event tree; Real-time data mining module: Based on the optimized event tree, it mines the multi-source data collected in real time, analyzes the data and predicts risks through the risk assessment model; Risk assessment module: used to analyze multi-source data collected in real time, make risk predictions through the established risk assessment model, and compare and analyze the prediction results with actual risk events; User interface module: provides a user interface for data access and operation, enabling users to view and operate data analysis results; Data storage module: used to store multi-source heterogeneous data, preliminary feature sets, event trees and optimized event trees, real-time collected data, risk assessment models and analysis results.

Images