Multi-source data fusion value quantification method based on event record log

Abstract

The invention relates to the technical field of data processing, and discloses a multi-source data fusion value quantification method based on event record logs, which comprises the following steps: setting event recorders in a plurality of processing nodes of a data fusion process to capture and generate event logs; based on the event log, constructing a data source independent value evaluation model, and calculating to obtain a basic value of each data source; calculating the overall value of the fusion data set, and determining the collaborative value of data fusion according to the sum of the overall value and the basic value; constructing an anti-fact value evaluation model, and calculating the comprehensive contribution degree of each data source in fusion; and finally, distributing the collaborative value according to the comprehensive contribution degree. According to the method, contribution degree tracing is carried out by constructing the distinguishing and quantifying model and combining anti-fact evaluation, so that objectiveness and accuracy of fusion gain are realized, and the problem that distribution is not public due to unclear value composition and difficult contribution definition in multi-source data collaboration is solved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to a method for quantifying the value of multi-source data fusion based on event logs. Background Technology

[0002] Multi-source data fusion refers to the technical process of comprehensively processing information from different data sources to generate new datasets with higher value and usability. Data value quantification, on the other hand, is the assessment and calculation of the utility or economic contribution of data resources as a factor of production, and is the foundation for realizing the operation and market circulation of data assets.

[0003] Among existing data valuation techniques, some methods focus on the cost accounting or market transaction price of the data source; others assess the value based on the application effect of the data in a specific business scenario. In scenarios involving multi-source data fusion, the evaluation often focuses on the overall performance improvement of the fused dataset, such as measuring its overall value by analyzing the performance of the fused data in a specific model or decision support system.

[0004] With the increasing demand for market-based allocation and collaborative sharing of data elements, there is an urgent need to accurately identify the individual contributions during data fusion. However, existing technologies, when assessing the value of fusion, generally struggle to separate the independent value of data sources from the collaborative value of fusion, and lack objective, quantifiable mechanisms to trace and allocate the collaborative gains generated by fusion. This lack of an assessment mechanism means that in multi-party data collaboration, the determination of contributions often relies on subjective negotiation, making it difficult to achieve fair value distribution. This not only hinders the smooth flow of data elements but also discourages data holders from participating in data fusion and sharing. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for quantifying the value of multi-source data fusion based on event logs. This method aims to solve the problems of difficulty in defining independent and collaborative values ​​in data fusion, as well as the lack of objective quantitative tracking and allocation mechanisms for collaborative gains.

[0006] To achieve the above objectives, the present invention provides the following technical solution: The first aspect of the present invention provides a method for quantifying the value of multi-source data fusion based on event logs, which can be executed by a data value quantification system.

[0007] In one embodiment, the method includes:

[0008] Step 1: Capturing and Generating Event Logs

[0009] Event loggers are set up at multiple processing nodes in the data fusion process to capture and generate event logs. The data fusion process may include nodes such as data access, data preprocessing, data transformation, data fusion, and data application. The event logs objectively record the processing and usage of data throughout the entire lifecycle of the fused application.

[0010] Specifically, the event logger can be configured using Aspect-Oriented Programming (AOP) technology. In this approach, event logger annotations and event logger aspects can be defined. The event logger aspects can then intercept business methods (such as a data fusion method or a data application method) marked by the event logger annotations.

[0011] When a business method is invoked, the event logging aspect automatically captures the method's context information, such as the input data source identifier, processing time, and operation type, and generates an event log in a predefined format, thus achieving non-intrusive recording of the business method.

[0012] Step Two: Calculation of Basic Value

[0013] Based on the event logs, an independent value assessment model for each data source is constructed to calculate the fundamental value of each data source. This fundamental value reflects the independent utility of the data source before it participates in the integration process.

[0014] Specifically, the construction of the data source independent value assessment model includes:

[0015] Based on the event logs, four metrics were calculated: usage frequency, processing complexity, business relevance, and timeliness value.

[0016] In one embodiment, the calculation uses a frequency metric, specifically including:

[0017] By analyzing the data application events in the event log, the scenario weight of the application scenario, the query complexity coefficient of the data query, and the data coverage ratio of the data source in the query are obtained. At the same time, by reconstructing the data usage path, the node processing quality coefficient in the data flow is obtained. Combining the scenario weight, the query complexity coefficient, the data coverage ratio, and the node processing quality coefficient, the usage frequency index is calculated.

[0018] In one embodiment, the computational complexity index specifically includes: obtaining the execution time, computational resource consumption coefficient, and data scale of each processing step by analyzing data preprocessing events, data transformation events, and data fusion events in the event log; and calculating the processing complexity index based on the execution time, the computational resource consumption coefficient, and the data scale.

[0019] The basic value is calculated by combining the value functions of the usage frequency indicator, the processing complexity indicator, the business relevance indicator, and the timeliness value indicator. For example, the basic value can be determined by a weighted sum of the four indicators, where the weight of each indicator is preset according to business needs.

[0020] Step 3: Determining the Collaborative Value

[0021] Based on the event logs, the overall value of the fused dataset is calculated, and the synergistic value of the data fusion is determined based on the sum of the overall value and the basic value.

[0022] In one specific implementation, the collaborative value is determined by subtracting the sum of the base values ​​of all data sources from the overall value of the merged dataset. The collaborative value represents the incremental value generated by the merging of the data sources.

[0023] Specifically, the overall value of the computationally fused dataset includes: based on the event log, quantifying information integrity improvement indicators, data quality improvement indicators, and decision support enhancement indicators, and using these three indicators as fusion performance indicators.

[0024] The independent value assessment model of the data source (i.e., the model framework in step two) is reused, and the fusion performance index is used to adjust the dimensional indicators in the independent value assessment model of the data source (for example, the fusion performance index can be used as an adjustment factor to adjust the calculation results of the dimensional indicators) to calculate the overall value of the fused dataset.

[0025] Step Four: Counterfactual Value Assessment and Contribution Calculation

[0026] A counterfactual value assessment model is constructed, and the counterfactual value is calculated by simulating a scenario of removing data sources. Based on the counterfactual value, the comprehensive contribution of each data source is calculated.

[0027] Specifically, constructing the counterfactual value assessment model includes: generating a simulated event log by excluding all relevant events from a specific data source from the event log; reusing the data source independent value assessment model (step two) for the simulated event log to calculate the remaining basic value of the simulated dataset; and reusing the step of calculating the overall value of the fused dataset (step three) for the simulated event log to calculate the counterfactual value.

[0028] Specifically, the calculation of the overall contribution of each data source includes:

[0029] (a) Calculate the removal impact index. The removal impact index is calculated by dividing the difference between the overall value and the counterfactual value by the overall value.

[0030] (b) Calculate the marginal contribution coefficient. The marginal contribution coefficient is obtained by dividing the value increment of the data source by the base value of the data source by the value increment of the data source. This step can be achieved by sampling different data subsets multiple times and calculating the value change before and after the addition of a specific data source.

[0031] (c) The removal impact index and the marginal contribution coefficient are weighted and summed, and then normalized to obtain the comprehensive contribution.

[0032] Step 5: Allocation of Individual Gain Value

[0033] Based on the overall contribution and the collaborative value, the collaborative value is allocated to obtain the individual gain value of each data source.

[0034] In one specific implementation, the individual gain value is obtained by proportionally allocating the collaborative value based on the overall contribution.

[0035] In an optional embodiment, the method further includes adding the individual gain value to the base value to obtain the final attribution value of each data source.

[0036] Second aspect of the invention: System embodiments

[0037] A second aspect of the present invention provides a multi-source data fusion value quantification system based on event logs, the system being used to execute the above-described method. The system includes:

[0038] The event logging module is used to set up event loggers in multiple processing nodes of the data fusion process to capture and generate event logs.

[0039] The independent value assessment module is used to construct an independent value assessment model for the data sources based on the event logs, and to calculate the basic value of each data source.

[0040] The collaborative value calculation module is used to calculate the overall value of the fused dataset based on the event logs, and determine the collaborative value of data fusion based on the sum of the overall value and the basic value.

[0041] The value tracing module is used to construct a counterfactual value assessment model, calculate the counterfactual value by simulating a scenario of removing data sources, and calculate the comprehensive contribution of each data source based on the counterfactual value.

[0042] The value allocation module is used to allocate the collaborative value based on the overall contribution and the collaborative value, thereby obtaining the individual gain value of each data source.

[0043] In the system embodiment, the specific implementation methods, internal logic, and calculation processes of each module can be referred to the corresponding step descriptions in the method embodiment of the first aspect of the present invention (such as steps one to five and their specific limitations), and will not be repeated here. The system can be implemented by hardware, software, or firmware, or any combination thereof.

[0044] This invention provides a method for quantifying the value of multi-source data fusion based on event logs. It has the following beneficial effects:

[0045] 1. This invention achieves non-intrusive recording of data processing and usage behavior throughout the entire fusion lifecycle by setting event loggers at multiple processing nodes in the data fusion process and using aspect-oriented programming (AOP) to capture event logs. This approach provides an objective and traceable data foundation for subsequent value quantification, overcoming the inaccuracies caused by unclear data sources or reliance on subjective assessments in traditional value evaluation.

[0046] 2. This invention calculates the basic value of each data source by constructing an independent value assessment model, and then calculates the overall value of the fused dataset based on this model, thereby determining the synergistic value of data fusion. This differentiated processing approach can clearly quantify the independent utility of the data sources and the incremental utility generated by fusion, making the composition and source of value clearly defined.

[0047] 3. This invention constructs a counterfactual value assessment model, which calculates the comprehensive contribution of a specific data source by simulating a scenario where that data source is removed, and allocates collaborative value based on this calculation. This method solves the problem of accurately defining the contribution of each data source in a multi-source fusion scenario by quantifying the impact of the absence of a specific data source on the overall value of the fused dataset, thus providing a technical path for the reasonable allocation of collaborative gains. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0049] Figure 2 This is a schematic diagram of the embedded event logging mechanism of the present invention;

[0050] Figure 3 This is a system architecture diagram of the present invention. Detailed Implementation

[0051] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] Please see Figure 1-2 This invention provides a method for quantifying the value of multi-source data fusion based on event logs, comprising the following steps:

[0053] S1. Design a data fusion process with an embedded event logging mechanism. Set up event loggers at all process nodes such as data access, preprocessing, transformation, fusion and application to capture and generate objective and traceable event logs.

[0054] S2. Based on event logs, process mining techniques are used to preprocess and standardize the data, reconstruct the data usage path, and build an independent value quantification model for each data source to calculate the basic value of each data source.

[0055] S3. By using the independent value quantification model and the basic value of calculation, construct a multi-dimensional integrated data value assessment model, and combine it with the analysis of joint usage patterns in event logs to quantify the collaborative value generated by data interaction and calculate the overall value gain brought by integration.

[0056] S4. Based on the overall value gain and multi-dimensional fusion data value assessment model, a value tracing mechanism is designed using a counterfactual causal reasoning framework. By calculating the removal impact index and marginal contribution coefficient of each data source, the value gain is fairly attributed to each participating data source.

[0057] The technical implementation details of each of the above steps will be described in detail below with reference to specific embodiments of the present invention.

[0058] Step S1 involves designing a data fusion process with an embedded event logging mechanism. Event loggers are set up at each node of the entire process, including data access, preprocessing, transformation, fusion, and application, to capture and generate objective and traceable event logs. The specific implementation of step S1 includes sub-steps S1.1, S1.2, and S1.3.

[0059] S1.1 Establish a standardized data fusion process.

[0060] This process establishes a unified data access layer. To achieve system decoupling, this access layer can employ message queue technology, such as Kafka or RabbitMQ, to establish data channels. For messages with different protocol formats, such as HTTP, MQTT, and TCP / IP, dedicated data receiving adapters are configured for protocol parsing and data extraction. During data access, a unique identifier is assigned to each data source, and its metadata information is recorded in detail, including data source, access time, and data structure description.

[0061] The incoming data enters the preprocessing stage. This stage utilizes a rule engine to automatically detect and handle missing values, outliers, and duplicate records. Data standardization then occurs, unifying data from different sources (e.g., timestamps in different formats) into a standard format and standardizing units (e.g., converting different pressure units to Pa) to ensure all data uses consistent units of measurement. Finally, an encoding mapping table is established between different system codes and IDs to resolve data alignment issues across heterogeneous systems.

[0062] The preprocessed data is fused using a multi-model collaborative strategy. For structured data, one approach is to use a feature-based matching method to extract common attributes between data, such as "device ID" or "geolocation code," as fusion keys to establish relationships between the data.

[0063] For semi-structured data, such as JSON or XML, and unstructured data, such as text documents, semantic analysis techniques, such as Natural Language Processing (NLP), are used to extract key entities and relationships, which are then mapped to structured data. For data with strong temporal and spatial correlations, specialized processing is applied. For example, interpolation or resampling is used to process time-series data of different frequencies to address synchronization issues in the time dimension, and geohashing or spatial indexing is used for location association identification in the spatial dimension. For data that conflicts during the fusion process, the system can retain multiple iterative versions of the original data and record in detail the judgment rules and decision evidence used during the fusion process, thus ensuring the traceability of the fusion process and the final result. The fused data is ultimately provided to the outside world through a unified data service layer.

[0064] S1.2 incorporates an event logging mechanism within the data fusion process of S1.1.

[0065] Specifically, event loggers are set up at each key processing node in the data fusion process (such as data access adapters, data cleaning engines, fusion strategy executors, etc.) to asynchronously record the entire data processing process without interrupting business operations.

[0066] The event logger captures at least the following event types:

[0067] 1) Data access events: Record data source access time, data source ID, data volume, data type, etc.;

[0068] 2) Data preprocessing events: Record operations such as cleaning, deduplication, and completion, as well as the specific rules and parameters used;

[0069] 3) Data conversion events: Operations such as record format conversion, unit unification, and encoding standardization;

[0070] 4) Data fusion events: Record the identifiers of the datasets involved in the fusion, the fusion rules used, and the results of conflict resolution, etc.

[0071] 5) Data application events: Record the subsequent use of data by downstream systems, such as querying, analyzing, and displaying it.

[0072] All captured events follow a standardized event log data structure. This structure defines multiple fields to comprehensively describe the context of the event.

[0073] Specifically, the event log data structure includes: a field for uniquely identifying the event; a field for recording the precise time the event occurred; a field for identifying the event type (such as access, preprocessing, etc.); a field for identifying one or more data sources involved in the event; a field for identifying the data object processed by the event; a field for describing the specific operation performed; a field for recording the parameters required for the operation in a format such as JSON; and a field for recording the measurement information of the results produced by the operation (such as the number of data rows processed).

[0074] S1.3 implements the event logging mechanism of S1.2.

[0075] The present invention preferably employs aspect-oriented programming (AOP) technology to achieve automatic event recording without intruding on the original data fusion process business logique code.

[0076] The specific implementation is as follows: First, define a custom event logging annotation. This annotation can be used to mark business methods in the data fusion process that need to have events logged. This annotation can carry multiple metadata parameters, such as: parameters to specify the event type, parameters to define the log level, parameters to specify the list of parameter names to be captured from the method input parameters, and boolean parameters to specify whether to capture the method return value.

[0077] Secondly, define an event logging aspect. This aspect uses a pointcut to specifically intercept all business methods marked by the aforementioned event logging annotations. This aspect is preferably executed using around-notification logic.

[0078] Specifically, before the target business method is executed, the surrounding notification logic first records a "start" event; then, it calls and executes the target business method; if the target method executes successfully, the logic captures its return value (if the annotation specifies that the return value needs to be captured) and uses it as the field value of the metric information used to record the operation result in S1.2, recording a "success" event; if the target method throws an exception during execution, the logic captures the exception information and records an "exception" event.

[0079] This implementation method, through its AOP approach, enables the automated and minimally impactful capture of events without intruding on the original data fusion process's business logic code, providing an objective and complete data foundation for the subsequent value quantification of S2.

[0080] Step S2 involves preprocessing and standardizing the event logs using process mining techniques, reconstructing the data usage path, and building an independent value quantification model for each data source to calculate its fundamental value. Step S2 is implemented through sub-steps S2.1 and S2.2.

[0081] S2.1 preprocesses and standardizes the event logs generated in S1.

[0082] The purpose of step S2.1 is to convert the raw logs recorded in the event log data structure of S1.2 into a standard format required for process mining analysis, laying the foundation for subsequent process refactoring and value analysis. This conversion process includes:

[0083] First, extract the field from S1.2 that identifies the data object being processed for the event, and use it as a case identifier to associate different events in the same processing flow.

[0084] Secondly, extract the fields from S1.2 that identify the event type or describe the specific operation performed, and use them as the activity name.

[0085] Next, the field in S1.2 used to record the precise time of the event is extracted and standardized as a timestamp, which is used to determine the event sequence and calculate time-related metrics.

[0086] Then, extract the fields from S1.2 that identify one or more data sources involved in the event, and use them as resources to record which data sources the event is associated with.

[0087] Finally, the fields in S1.2 used to record the parameters required for the operation and the fields used to record the measurement information generated by the operation are taken as additional attributes of the event.

[0088] After the above extraction is completed, this step further sorts all events under each case identifier by timestamp to form a complete and ordered event sequence, which reconstructs the data usage path for analysis by the model in S2.2.

[0089] Based on the standardized and ordered event sequence in S2.1, S2.2 uses the process analysis results to construct an independent value assessment model for the data source.

[0090] The data source independent value assessment model aims to quantify the fundamental value of data sources before integration using multi-dimensional indicators. This model can be represented by a holistic data value function V(D). This function is multi-dimensional and can be specifically expressed as:

[0091] V(D)=f(U f (D),P c (D),B r (D),T v (D));

[0092] In the formula, V(D) represents the overall data value of data source D; D represents the data source; f is the value function calculated by combining indicators of various dimensions, such as weighted summation or a more complex nonlinear function; U f (D) is the frequency of use indicator; P c (D) is a metric for processing complexity; B r (D) is a business relevance indicator; T v (D) is the timeliness value indicator.

[0093] The four key dimensions of this expression are explained below:

[0094] 1) Use frequency index U f (D) measures the breadth and depth of data usage. This invention calculates this metric by analyzing the standardized event sequences in S2.1, particularly the frequency, associated attributes, and reconstructed data usage paths of "data application events." The specific calculation formula for this metric is as follows:

[0095]

[0096] In the formula, U f (D) represents the frequency of data source D; n represents the total number of times the data is used, which can be calculated based on the number of specific "data application events" in the event log; i represents the index of data usage; w i This is the scenario weight for the i-th use, which can be pre-set according to the importance of the business scenario (such as core business or auxiliary business); q i The query complexity coefficient can be determined based on the query parameter complexity recorded in the event's attached attributes; d iFor data coverage ratio; Path i For all nodes on the data flow path, this path is reconstructed from the event sequence of S2.1; j is the index of the node on the flow path; p j This is the processing quality coefficient for node j, which can be configured by the system based on information such as historical processing error rate and data integrity.

[0097] 2) Processing complexity index P c (D) is used to measure the cost and difficulty of data processing. This metric is calculated by analyzing the accompanying attributes of "data preprocessing events," "data transformation events," and "data fusion events" in the event log. The accompanying attributes extracted in S2.1 (derived from the fields in S1.2 used to record the measurement information of the operation results) provide objective data such as execution time and number of rows processed for calculating this metric. The specific calculation formula is as follows:

[0098]

[0099] In the formula, P c (D) represents the processing complexity metric for data source D; m is the number of processing steps, which can be derived from the event sequence length of a specific case or the number of specific operation events; K is the index of the processing steps; t k The execution time of step k can be obtained from the timestamp difference of the event or from an attached attribute; c k To calculate the resource consumption coefficient, this coefficient can be predefined according to different operation types (such as CPU-intensive or I / O-intensive); k The data size involved in step k can be obtained from the event's attached properties.

[0100] 3) Business Relevance Indicator B r (D) measures the contribution of data to business objectives. This metric is also calculated based on event logs, specifically linking "data application events" with predefined business goals. The specific calculation formula is as follows:

[0101]

[0102] In the formula, B r (D) represents the business relevance index of data source D; b represents the preset number of business targets; i represents the index of the business target; v i The importance weight for the i-th business objective can be provided by the business rule base; f i Let e ​​be the frequency of data usage in the i-th business objective. This frequency can be obtained by counting the number of "data application events" related to business objective i in the event log. i To use the effectiveness coefficient, this coefficient can be quantified or configured based on whether the use of the data triggers key business decisions or generates business benefits.

[0103] 4) Time-effectiveness value index T v (D) measures how the value of data changes over time. This metric relies on timestamp information in the event log. The specific formula is as follows:

[0104]

[0105] In the formula, T v (D) represents the timeliness value index of data source D; D is the data source; d is a data item in data source D; V0(d) is the initial value of data item d, which can be configured according to the data type; e is the natural constant; λ d α is the decay coefficient of data item d, which can be preset according to business needs (e.g., news data decays quickly, while archive data decays slowly); (t-t0(d) is the generation time of data item d; t is the current time; RefCount(d,t) is the number of times data d has been referenced before time t; α is the reference value coefficient, which can be preset to reflect the value enhancement effect of references.

[0106] Steps S2.1 and S2.2 objectively quantify the independent value of the data sources. The event log standardization and path reconstruction in step S2.1 provide objective and traceable data input for the quantification model in step S2.2, resolving the problem of reliance on subjective scoring and estimation in traditional value assessment. The basic value of each data source calculated in step S2.2 will serve as the calculation benchmark for evaluating the fusion gain and synergistic value in step S3.

[0107] Step S3, based on the fundamental value of the data source calculated in S2 and the data fusion and application event logs captured in S1, quantifies and evaluates the fusion gain and collaborative value generated by the data fusion behavior. The specific implementation of step S3 includes sub-steps S3.1, S3.2, and S3.3.

[0108] S3.1 Quantifying Fusion Performance Metrics Based on Event Logs This step aims to objectively quantify the specific performance improvements brought about by data fusion behavior from multiple dimensions using the event logs from S1. These metrics will serve as the basis for calculating the value V(D) of the fused dataset in S3.2. F The key adjustment coefficient.

[0109] 1) Information integrity improvement index C e : Evaluate the degree of improvement in the integrity of the fused data compared to the single-source data.

[0110]

[0111] In the formula, C eThis is an indicator for improving information integrity; n is the number of data sources participating in the fusion; i is the index of the data source; and MissingRate is the data missing rate. For data source D in S2 i Independent missing rate; For Di Di's data in the fused dataset D F The missing rate of the corresponding part.

[0112] Independent missing rate and the missing rate after fusion All of these results were obtained by analyzing the proportion of missing, NULL, or invalid values ​​recorded in the "data preprocessing events" or "data application events" of S1 to the total amount of data.

[0113] 2) Data quality improvement indicator Q e : Evaluate the improvement in data consistency and accuracy achieved by the fusion process.

[0114]

[0115] In the formula, Q e q represents the data quality improvement metric; p is the total number of quality metrics; q is the index of the quality metric; QualityMetric q (D F ) for fusion dataset D F The qth quality score; AverageQualityMetric q (D indep ) represents the average of the q-th quality scores from each independent data source before merging; w iq Let q be the weight of the q-th quality indicator.

[0116] The quality scores are all calculated based on the data quality inspection results and rule violations recorded in the "data transformation event" or "data preprocessing event" in S1.

[0117] 3) Decision support enhancement indicator D e : Evaluate the effect of data fusion on improving decision support capabilities.

[0118]

[0119] In the formula, D e DecisionAccuracy is an enhanced indicator for decision support; d represents the total number of decision scenarios; j represents the index of the decision scenario; j (D F ) for using fused data D F DecisionAccuracy j (Dindep ) represents the accuracy on the j-th type of decision using independent data; w jd represents the weight of the j-th type of decision. (The decision accuracy is calculated based on the decision results (such as prediction and classification) recorded in the "Data Application Events" section of S1 and their subsequent actual verification results.)

[0120] S3.2 Calculating the overall value of the fusion dataset based on the S2 model and S3.1 indicators.

[0121] This step reuses the value function V(D) = f(U) from S2.2. f (D),P c (D),B r (D),T v (D)), but the performance index (C) calculated using S3.1 e Q e D e This is used to adjust the various dimensional metrics defined in S2, thereby calculating the fused dataset D. F Overall data value V(D) F ).

[0122] A specific example of adjustment is as follows:

[0123] U f (D F )=U f (D avg )·β u ·D e (Data fusion has gained wider use due to its ability to improve decision-making.)

[0124] P c (D F ) = P c (D avg ) / (β p ·Q e (The improved quality of the merged data reduces the complexity of subsequent processing);

[0125] B r (D F ) = B r (D avg )·β b ·D e (Data fusion significantly enhances business relevance due to improved decision-making accuracy);

[0126] T v (D F ) = T v (D avg )·β t ·C e(Data integration enhances data completeness and extends its timeliness value);

[0127] In the formula, D avg This represents the weighted average of the S2 metrics from each independent data source before merging; C e Q e D e The index calculated for S3.1; β u ,β p ,β b ,β t This is the preset sensitivity adjustment coefficient.

[0128] Through the above adjustments, the specific improvements in S3.1 (C) e Q e D e The data is objectively transmitted to the value model of S2, and the overall data value V(D) is ultimately calculated. F ).

[0129] S3.3 The Collaborative Value of Computational Data Fusion

[0130] This step aims to quantify the overall gain created by the data fusion process, i.e., the collaborative value. This collaborative value is derived by subtracting the sum of the base values ​​calculated in S2.2 for all participating data sources from the overall value of the fused dataset calculated in S3.2. The specific calculation formula is as follows:

[0131]

[0132] In the formula, V syn The collaborative value of data fusion; V(D) F The fused dataset D calculated by S3.1 F The overall value; N is the total number of data sources participating in the integration; V(D) i ) represents the base value of the i-th data source calculated by S2.2; i is the index of the data source participating in the fusion.

[0133] Through steps S3.1 to S3.4, this invention not only quantifies the independent value of the data source (S2), but also, based on the objective event logs in S1, first quantifies the specific performance improvement brought about by the fusion (S3.1), then calculates the total value after fusion based on this performance improvement (S3.2), and finally calculates the collaborative value V created by the fusion. syn Collaborative Value V syn It will be used as the calculation target for value tracing and allocation in step S4.

[0134] Step S4 involves designing a value tracing mechanism based on the data value quantification model and allocating fusion gain.

[0135] This step aims to transform the collaborative value V calculated in S3.3 into... syn (That is, the overall gain created by the integration), through a rigorous value tracing mechanism, is reasonably attributed to each data source D participating in the integration. i This step employs a counterfactual causal reasoning framework, simulating scenarios of "how the total value would change if a certain data source were removed" to accurately assess the contribution of that data source.

[0136] The specific implementation of step S4 includes sub-steps S4.1, S4.2, S4.3 and S4.4.

[0137] S4.1 Constructing a Counterfactual Valuation Model

[0138] This step defines the computational basis for counterfactual assessment. The calculations of all formulas in S4.2 and S4.3 rely on the models established in S1, S2, and S3.2.

[0139] Specifically, the "total value of the complete dataset" V(D) required in the formula is the overall value V(D) of the fused dataset already calculated in S3.2. F ).

[0140] And calculate "remove data source D" i The value of the dataset after processing is V(D\{D) i When (i.e., counterfactual value) is involved, the present invention is achieved in the following ways:

[0141] Simulated data removal: Filter and exclude all data from the S1 event log that is related to data source D. i Related "data access events" and "data fusion events".

[0142] Recalculate Independent Values: For the "incomplete" event logs of this simulation, re-execute the model in S2.2 to calculate the simulation dataset D. i The independent value of the remaining data sources in}.

[0143] Recalculate the fusion value: For this simulation log, re-execute the calculations of S3.1 and S3.2, i.e., quantify the performance index (C) of the simulation fusion. e Q e D e Substituting this into the model in S3.2, the counterfactual value V(D\{D)) is finally calculated. i}).

[0144] S4.2 Calculates the basic contribution indicators. Based on the counterfactual evaluation model established in S4.1, this step calculates the two basic indicators required for evaluating the contribution:

[0145] 1) Remove the impact index RI(D) i ):

[0146]

[0147] In the formula, RI(D) i () is the data source D i The removal impact index; V(D) is the value of the complete fused dataset calculated in S3.2. F );V(D\{D i}) is calculated in S4.1 and removes D i The counterfactual value that follows.

[0148] 2) Marginal contribution coefficient γ(D) i D j ):

[0149]

[0150] In the formula, γ(D) i D j () is the data source D i Relative to data subset D j The marginal contribution increasing coefficient; D j It is any one that does not contain D i Data subset; V({D i}) is D calculated by S2.2 i Independent value; V(D) j ) and V(D j ∪{D i All values ​​were calculated using the counterfactual assessment model of S4.1.

[0151] S4.3 Calculate the overall contribution C(D) i )

[0152] This step combines the removal of impacts and marginal contributions to calculate data source D. i The final contribution.

[0153]

[0154] In the formula, C(D) i () is the data source D i The overall contribution; α and β are pre-set weighting coefficients used to balance the importance of the two indicators; n is the total number of data sources; j is the index of the data subset; w j For data subset D j The importance weight can be determined based on the frequency of the subset's occurrence in the S1 event log.

[0155] To ensure that the sum of all contributions is 1, this step will also process all C(D) contributions. iThe values ​​are then normalized to obtain the final contribution weights.

[0156] The specific method of unification is as follows: first, calculate the total contribution C of all data sources. total Then, C(D) for each data source i Dividing by this sum yields the normalized contribution weight C′. norm (D i ).

[0157] S4.4 Value Gain Allocation Based on Contribution: This step is the final application of value tracing. This step uses the normalized comprehensive contribution weight C(D) calculated in S4.3. i To allocate the collaborative value V calculated in S3.3. syn .

[0158] 1) Individual gain value V gain (D i ):

[0159] V gain (D i )=C(D i )·V syn ;

[0160] In the formula, V gain (D i () is the data source D i The value of the fusion gain to be allocated; C(D) i V represents the normalized contribution weight calculated in S4.3; syn The collaborative value calculated for S3.3.

[0161] 2) The final value V of the data source final (D i ):

[0162] V final (D i )=V(D i )+V gain (D i );

[0163] In the formula, V final (D i () is the data source D i Ultimate attribution value; V(D) i The base value of the data source is calculated by S2.2.

[0164] In step S4, this invention reuses the models from S1, S2, and S3 to support counterfactual calculations, and (S4.2-S4.3) rigorously calculates the contribution weight C(D) of each data source. i), and ultimately the synergistic value V of S3.3 syn Distribute fairly among the various data sources.

[0165] Reference Appendix Figure 3 This invention provides a multi-source data fusion value quantification system based on event logs, comprising:

[0166] The event logging module is used to set up event loggers in multiple processing nodes of the data fusion process to capture and generate event logs.

[0167] The independent value assessment module is used to build an independent value assessment model for data sources based on event logs and calculate the basic value of each data source.

[0168] The collaborative value calculation module is used to calculate the overall value of the fused dataset based on event logs, and to determine the collaborative value of data fusion based on the sum of the overall value and the basic value.

[0169] The value tracing module is used to build a counterfactual value assessment model. It calculates the counterfactual value by simulating scenarios where data sources are removed, and calculates the comprehensive contribution of each data source based on the counterfactual value.

[0170] The value allocation module is used to allocate collaborative value based on overall contribution and collaborative value, thereby obtaining the individual gain value of each data source.

Claims

1. An event record log-based multi-source data fusion value quantification method, characterized in that, Includes the following steps: Event loggers are set up in multiple processing nodes of the data fusion process to capture and generate event logs; Based on the event logs, an independent value assessment model for data sources is constructed to calculate the basic value of each data source. Based on the event logs, the overall value of the fused dataset is calculated, and the synergistic value of the data fusion is determined based on the sum of the overall value and the basic value. A counterfactual value assessment model is constructed, and the counterfactual value is calculated by simulating a scenario of removing data sources. Based on the counterfactual value, the comprehensive contribution of each data source is calculated. Based on the overall contribution and the collaborative value, the collaborative value is allocated to obtain the individual gain value of each data source.

2. The method of claim 1, wherein, The setting of the event logger specifically includes: Aspect-oriented programming (AOP) is used to define event logging annotations and event logging aspects. By intercepting business methods marked by the event recording annotation through the event recording aspect, the event log can be automatically recorded.

3. The method of claim 1, wherein, The construction of the data source independent value assessment model specifically includes: Based on the event logs, usage frequency, processing complexity, business relevance, and timeliness value metrics are calculated respectively. The basic value is calculated by combining the value functions of the usage frequency index, the processing complexity index, the business relevance index, and the timeliness value index.

4. The method of claim 1, wherein, The overall value of the computational fusion dataset specifically includes: Based on the event log, quantitative information integrity improvement indicators, data quality improvement indicators, and decision support enhancement indicators are used as fusion performance indicators. The independent value assessment model of the data source is reused, and the dimensional indicators in the independent value assessment model of the data source are adjusted using the fusion performance index to calculate the overall value of the fused dataset.

5. The method for quantifying the value of multi-source data fusion based on event logs according to claim 1, characterized in that, The construction of the counterfactual value assessment model specifically includes: A simulated event log is generated by excluding relevant events from specific data sources from the event log; For the simulated event logs, the independent value assessment model of the data source is reused to calculate the remaining basic value of the simulated dataset; The counterfactual value is calculated by reusing the steps of calculating the overall value of the fused dataset for the simulated event log.

6. The method for quantifying the value of multi-source data fusion based on event logs according to claim 1, characterized in that, The calculation of the overall contribution of each data source specifically includes: Calculate the removal impact index, which is obtained by dividing the difference between the overall value and the counterfactual value by the overall value; The marginal contribution coefficient is calculated by dividing the value increment of the data subset added to the data source by the base value of the data source. The weighted sum of the removal impact index and the marginal contribution coefficient is then normalized to obtain the comprehensive contribution.

7. The method for quantifying the value of multi-source data fusion based on event logs according to claim 3, characterized in that, The frequency indicators used in the calculation specifically include: By analyzing the data application events in the event logs, we can obtain the scene weight, query complexity coefficient, and data coverage ratio. By reconstructing the data usage path, the node processing quality coefficient can be obtained; The usage frequency index is calculated by combining the scenario weight, the query complexity coefficient, the data coverage ratio, and the node processing quality coefficient.

8. The method for quantifying the value of multi-source data fusion based on event logs according to claim 3, characterized in that, The computational complexity metrics specifically include: By analyzing the data preprocessing events, data transformation events, and data fusion events in the event log, the execution time of the processing steps, the resource consumption coefficient, and the data scale can be obtained. The processing complexity index is calculated based on the execution time, the computational resource consumption coefficient, and the data size.

9. The method for quantifying the value of multi-source data fusion based on event logs according to claim 1, characterized in that, The method further includes adding the individual gain value to the base value to obtain the final attribution value of each data source.

10. A multi-source data fusion value quantification system based on event logs, characterized in that, include: The event logging module is used to set up event loggers in multiple processing nodes of the data fusion process to capture and generate event logs; The independent value assessment module is used to construct an independent value assessment model for the data sources based on the event logs and calculate the basic value of each data source. The collaborative value calculation module is used to calculate the overall value of the fused dataset based on the event logs, and determine the collaborative value of data fusion based on the sum of the overall value and the basic value. The value tracing module is used to construct a counterfactual value assessment model, calculate the counterfactual value by simulating a scenario of removing the data source, and calculate the comprehensive contribution of each data source based on the counterfactual value; The value allocation module is used to allocate the collaborative value based on the overall contribution and the collaborative value, thereby obtaining the individual gain value of each data source.

Images