Online truth value discovery system for multi-source dynamic data based on autoregressive model

Abstract

The present invention relates to an online truth value discovery system for multi-source dynamic data based on an autoregressive model, comprising the following modules: a data source, which collects multi-source dynamic data; an online truth value discovery algorithm module, which comprises three submodules: an autoregressive model submodule, a dynamic data source weight calculation submodule, and an optimized truth value calculation submodule; the autoregressive model submodule, which is used to mine the time pattern of truth value evolution from historical truth value data; the dynamic data source weight calculation submodule, which compares the historical truth value information calculated by the optimized truth value calculation submodule with the data information provided by the data source, dynamically adjusts the data source weight, and transmits it to the optimized truth value calculation submodule; the optimized truth value calculation submodule, which uses the information obtained by the dynamic data source weight calculation submodule and the truth value information predicted by the autoregressive model submodule, and dynamically calculates the potential truth value at the current moment by minimizing the objective function. The present invention can quickly and accurately calculate the potential truth value in multi-source dynamic conflict data.

Description

Technical Field

[0001] The invention belongs to the technical field of data mining and database, and in particular relates to an online truth value discovery system for mining potential real data in multi-source dynamic conflict data. Background Art

[0002] With the development of network technology and the advancement of science and technology, massive amounts of data are generated every day in the real world. These data may come from the Internet, from sensors collected from the real physical world, or from some manual records.

[0003] Due to the inevitable existence of noise in the objective world, the data people obtain may not be true. Especially when people obtain data on a certain thing from multiple sources, different data sources often produce conflicting data on the same thing.

[0004] There are many reasons for this problem. Objectively, the accuracy of the data collection equipment causes deviations from the actual data. Subjectively, due to the data collector's incorrect operation or deliberate data falsification, the data people get will deviate from the actual data. At the same time, different data sources have different reliability for the same information. People have reason to believe that news reported by well-known media is much more credible than news from an unknown self-media that they have never heard of.

[0005] The purpose of truth value discovery is to mine the most likely correct potential real data from multiple conflicting data sources. In recent years, the problem of truth value discovery has attracted the attention of a large number of researchers, who have proposed various truth value discovery methods from different perspectives. For example, the simplest Voting, Average, and Median algorithms use voting, averaging, and median to find the truth value, respectively. However, this type of algorithm ignores a basic problem: all data sources do not have the same reliability. Obviously, these three algorithms do not consider the reliability of the data source, that is, they regard the reliability of all data sources as the same. Therefore, although this type of algorithm is simple, its actual performance is often not very good.

[0006] Taking into account the reliability of the data source, the existing CRH algorithm designs different loss functions for different data types, and the CATD algorithm is used to solve the truth value discovery problem when there are a large number of long-tail effects in the data; the LTM algorithm based on the probabilistic graphical model uses the Bayesian method to study the truth value discovery problem in multi-source classification data, and the GTM algorithm is used to solve the truth value discovery problem in numerical data.

[0007] However, most of the existing truth value discovery algorithms treat the truth value as a static data. These truth value discovery algorithms are designed for static data. These algorithms are batch processing algorithms, suitable for processing static batch data. Although these algorithms have high accuracy, their complexity and computational effort are not suitable for truth value discovery problems in dynamic flow data. These algorithms also ignore an important issue in truth value discovery - the truth value evolution problem.

[0008] In many real-life scenarios, the underlying truth value often changes dynamically over time rather than remaining static. Moreover, for dynamic streaming data, the weight information of the data source should be changed dynamically by comparing the real data calculated by the algorithm with the data provided by the data source, rather than keeping the weight information of the data source unchanged.

[0009] Based on the above situation, the present invention studies the problem of dynamic truth value discovery and proposes a new online truth value discovery technology solution, which uses the autoregressive model in time series analysis to mine the basic patterns in the truth value evolution process, uses the calculated historical truth value data to predict the possible truth value data at the current moment, and applies this to the objective function of truth value optimization, combining the truth value evolution pattern with the truth value optimization problem, thereby calculating the potential truth values ​​in multi-source dynamic conflict data, and dynamically calculating the weight information of the data source. Summary of the invention

[0010] Based on the deficiencies in the prior art, the present invention provides an online truth value discovery system based on an autoregressive model, which is used to mine potential real data from multi-source dynamic data.

[0011] The present invention combines the optimization technology in traditional truth value discovery and also utilizes the characteristics of the autoregressive model in time series analysis to mine the evolution pattern of the truth value on a time scale.

[0012] Different from the traditional method, the present invention involves an online algorithm that can quickly process dynamic flow data from different data sources by only using estimated historical true value data without using all historical data, so as to efficiently and accurately estimate the true value of the data at the current moment, and combine the calculated historical true value information with the data information provided by the data source to dynamically derive the weight information of different data sources.

[0013] In order to achieve the above object, the present invention adopts the following technical solutions:

[0014] The technical solution of the present invention involves the following basic concepts:

[0015] Data source: provides the source of conflicting data, such as websites, databases, sensors, etc.

[0016] Entity: Something that exists in the real world, such as a city, a mountain, etc.

[0017] Entity attributes: Attributes that describe the characteristics of an entity, such as the average daily temperature of a city, the altitude of a mountain, etc.

[0018] Truth value: objective and real data information that describes an entity.

[0019] The present invention solves the source weight and the data truth value by minimizing the following formula:

[0020]

[0021] in,

[0022]

[0023] In the formula, w s The weight corresponding to the data source s, represents the potential truth value of entity i at time t to be calculated, is the data information about entity i provided by data source s at time t, Indicates the number of data information that data source s can provide from time 1 to the current time t, Represents the entity set that data source s can provide data information for at time t. It is the true value predicted by the autoregressive model about the data source s at time t.

[0024] The present invention provides an online truth value discovery system for multi-source dynamic data based on an autoregressive model, which includes the following modules:

[0025] Data source, collects dynamic data from multiple sources; each data source can provide data information to multiple entities, and at any time, only one piece of data information can be provided to the same entity;

[0026] The online truth value discovery algorithm module runs in real time on the computing device and includes three submodules: the autoregressive model submodule, the dynamic data source weight calculation submodule, and the optimized truth value calculation submodule;

[0027] The autoregressive model submodule is used to mine the temporal pattern of truth value evolution from historical truth value data, so that the calculated truth value information is more consistent with the development direction of the truth value trend;

[0028] The dynamic data source weight calculation submodule is used to dynamically update the data source weight; compare the historical truth value information calculated by the optimal truth value calculation submodule with the data information provided by the data source, dynamically adjust the data source weight, and pass it to the optimal truth value calculation submodule for application in truth value calculation;

[0029] The optimized truth value calculation submodule uses the information obtained by the dynamic data source weight calculation submodule and the truth value information predicted by the autoregressive model submodule to dynamically calculate the potential truth value at the current moment by minimizing the objective function.

[0030] Preferably, the objective function is as follows:

[0031]

[0032] Preferably, the autoregressive model submodule reflects the potential evolution pattern of the true value through modeling, that is, in, represents the truth value of entity i at the current time t; represents the predicted true value of the autoregressive model of entity i at the current time t; φ j are the coefficients of the autoregressive model; ∈ t is Gaussian white noise. In actual prediction, the Gaussian white noise term can be removed to obtain

[0033] Preferably, the dynamic data source weight calculation submodule dynamically updates the data source weight, and the weight calculation formula of the data source s is:

[0034]

[0035] in, Indicates the number of data information that data source s can provide from time 1 to the current time t. Indicates the number of data information that the data source s can provide at the current time t, represents the cumulative error of the data source s at the current time t, It represents the entity set that can provide data information for data source s at time t.

[0036] Update the weight w of data source s each time s When recording and Then update w next time s When the number of nodes is increased, incremental updates can be achieved, which greatly improves the calculation speed.

[0037] Preferably, the optimal truth value calculation submodule uses the Smooth L1 loss function to measure and The difference between the two is of the form:

[0038]

[0039] Among them, represents the potential truth value of entity i at time t to be calculated, is the predicted true value of data source s at time t predicted by the autoregressive model. This loss function is included in the objective function to be optimized by this algorithm and determines the calculation of the final estimated true value .

[0040] Preferably, the online multi-source dynamic truth discovery algorithm module is specifically as follows:

[0041] S1. Initialize parameters: P, w s , where s ∈ {1, 2,..., S} represents all data sources, and S is the total number of all data sources. w s is the weight corresponding to data source s, corresponding to the reliability of data source s. w s The larger it is, the more trustworthy the information provided by data source s is. Vice versa, if the information provided by data source s is closer to the true information, then its weight w s should be larger. P is the order of the autoregressive model. And let the current time t = 1, and enter S2;

[0042] S2. If the current time t is less than the order P + 1 of the autoregressive model, that is, t < P + 1, then for each entity, at the current time t, for all entities of all data sources s, there is

[0043]

[0044] i represents the i-th entity of the data source, represents the potential truth value to be calculated, is the data information about entity i provided by the s-th data source at time t, and enter S5; if t ≥ P + 1, then enter S3;

[0045] S3. Calculate the true value predicted by the autoregressive model

[0046]

[0047] is the historical true value of the i-th entity at time t - j, φ j is the coefficient of the autoregressive model, and j ∈ {1, 2,..., P}, and there are P coefficients in the P-order autoregressive model. Denote φ = [φ1, φ2,..., φ P as the autoregressive model coefficient vector, is the historical true value matrix of entity i, is the historical true value vector of entity i, then φ = (A T A) -1 AT X. And enter S4;

[0048] S4: This step is based on the weight information w of the data source s calculated in the above steps. s , the true value of entity i at the current time t predicted by the autoregressive model about the data source s And the data information about entity i provided by data source s at the current time t The relationship between can be divided into the following three cases to discuss the truth value calculation of entity i at the current moment:

[0049] 1. If So:

[0050]

[0051] 2. If So:

[0052]

[0053] 3. If So:

[0054]

[0055] In this step, first calculate the truth value from the corresponding formula of condition 1, and then determine whether it satisfies condition 1. If it does, directly proceed to the next step S5; if not, calculate the truth value using the formula of condition 2, and then determine whether it satisfies condition 2. If it does, proceed to the next step S5; if it does not satisfy condition 2, directly calculate the truth value using the formula of condition 3, and then proceed to S5;

[0056] S5. For all data sources s∈{1,2,…,S}, all historical truth values ​​of all entities i calculated using the above steps And the data information of all entities provided by data source s at the current time t, calculate the weight information of data source s:

[0057]

[0058] in, Indicates the number of data information that data source s can provide from time 1 to the current time t. Indicates the number of data information that the data source s can provide at the current time t, Record represents the cumulative error of the data source s at the current time t, It represents the entity set that can provide data information for data source s at time t. Record And enter S6;

[0059] S6. Let t = t + 1, and determine whether new data can continue to be received. If no new data is received, end; if new data is received, return to step S2; that is, the true value discovery process of multi-source dynamic conflict data based on the autoregressive model is realized.

[0060] As a preferred technical solution of the present invention: in S1, the initialization of w s adopts a uniform generation method, that is, let all w s be equal: w s = 1 / S, where S is the total number of data sources.

[0061] As a preferred technical solution of the present invention: in S2, since at least P true value data are required to model the autoregressive model in this invention, therefore, when t < P + 1, there is no need to use the autoregressive model to predict the data true value.

[0062] As a preferred technical solution of the present invention: in S3, φ is the coefficient vector of the autoregressive model, which is estimated by the least squares method and will change dynamically with time.

[0063] As a preferred technical solution of the present invention: in S4, for three cases, the data true value should be calculated by formula (3) or (4) or (5) in sequence first, and then brought into the corresponding conditions to determine whether it meets the condition constraints.

[0064] As a preferred technical solution of the present invention: in S5, based on the calculated historical true values, the weight information of the data sources can be dynamically updated and applied to the true value calculation of the next moment. The present invention is an online true value discovery method and system for multi-source dynamic conflict data based on the autoregressive model. The present invention adopts the above technical solutions and has the following effects compared with the prior art:

[0065] (1) The online true value discovery method for multi-source dynamic conflict data based on the autoregressive model of the present invention considers the true value evolution problem in the true value discovery process. Regarding the true value discovery problem in multi-source dynamic data as a dynamic process, the present invention can discover the evolution mode of the data true value on the time scale and quickly calculate the true value data in the dynamically flowing data from multiple data sources in an online form.

[0066] (2) The online true value discovery method for multi-source dynamic conflict data based on the autoregressive model of the present invention considers the reliability of the data sources, that is, the weight information of the data sources. The weight information of the data sources is dynamically calculated by using the historical true values and the data information provided by the data sources, and this weight information is applied to the true value calculation of the next moment.

[0067] (3) The online truth value discovery method for multi-source dynamic conflict data based on the autoregressive model of the present invention combines the autoregressive model in time series analysis with the optimization method in the traditional truth value discovery algorithm, thereby having the advantages of both the accuracy of the traditional truth value discovery algorithm using the optimization method to process batch static data and the speed of the autoregressive model in time series analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 is a flow chart of an online truth value discovery method for multi-source dynamic data according to an embodiment of the present invention;

[0069] Figure 2 It is a schematic diagram of a model of an online truth value discovery algorithm for multi-source dynamic data according to an embodiment of the present invention;

[0070] Figure 3 The present invention is a block diagram of an online truth value discovery system for multi-source dynamic data based on an autoregressive model. DETAILED DESCRIPTION

[0071] The specific implementation of the present invention is further described in more detail below in conjunction with the specification and drawings.

[0072] The problem to be solved by the present invention is to mine the most likely real data information in an environment of dynamic flow data from multiple data sources, while ensuring its accuracy and speed.

[0073] In the real world, information about an entity can be collected from multiple data sources. However, the data collected directly from these data sources is likely not to be objective and true data that reflects the attributes of the entity. Moreover, since different data sources use different technologies and collect information in different ways, the information collected by each data source for the same entity is likely to be different.

[0074] Some data is not static, but is generated continuously over time. Therefore, traditional batch processing algorithms are not suitable for processing such dynamic data due to their time complexity.

[0075] In view of the above problems, an embodiment of the present invention provides an online truth value technology solution for processing multi-source dynamic data, which can quickly and accurately mine out real data information that is most likely to conform to objective facts from multiple data sources.

[0076] The technical solution of this embodiment will involve the following basic concepts:

[0077] Data sources: Sources that provide conflicting data, such as various weather websites.

[0078] Entity: An entity is something that exists in the real world, such as a city, for example Hangzhou.

[0079] Entity attributes: attributes that describe the characteristics of an entity, such as the average daily temperature in Hangzhou.

[0080] True value: describes the objective and real data information of an entity, such as the objective and real daily average temperature in Hangzhou.

[0081] Based on the analysis of the multi-source dynamic data truth discovery model and real-world data sets, this embodiment is based on the following principles:

[0082] (1) Weight w of data source s s Reflects its reliability. s The larger it is, the more reliable the information provided by the data source s is.

[0083] (2) Data source weight w s Should change dynamically over time;

[0084] (3) A data source can only provide one piece of data information for an entity attribute at the same time.

[0085] like Figure 3 As shown, this embodiment is a multi-source dynamic data online truth value discovery system based on an autoregressive model, which includes the following modules (each module is Figure 3 Connection shown):

[0086] Data source, collects dynamic data from multiple sources; each data source can provide data information to multiple entities, and at any time, only one piece of data information can be provided to the same entity;

[0087] The online truth value discovery algorithm module runs in real time on the computing device and includes three submodules: the autoregressive model submodule, the dynamic data source weight calculation submodule, and the optimized truth value calculation submodule;

[0088] The autoregressive model submodule is used to mine the temporal pattern of truth value evolution from historical truth value data, so that the calculated truth value information is more consistent with the development direction of the truth value trend;

[0089] The dynamic data source weight calculation submodule is used to dynamically update the data source weight; compare the historical truth value information calculated by the optimal truth value calculation submodule with the data information provided by the data source, dynamically adjust the data source weight, and pass it to the optimal truth value calculation submodule for application in truth value calculation;

[0090] The optimization truth value calculation submodule uses the information obtained by the dynamic data source weight calculation submodule and the truth value information predicted by the autoregressive model submodule to minimize the following objective function:

[0091]

[0092] The potential truth value at the current moment is calculated.

[0093] In this embodiment, the autoregressive model submodule reflects the potential evolution pattern of the true value through modeling, that is, in, represents the truth value of entity i at the current time t; represents the predicted true value of the autoregressive model of entity i at the current time t; φ j are the coefficients of the autoregressive model; ∈ t is Gaussian white noise. In actual prediction, the Gaussian white noise term can be removed to obtain

[0094] In this embodiment, the dynamic data source weight calculation submodule dynamically updates the data source weight, and the weight calculation formula of the data source s is:

[0095]

[0096] in, Indicates the number of data information that data source s can provide from time 1 to the current time t. Indicates the number of data information that the data source s can provide at the current time t, represents the cumulative error of the data source s at the current time t, It represents the entity set that can provide data information for data source s at time t.

[0097] Update the weight w of data source s each time s When recording and Then update w next time s When the number of nodes is increased, incremental updates can be achieved, which greatly improves the calculation speed.

[0098] In this embodiment, the optimization truth value calculation submodule uses the Smooth L1 loss function to measure and The difference between the two is of the form:

[0099]

[0100] in, represents the potential truth value of entity i at time t to be calculated, The predicted true value of the data source s at time t predicted by the autoregressive model. This loss function is included in the objective function to be optimized by this algorithm and determines the final estimated true value Calculation.

[0101] In this embodiment, if Figure 1 , 2 As shown, the online multi-source dynamic truth value discovery algorithm module specifically operates in the following steps:

[0102] S1, initialization parameters P, w s , where s∈{1,2,3} indicates that there are only three data sources, that is, the data comes from three meteorological websites. s The initialization adopts uniform generation method, that is, all w s All are equal: w s =1 / S, then the weights of the three weather websites are: w1=w2=w3=1 / 3. P is the order of the autoregressive model, let P=2. And let the current time t=1, enter S2;

[0103] S2, if the current day is the fourth day, that is, t = 4. Then t = 4> 2 + 1, so the autoregressive model does not lack data modeling, so enter S3;

[0104] S3. If the historical daily average temperature of the previous three days has been estimated: By formula The historical truth matrix can be calculated Use the formula The historical truth vector X = (18, 17.5) is calculated, and the formula φ = (A T A) -1 A T X, calculate the regression coefficient vector φ = [0.93, 0.11]. Then when t = 4, according to formula (2), the predicted true value of the daily average temperature on the fourth day can be calculated as and enter S4;

[0105] S4. The average daily temperatures in Hangzhou provided by three meteorological websites on the fourth day are: In the last cycle, the weights of the three data sources are calculated as follows: w1 = 0.5, w2 = 0.1, and w3 = 0.4. First, calculate the estimated true value of the average daily temperature on the fourth day according to formula (3):

[0106]

[0107] and

[0108]

[0109] The corresponding conditions are met, so the estimated true value of the average daily temperature on the fourth day is: and enter S5;

[0110] S5. The daily average temperature data for Hangzhou area in the previous three days provided by three known meteorological websites are: Based on the above data, we can first calculate and By formula Knowable Record By formula It can be seen that: Record Therefore, according to formula (6), the weights of the three meteorological websites are updated as follows: and enter S6;

[0111] S6, let t = 4 + 1 = 5. And determine whether new data can be received at present. If no new data is received within 24 hours, the algorithm ends; if new data is received, return to S2 and continue the next cycle calculation.

[0112] It can be seen from the above embodiments that the present invention can quickly and accurately calculate the potential true value in multi-source dynamic conflict data.

[0113] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. An online truth value discovery system for multi-source dynamic data based on autoregressive model, characterized in that: Includes the following modules: Data source, collects dynamic data from multiple sources; The online truth value discovery algorithm module includes three submodules: autoregressive model submodule, dynamic data source weight calculation submodule, and optimal truth value calculation submodule; The autoregressive model submodule is used to mine the temporal pattern of truth value evolution from historical truth value data; The dynamic data source weight calculation submodule compares the historical truth value information calculated by the optimal truth value calculation submodule with the data information provided by the data source, dynamically adjusts the data source weight, and passes it to the optimal truth value calculation submodule; The optimized truth value calculation submodule uses the information obtained by the dynamic data source weight calculation submodule and the truth value information predicted by the autoregressive model submodule to dynamically calculate the potential truth value at the current moment by minimizing the objective function.

2. The multi-source dynamic data online truth value discovery system based on autoregressive model according to claim 1 is characterized by: The autoregressive model submodule reflects the potential evolution pattern of the true value through modeling, namely in, represents the truth value of entity i at the current time t; represents the predicted true value of the autoregressive model of entity i at the current time t; φ j are the coefficients of the autoregressive model; ∈ t is Gaussian white noise.

3. The online truth value discovery system for multi-source dynamic data based on autoregressive model according to claim 2 is characterized by: The dynamic data source weight calculation submodule dynamically updates the data source weight. The weight calculation formula of data source s is: in, Indicates the number of data information that data source s can provide from time 1 to the current time t. Indicates the number of data information that the data source s can provide at the current time t, represents the cumulative error of the data source s at the current time t, It represents the entity set that can provide data information for data source s at time t.

4. The multi-source dynamic data online truth value discovery system based on autoregressive model according to claim 3 is characterized by: The optimization truth value calculation submodule uses the Smooth L1 loss function to measure and The difference between the two is of the form: in, represents the potential truth value of entity i at time t to be calculated, It is the true value predicted by the autoregressive model about the data source s at time t.

5. The multi-source dynamic data online truth value discovery system based on autoregressive model according to claim 4 is characterized by: The online multi-source dynamic truth value discovery algorithm module is as follows: S1, initialize autoregressive model coefficient P, data source weight w s , and let the current time t = 1; S2. If the current time t < P + 1, calculate the truth value of entity i at the current time t using the following formula and proceed to S5; if t ≥ P + 1, then proceed to S3; S3. Use the following formula to calculate the true value predicted by the autoregressive model: S4. Discuss the following three cases separately and use the corresponding formula to calculate the true value of entity i at the current time t 1) If but: 2) If but: 3) If but: S5. Calculate the weight information w of data source s using the following formula: s ; S6. Let t=t+1. If the data is received within the expected time, return to S2; if the data is not received within the expected time, end.

6. The multi-source dynamic data online truth value discovery system based on autoregressive model according to claim 5 is characterized by: In S1, w s The initialization uses uniform initialization.

7. The multi-source dynamic data online truth value discovery system based on autoregressive model according to claim 5 is characterized by: In S3, φ is the coefficient vector of the autoregressive model, which is estimated by least squares and changes dynamically with time.

8. The multi-source dynamic data online truth value discovery system based on autoregressive model according to claim 5, characterized in that: In S5, the weight information of the data source is dynamically updated based on the calculated historical true value and applied to the true value calculation at the next moment.

Images