Reliable multi-source data fusion method based on cross validation

Abstract

The invention discloses a reliable multi-source data fusion method based on cross validation. Malicious workers exist in a crowd sensing network, the truth phase of data may be affected, and the whole network is damaged, so that the reliable truth phase is obtained by using an unmanned aerial vehicle assisted multi-source data fusion method, and credible workers are identified according to the truth phase to solve the problem; firstly, a platform recruits workers according to worker credibility and quotation; then, unmanned aerial vehicle assistance is recruited for specific tasks which are not executed by workers and are executed by some suspicious malicious workers; after true phase data of the unmanned aerial vehicle data and the worker data are obtained, the true phase of an unexecuted task and the true phase of a task executed by a suspicious worker are recovered according to a depth matrix decomposition method; and finally, evaluating the worker data and updating the worker credibility according to the reliable fusion truth, the complementarity and cross validation of the multi-source data are fully utilized, the data fusion quality is improved, the credibility of the workers is further verified, and sustainable services are constructed.

Description

Technical Field

[0001] The present invention belongs to the field of multi-source data fusion of mobile crowd sensing, and in particular to a method for how crowd sensing can integrate data from various sources to build a high-quality data collection service. Background Art

[0002] Mobile crowd sensing is a popular and efficient multi-source data collection model, which consists of three main components: mobile workers who perceive data, perception platforms that process data and provide perception services, and requesters who need to use and pay for perception services. Faced with the large amount of multi-source data collected by mobile crowd sensing, multi-source data fusion plays a vital role in building high-quality data collection services. Multi-source data fusion integrates data from various sources, such as deployed sensors, drones, and mobile vehicles, to produce reliable fusion results, thereby improving the effectiveness of mobile crowd sensing in providing high-quality data collection services.

[0003] In order to extract true information from conflicting multi-source data, a series of truth discovery methods are widely used in mobile crowdsensing for multi-source data fusion. Existing truth discovery methods usually rely on the assumption that most workers (i.e., sensing sources) are trustworthy, so the estimated true value will tend to be trustworthy. However, even if most workers are trustworthy, erroneous data from a small number of untrustworthy workers can still affect the final results of current truth discovery methods, resulting in bias. More importantly, its inaccuracy increases with the increase in the proportion of untrustworthy workers, and when untrustworthy or malicious workers exceed the majority, attackers can manipulate the fusion results.

[0004] On the other hand, some truth recovery methods have been applied in mobile crowdsensing to address the data sparsity problem. These methods exploit the spatiotemporal correlations between perceived data to recover or infer unperceived data. These methods assume that only a small amount of credible data can be collected and aim to recover (i.e., infer) unperceived data. However, these methods face similar challenges as truth discovery, namely the presence of untrustworthy workers. If the perception platform mistakenly adopts the data forged by malicious workers as a baseline to recover the perception matrix, it may cause error propagation and amplification, making the estimated true value unusable.

[0005] Therefore, the method of the present invention is to propose an effective drone-assisted multi-source data fusion truth discovery method, solve the sparse truth problem through data recovery, and use the obtained truth to effectively evaluate the credibility of workers, which is an effective solution. Summary of the invention

[0006] The present invention discloses a reliable multi-source data fusion method based on cross-validation. The method of the present invention is proposed for obtaining reliable real data in an untrustworthy environment in the current crowd intelligence network to provide high-quality data collection services. Specifically, the present invention first recruits workers based on their quotations and workers' trust; then, based on the recruitment results, a specific task is selected to dispatch drones for active data collection to obtain completely reliable truth data; subsequently, the present invention calculates the estimated truth for other tasks based on the data of the recruited workers, thereby reducing the inevitable negative impact of recruiting untrustworthy employees on the data. After obtaining the drone data and the worker's estimated truth, the platform uses a deep matrix decomposition method to capture the spatiotemporal correlation of data in the sparse data matrix and complete the scarce values ​​in the sparse data matrix; finally, the platform uses the fused truth data to evaluate the worker data and update the worker's trust, thereby more accurately identifying trusted workers round by round.

[0007] The technical solution of the invention is as follows:

[0008] A reliable multi-source data fusion method based on cross-validation, characterized by comprising the following steps:

[0009] (1) The perceived service requester sends M tasks and total budget Submit to the perception platform. The task is divided into multiple rounds, and the number of rounds is represented by t∈{1,2,…}. There are N unknown workers in the perception platform, represented by In each round, the perception platform releases tasks, and workers submit tasks of interest in round t to the perception platform. and the claimed cost of completing these tasks b i,t ; The selected workers will perform the tasks and report the data, Represents worker W i Credibility in round t; Next, the perception platform will obtain reliable data truth through the following methods;

[0010] (2) First, each worker sends the set of tasks that he or she wishes to perform in round t to the platform. and the claimed cost b i,t , the platform will calculate the credibility-cost ratio of each worker:

[0011]

[0012] In the initial stage, the platform will is set to 0.5 to equally recruit unknown workers to estimate their credibility; the platform then selects i,t Sort the first K workers and set The platform gives certain rewards to workers who complete the tasks.

[0013] (3) After recruiting workers in round t, we use λ i,t ∈{0,1} to estimate the number of recruited workers W i Is it credible?

[0014]

[0015] in is a confidence threshold, for example, it can be set to 0.5; when λ i,t = 1, the platform believes that worker W i In the tth round, it is credible, otherwise it is untrustworthy; then, the perception platform will send drones to assist in data collection and worker credibility verification; there are two situations in which drones are used for data collection: one is a task that lacks worker participation but is critical to data recovery reasoning; the other is a task involving suspicious malicious workers; the platform will dispatch drones for x tasks per round, as follows:

[0016] The method of sending drones to collect data for tasks that lack worker participation but are critical for data recovery reasoning is as follows: The perception platform first calculates the historical experience score for each task

[0017]

[0018] Among them, τ is an integer hyperparameter, and They respectively indicate whether the sensing platform is task T in round t. j Send out drones and recruit trusted workers. Represents worker W i Whether to be recruited to perform task T in round t j ; The platform will select the historical experience score s j,t Lower front Send drones for each mission;

[0019] The method of sending drones to collect data for tasks that lack worker participation but are critical for data recovery reasoning is as follows: The perception platform calculates the quality prediction score for each task

[0020]

[0021] The platform also selects the quality prediction score u j,t Lower front Send drones for each mission;

[0022] (4) Then, the platform calculates the truth based on the data from the workers and the drones. The platform believes that the data collected by the drones during the mission is completely credible, so it is directly used as the truth. As for the data collected by the workers, the perception platform uses a weighted average method to discover the truth. The truth discovery method iterates through two stages: weight update and truth aggregation. In the weight update stage, each worker W i For task T j The weight of the reported data is based on the current estimate of the truth calculate, Represents worker W i In round t, task T j The collected data is calculated as follows

[0023]

[0024] represents the distance between the two; in the truth aggregation stage, we perform weighted averaging on the data submitted by the trusted workers and update the estimated truth data d j,t , the update formula is as follows:

[0025]

[0026] The perception platform first initializes the truth data, iteratively updates the weights and aggregates the truth values ​​until the convergence criteria are met;

[0027] (5) Next, the perception platform will use the deep matrix decomposition method to capture the spatiotemporal correlation of the sparse data matrix, complete the missing data in the sparse matrix, and obtain the reliable truth of multi-source data fusion; the perception platform first uses the drone data from the max(t-τ+1,1)th round to the tth round and the trusted worker calculation truth to construct a sparse matrix Then randomly initialize the spatiotemporal correlation matrix The function f(·) represents the correlation between data, so the platform can use the spatiotemporal correlation matrix Fitting a sparse matrix Then the optimization goal of truth recovery can be expressed as:

[0028]

[0029] Among them Ψ ′ Is a binary matrix, if the original sparse matrix If there is drone data or trusted worker truth at [i,j], then Ψ ′ [i,j]=1; otherwise, Ψ ′ [i,j]=0. It is the regularization penalty term that limits the matrix Z to prevent the network from overfitting, and γπ(f) represents the penalty for f(·).

[0030] Deep matrix decomposition uses a deep structured neural network. Assuming there are h hidden layers, the parameters of the neural network are expressed as:

[0031]

[0032] Each layer of the neural network is represented as follows:

[0033]

[0034] Penalty term π(f) and weight matrix ω x and b x Associated, where the weight matrix ω x The influence of is the greatest. Therefore, π(f) can be expressed as:

[0035]

[0036] After training the neural network, the perception platform obtains the parameter ω x and b x Derive the expression of the nonlinear mapping f(·); thus use Perform matrix completion on the sparse matrix so that each task has an estimated fusion truth;

[0037] (6) Finally, the perception platform compares the data submitted by the worker with the corresponding truth data in the data fusion matrix to update the worker’s credibility. i Execute task T in round t j The quality of the reported data is defined as:

[0038]

[0039] in is the task T obtained in round t j The fusion truth data, Execute task T for workers j Reported data; perceived platform usage To represent worker W i In round t, task T j Submitted Data Is it considered credible?

[0040]

[0041] The credibility of workers is modeled as a Beta distribution Beta(Θ,Ψ), and the worker W in round t is i The credibility is expressed as

[0042]

[0043] Next, the perception platform updates the and

[0044]

[0045] in, is an exponential decay parameter, σ(·) returns a regulation factor that controls the speed at which workers’ trust is updated. For drone data, the trusted worker calculates the truth, and the data is restored to complete the truth. The value of is set to σ1>σ2>σ3; then the perception platform calculates a more accurate worker trust to optimize the next round of worker recruitment.

[0046] Beneficial Effects

[0047] The present invention discloses a reliable multi-source data fusion method based on cross-validation. The method of the invention has more beneficial effects than previous methods as follows: Current multi-source data fusion methods usually require that most of the participating sensor sources are credible, or assume that the credibility of the workers is known in advance or easy to calculate. However, these methods are impractical due to the existence of untrustworthy environments and unverified information extraction problems. This may lead to a vicious cycle, where the recruitment of untrustworthy employees will affect the quality of the estimated truth, which in turn will lead to misjudgments of the credibility of employees, thereby exacerbating the quality of subsequent recruitment. The present invention establishes multi-level true value data through three different data sources: drones, trusted workers, and deep matrix decomposition. In addition, the present invention makes full use of the complementarity and cross-validation of multi-source data to improve the quality of data fusion, thereby verifying the credibility of the staff, forming a complementary closed loop, thereby building a reliable and sustainable service, and the experimental results are as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is the specific working process of the present invention;

[0049] Figure 2 Identify effects for trusted workers;

[0050] Figure 3 This is the effect of multi-source data fusion under random attack;

[0051] Figure 4 Verify results for workers’ trust; DETAILED DESCRIPTION

[0052] In order to facilitate the understanding of the present invention, the present invention will be described more comprehensively and meticulously below in conjunction with the accompanying drawings and preferred embodiments of the specification, but the protection scope of the present invention is not limited to the following specific embodiments.

[0053] Unless otherwise defined, all professional terms used below have the same meanings as those generally understood by those skilled in the art. The professional terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the scope of protection of the present invention.

[0054] Unless otherwise specified, various raw materials, reagents, instruments and equipment used in the present invention can be purchased from the market or prepared by existing methods.

[0055] Example:

[0056] In a smart city, it is necessary to monitor the environment in the city in real time, such as temperature, noise, traffic flow, etc., for data perception. At this time, there are a large number of participants with mobile devices distributed in various areas of the city, so the perception platform can participate in data perception by giving workers a certain amount of compensation. However, there are some untrustworthy participants among the participants who report false and malicious data and attack the platform. In order to obtain reliable truth values ​​for sustainable urban quality service construction, the present invention discloses a reliable multi-source data fusion method based on cross-validation to identify malicious participants, thereby improving the credibility of the sensor source and the quality of data fusion. The experimental method is as follows: suppose there are 40 data collection tasks in a city, and there are 400 participants, of which the proportion of untrustworthy participants is 20%. The other participants are all credible participants. Credible participants always report real data, but there is an allowable error between their data and the true value. Malicious participants always submit data with a larger error than the true data to interfere and attack the platform.

[0057] The experimental results of the inventive method are given below.

[0058] Figure 1 The detailed workflow of the present invention is shown. First, the service requester submits the perception demand to the perception platform, and the platform packages it into tasks and publishes it to the workers. The workers submit the tasks and corresponding quotations that they intend to perform in the current round to the perception platform. Next, the perception platform calculates the worker's confidence interval upper bound index and credibility-cost ratio and recruits workers according to the credibility-cost ratio. The recruited workers perform tasks and report data. Then, the perception platform establishes three-stage truth data through three different data sources. First, the perception platform dispatches drones to reveal the truth to obtain golden truth data to assist in data collection and trust verification. Second, the perception platform uses the data reported by the workers for truth discovery to obtain silver truth data. Then, the perception platform uses spatiotemporal deep matrix decomposition to perform truth recovery to obtain bronze truth data. Finally, the perception platform performs a posteriori worker trust verification and calculates the worker's estimated data collection rate to optimize the next round of high-credibility worker recruitment. The perception platform pays a certain amount of remuneration to the recruited workers, and the final fused truth data is fed back to the requester, and the whole process ends.

[0059] Figure 2 The experimental results given are the recognition effects of different algorithms for trusted workers. The comparison algorithms include 0.3-Greedy (randomly select workers with a probability of 0.3, and select workers according to the average of the workers' historical data collection rate with a probability of 0.7), 0.7-Greedy (randomly select workers with a probability of 0.7, and select workers according to the average of the workers' historical data collection rate with a probability of 0.3), and Random (randomly select workers). It can be seen from the experimental results that 85% of the workers recruited by the method of the present invention are trustworthy. In contrast, the proportions of trustworthy employees recruited by 0.3-Greedy and 0.7-Greedy are 72% and 56% respectively. The method of the present invention is superior to other solutions in recruiting trustworthy employees.

[0060] Figure 3 The experimental results given are the effects of multi-source data fusion under random attacks. The comparison algorithms include mean, median and weighted mean. The mean method and median method calculate the mean and median of all data as fused truth data respectively. In the weighted average method, the fused truth data is calculated as the weighted average of the data, and its weight is determined by the data collection rate estimated by the workers. The proportion of untrusted workers is set to 0.2, 0.4, 0.6, and 0.8, and the average absolute error of the fused truth data after T rounds at different proportions is collected. The experimental results show that under both attacks, the average absolute error of all schemes increases with the increase of the proportion of untrusted workers. However, the average absolute error of the method of the present invention grows the slowest and is the lowest among all algorithms.

[0061] Figure 4 The changes in the estimated data collection rates of trusted and untrusted workers under random attacks are given under different methods. The proportion of untrusted workers is set to 0.6 and the average estimated data collection rate of workers in every 20 rounds is calculated. The experimental results show that the method of the present invention is the fastest and most accurate for updating the estimated data collection rate of workers.

Claims

1. A reliable multi-source data fusion method based on cross-validation, characterized by The following steps are involved: (1) The perceived service requester sends M tasks and total budget Submit to the perception platform. The task is divided into multiple rounds, and the number of rounds is represented by t∈{1,2,…}. There are N unknown workers in the perception platform, represented by In each round, the perception platform releases tasks, and workers submit tasks of interest in round t to the perception platform. and the claimed cost of completing these tasks b i,t ; The selected workers will perform the tasks and report the data, Represents worker W i Credibility in round t; Next, the perception platform will obtain reliable data truth through the following methods; (2) First, each worker sends the set of tasks that he or she wishes to perform in round t to the platform. and the claimed cost b i,t , the platform will calculate the credibility-cost ratio of each worker: In the initial stage, the platform will Set to 0.5 to equally recruit unknown workers to estimate their credibility; Then, the platform will select the i,t Sort the first K workers and set The platform gives certain rewards to workers who complete the tasks. (3) After recruiting workers in round t, we use λ i,t ∈{0,1} to estimate the number of recruited workers W i Is it credible? in is a confidence threshold, for example, it can be set to 0.5; when λ i,t = 1, the platform believes that worker W i In the tth round, it is credible, otherwise it is untrustworthy; then, the perception platform will send drones to assist in data collection and worker credibility verification; there are two situations in which drones are used for data collection: one is a task that lacks worker participation but is critical to data recovery reasoning; the other is a task involving suspicious malicious workers; the platform will dispatch drones for x tasks per round, as follows: The method of sending drones to collect data for tasks that lack worker participation but are critical for data recovery reasoning is as follows: The perception platform first calculates the historical experience score for each task Among them, τ is an integer hyperparameter, and They respectively indicate whether the sensing platform is task T in round t. j Send out drones and recruit trusted workers. Represents worker W i Whether to be recruited to perform task T in round t j ; The platform will select the historical experience score s j,t Lower front Send drones for each mission; The method of sending drones to collect data for tasks that lack worker participation but are critical for data recovery reasoning is as follows: The perception platform calculates the quality prediction score for each task The platform also selects the quality prediction score u j,t Lower front Send drones for each mission; (4) Then, the platform calculates the truth based on the data from the workers and the drones. The platform believes that the data collected by the drones during the mission is completely credible, so it is directly used as the truth. As for the data collected by the workers, the perception platform uses a weighted average method to discover the truth. The truth discovery method iterates through two stages: weight update and truth aggregation. In the weight update stage, each worker W i For task T j The weight of the reported data is based on the current estimate of the truth calculate, Represents worker W i In round t, task T j The collected data is calculated as follows represents the distance between the two; in the truth aggregation stage, we perform weighted averaging on the data submitted by the trusted workers and update the estimated truth data d j,t , the update formula is as follows: The perception platform first initializes the truth data, iteratively updates the weights and aggregates the truth values ​​until the convergence criteria are met; (5) Next, the perception platform will use the deep matrix decomposition method to capture the spatiotemporal correlation of the sparse data matrix, complete the missing data in the sparse matrix, and obtain the reliable truth of multi-source data fusion; the perception platform first uses the drone data from the max(t-τ+1,1)th round to the tth round and the trusted worker calculation truth to construct a sparse matrix Then randomly initialize the spatiotemporal correlation matrix The function f(·) represents the correlation between data, so the platform can use the spatiotemporal correlation matrix Fitting a sparse matrix Then the optimization goal of truth recovery can be expressed as: Where Ψ′ is a binary matrix. If the original sparse matrix If there is drone data or trusted worker truth at [i,j], then Ψ′[i,j]=1; otherwise, Ψ′[i,j]=0. is the regularization penalty term that limits the matrix Z to prevent the network from overfitting, γ π (j) represents the penalty for f(·). Deep matrix decomposition uses a deep structured neural network. Assuming there are h hidden layers, the parameters of the neural network are expressed as: Each layer of the neural network is represented as follows: Penalty term π(f) and weight matrix ω x and b x Associated, where the weight matrix ω x Therefore, π(f) can be expressed as: After training the neural network, the perception platform obtains the parameter ω x and b x Derive the expression of the nonlinear mapping f(·); thus use Perform matrix completion on the sparse matrix so that each task has an estimated fusion truth; (6) Finally, the perception platform compares the data submitted by the worker with the corresponding truth data in the data fusion matrix to update the worker’s credibility. i Execute task T in round t j The quality of the reported data is defined as: in is the task T obtained in round t j The fusion truth data, Execute task T for workers j Reported data; perceived platform usage To represent worker W i In round t, task T j Submitted Data Is it considered credible? The credibility of workers is modeled as a Beta distribution Beta(Θ,Ψ), and the worker W in round t is i The credibility is expressed as Next, the perception platform updates the and in, is an exponential decay parameter, σ(·) returns a regulation factor that controls the speed at which workers’ trust is updated. For drone data, the trusted worker calculates the truth, and the data is restored to complete the truth. The value of is set to σ1>σ2>σ3; then the perception platform calculates a more accurate worker trust to optimize the next round of worker recruitment.

Images