A method, device, and medium for multi-source data fusion based on trust configuration

Abstract

This application discloses a method, apparatus, and medium for multi-source data fusion based on trustworthiness configuration, relating to the field of data processing technology. The method includes: standardizing the raw data from each data source to obtain standardized data for each data source; determining the trustworthiness score of each data item in the standardized data for each data source; responding to a data fusion operation with received target data, determining the target data source for each data item in the target data based on the trustworthiness score of the data item in each data source; and fusing the data items corresponding to each target data source to obtain fused data. This effectively improves the efficiency and quality of data fusion.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, device and medium for multi-source data fusion based on trust configuration. Background Technology

[0002] In the data governance process, data fusion is a key step in generating standardized data. However, during the data fusion process, due to the different overall accuracy of different data sources, and the differences in accuracy of different data items within the same data source, the existing technology only rates the overall data of the data source, which makes it difficult to ensure that each data item accurately matches the quality requirements. Summary of the Invention

[0003] This application provides a method, device, and medium for multi-source data fusion based on trust configuration, in order to solve the following technical problem: how to effectively improve the efficiency and quality of data fusion.

[0004] In a first aspect, embodiments of this application provide a multi-source data fusion method based on trustworthiness configuration, including: The raw data from each data source is standardized to obtain standardized data for each data source. For each of the data sources, determine the credibility score of each data item of the standardized data in that data source; In response to the data fusion operation of the received target data, for each data item of the target data, the target data source of the data item is determined based on the confidence score of the data item in each data source; The data items corresponding to each target data source are merged to obtain merged data.

[0005] Secondly, embodiments of this application also provide a multi-source data fusion device based on trust configuration. The device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a multi-source data fusion method based on trust configuration as described above.

[0006] Thirdly, embodiments of this application also provide a computer storage medium storing computer-executable instructions, which, when executed, implement a multi-source data fusion method based on trust configuration as described above.

[0007] The multi-source data fusion method, device, and medium based on trustworthiness configuration provided in this application have the following beneficial effects: First, the raw data from each data source is standardized to obtain standardized data for each data source. This standardization unifies the data format, providing a foundation for subsequent data processing. Next, for each data source, the reliability of each data item in the standardized data is evaluated, resulting in a reliability score for each data item. This achieves refined evaluation at the data item level and provides a reference for subsequent data fusion. Then, in response to the data fusion operation of the received target data, for each data item in the target data, the target data source is determined based on the data item's reliability score in each data source. By independently selecting the optimal source for each data item based on its reliability score, the generated fused data has higher reliability. Finally, the data items corresponding to each target data source are fused to obtain fused data. By fusing the data items with the highest reliability, data with high accuracy and reliability can be obtained. Attached Figure Description

[0008] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart of a multi-source data fusion method based on trust configuration is provided for embodiments of this application; Figure 2 This is a schematic diagram of the internal structure of a device provided in an embodiment of this application. Detailed Implementation

[0009] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0010] It is understood that in the embodiments of this disclosure, data related to user information (such as user names) is involved. When the embodiments of this disclosure are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0011] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. The terminology used herein is for the purpose of describing embodiments of this disclosure only and is not intended to be limiting of this disclosure.

[0012] In the following description, the terms “first, second, ...” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first, second, ...” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0013] Figure 1 This document presents a flowchart illustrating a multi-source data fusion method based on trust configuration, provided for one or more embodiments within this specification. This method can be applied to various data fusion scenarios, such as enterprise core data governance (enterprises extracting customer or product data from different systems), financial risk control and credit reporting (banks integrating internal and external data), smart cities and public management (integrating data from different government departments), and healthcare (integrating patient records from different hospitals). Certain input parameters or intermediate results in the process can be manually adjusted to help improve accuracy.

[0014] This application provides a multi-source data fusion method based on trust configuration. It should be noted that the execution entity in these embodiments can be a server or any terminal device with data processing capabilities. For example, the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal device can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, in-vehicle terminal, etc., but is not limited to these.

[0015] like Figure 1 As shown in the figure, the multi-source data fusion method based on trust configuration provided in this application embodiment specifically includes the following steps: Step 101: Standardize the raw data of each data source to obtain standardized data for each data source.

[0016] It should be noted that standardization refers to the process of converting data from different data sources, with varying formats and ambiguous meanings, into a standardized form that follows unified specifications, has a consistent format, and a clear meaning. This can include format standardization, value range / encoding standardization, structural standardization, business meaning standardization, etc., without specific limitations here.

[0017] As an example, suppose user A's information is scattered across four different data sources: an e-commerce website, offline stores, the official app, and the customer service system. Through data collection, we can obtain the following information for user A on the e-commerce website: "Name: Ms. Wang, Mobile Number: 13812345678, Savings Card Balance: ¥580, Purchase Preferences: [Beauty, Clothing], Last Purchase Date: 2023-05-15"; for offline stores: "Name: Wang Nv Shi, Mobile Number: (+86)13812345678, Savings Card Balance: ¥580, Purchase Preferences: Handbags, Shoes, Last Purchase Date: 15-May-2023"; for the official app: "Name: Ms. Wang, Mobile Number: 13812345678, Savings Card Balance: ¥580, Purchase Preferences: Clothing, Accessories, Last Purchase Date: 20230515"; and for the customer service system: "Name: Wang..." The data source for each data source is: "Name: Ms. Wang, Mobile Number: 138-1234-5678, Savings Card Balance: 600 RMB, Spending Preference: Gift Cards, Last Purchase Date: Last Week". The data is then standardized according to the following rules: "Name: 2-3 Chinese characters, Mobile Number: 11 digits without symbols, Savings Card Balance: Pure numbers (RMB) without units, Spending Preference: Chinese phrases separated by commas, Last Purchase Date: 'Year-Month-Day' format". The raw data from each data source is then standardized according to these rules. The standardized information for user A from the e-commerce website is: "Name: Ms. Wang, Mobile Number: 13812345678, Savings Card Balance: 580 RMB, Spending Preference: 'Beauty, Clothing'". The standardized data for user A, after processing at offline stores (name: Ms. Wang, mobile number: 13812345678, savings card balance: 580, consumption preference: 'handbags, shoes', last consumption date: 2023-05-15'), and after processing on the official APP (name: Ms. Wang, mobile number: 13812345678, savings card balance: 580, consumption preference: 'clothing, accessories', last consumption date: 2023-05-15'), and after processing on the customer service system (name: Mr. Wang, mobile number: 13812345678, savings card balance: 600, consumption preference: 'gift cards', last consumption date: 2023-05-18'), yields standardized data for each data source.

[0018] Step 102: For each of the data sources, determine the credibility score of each data item of the standardized data in the data source.

[0019] It should be noted that a data item is the smallest indivisible unit of information in the data. For example, in a customer information record, "name", "gender", "date of birth" and "contact number" are each an independent data item. The credibility score is a quantitative value used to dynamically and objectively measure the reliability of a data item in a data source in meeting business needs at the current business point. The credibility score can be evaluated from a single overall dimension or from multiple dimensions, and no specific limitation is made here.

[0020] As an example, suppose we evaluate the credibility score of the data item "Last Consumption Date" from different data sources. We evaluate the credibility score of the current data source based on the most recent update time. For example, if the current date is May 16th, the update time of data source 1 (e-commerce website) is May 15th, 2023 (i.e., one day ago), and the credibility score is 95. Similarly, the update date of data source 2 (official store) is May 10th, 2023 (i.e., six days ago), and the credibility score is 85. The update time of data source 3 (official APP) is May 13th, 2023 (i.e., three days ago), and the credibility score is 90. The update time of data source 4 (customer service system) is May 13th, 2023 (i.e., three days ago), and the credibility score is 90. We evaluate the credibility of each data item from each data source in this way to obtain the corresponding data source credibility score.

[0021] In some embodiments, determining the credibility score of each data item of the standardized data in the data source in step 102 above can be achieved as follows: For each data item of the standardized data, perform the following processing respectively: based on a first rule, evaluate the accuracy of the data item to obtain an accuracy score; based on a second rule, evaluate the integrity of the data item to obtain an integrity score; based on a third rule, evaluate the timeliness of the data item to obtain a timeliness score; and fuse the accuracy score, the integrity score, and the timeliness score to obtain the credibility score of the data item.

[0022] Thus, by deconstructing credibility into three independent and quantifiable evaluation dimensions—accuracy, completeness, and timeliness—and setting clear calculation rules for each dimension (rules one, two, and three), a comprehensive quantitative score is obtained through fusion. This overcomes the drawbacks of traditional data quality assessment, such as strong subjectivity and inconsistent standards. It transforms data quality from a vague and relative concept into a precise and comparable engineering indicator, thereby achieving measurability of data quality. Furthermore, by designing the evaluation rules for accuracy, completeness, and timeliness as independent modules and dynamically adjusting the weights during the fusion phase, the evaluation model can flexibly adapt to the differentiated data quality requirements of different business scenarios, thus ensuring the accuracy of the data item reliability score.

[0023] It should be noted that the first rule is used to determine the degree to which the value of a data item conforms to its actual situation or a generally accepted standard. For example, the first rule could be a format conformity rule (checking whether the value of a data item conforms to a predefined format specification), a range reasonableness rule (checking whether numerical data falls within a reasonable business range), an external reference rule (comparing a data item to a highly trusted external "gold standard" or authoritative data source), or an internal consistency rule (checking whether the logical relationship between the data item and other data items in the same record holds). The second rule is used to measure whether a data item has a value or the degree of missing values. For example, the second rule could be a non-null value ratio rule (calculating the number of records in a dataset (such as a table or view) where a data item has a value). The scoring rules include the proportion of total records, the effective value ratio rule (going a step further than non-null values, requiring not only "values" but also "meaningful values"), etc.; the third rule is used to measure the "newness" of data items and whether they meet the business requirements for data freshness. For example, the third rule can be the timestamp step decay rule (based on the difference between the "last update time" of the data item and the current time, it falls into different time intervals and is given different scores), the exponential decay rule (the score decreases exponentially over time), the update frequency rule (to assess whether the data update frequency of the data source itself meets business expectations), etc.; the scoring fusion method can be the linear weighted average method, the multiplicative fusion method, the maximum / minimum value method, the machine learning model method, etc., without specific limitations here.

[0024] As an example, taking the evaluation of the reliability of common vehicle data items as an example, assume that the first rule (accuracy rule) is "the license plate number format must be correct and the vehicle identification number must be 17 digits", the second rule (completeness rule) is "whether both the license plate number and the vehicle identification number fields have values", and the third rule (timeliness rule) is "the time since the last vehicle update". The standardized vehicle data item of data source 1 (core insurance policy system) is "License plate: Jing N8XXXX, Vehicle identification number: LSVAM418...". According to the first rule for accuracy evaluation, it can be seen that both the license plate number and the vehicle identification number formats are correct, and the corresponding accuracy score is 100. According to the second rule for completeness evaluation, it can be seen that both fields have values, and the corresponding completeness score is 100. According to the third rule for timeliness evaluation, it can be seen that the information was entered when insured 3 years ago, and the corresponding timeliness score is 30. The weights corresponding to accuracy, completeness, and timeliness are 60%, 10%, and 30% respectively. Through weighted fusion, the reliability score of the current data item in data source 1 can be obtained as 79; similarly, the standardized vehicle data item of data source 2 (claims history system) is "License plate: Jing N8XXXX, Vehicle identification number: LSVAM418...". According to the first rule for accuracy evaluation, it can be seen that both the license plate number and the vehicle identification number formats are correct, and the corresponding accuracy score is 100. According to the second rule for completeness evaluation, it can be seen that both fields have values, and the corresponding completeness score is 100. According to the third rule for timeliness evaluation, it can be seen that the information was confirmed during a claim 1 year ago, and the corresponding timeliness score is 60. The weights corresponding to accuracy, completeness, and timeliness are 60%, 10%, and 30% respectively. Through weighted fusion, the reliability score of the current data item in data source 2 can be obtained as 88; the standardized vehicle data item of data source 3 (customer behavior APP) is "License plate: Jing N8XXXX, Vehicle identification number: [empty]". According to the first rule for accuracy evaluation, it can be seen that the license plate number is correct and the vehicle identification number is empty, and the accurate information is incomplete, and the corresponding accuracy score is 50. According to the second rule for completeness evaluation, it can be seen that the vehicle identification number is missing, and the corresponding completeness score is 50. According to the third rule for timeliness evaluation, it can be seen that the information was uploaded one week ago, and the corresponding timeliness score is 95. The weights corresponding to accuracy, completeness, and timeliness are 60%, 10%, and 30% respectively. Through weighted fusion, the reliability score of the current data item in data source 3 can be obtained as 63.5; the standardized vehicle data item of data source 4 (agent channel system) is "License plate: Jing NBXXXX, Vehicle identification number: Vehicle identification number: LSVAM418...According to the first rule, the accuracy assessment shows that the license plate number is incorrect and does not match the vehicle identification number (VIN), resulting in an accuracy score of 20. According to the second rule, the completeness assessment shows that both fields have values, resulting in a completeness score of 100. According to the third rule, the timeliness assessment shows that the information was updated by the agent last month, resulting in a timeliness score of 70. The weights for accuracy, completeness, and timeliness are 60%, 10%, and 30%, respectively. Through weighted fusion, the reliability score of the current data item in data source 4 is 43.

[0025] In some embodiments, determining the credibility score of each data item of the standardized data in the data source in step 102 above can be achieved by the following method: for each data item of the standardized data, the following processing is performed respectively: feature extraction is performed on the data item to obtain a first feature corresponding to the data item; based on the first feature, credibility prediction is performed on the data item to obtain the credibility score of the data item.

[0026] Thus, through feature extraction, the original data items are transformed into a feature vector (first feature) that can more comprehensively and profoundly describe their quality attributes. This expands the description of data item quality from a limited, pre-defined set of dimensions (such as accuracy and completeness) to a high-dimensional, non-linear feature space. This allows the model to better capture implicit quality patterns that are difficult to quantify using traditional rules, thus addressing the limitations of traditional rule engines when facing complex data patterns. Then, using the extracted first feature, a pre-trained machine learning or deep learning prediction model is used to output the final credibility score. This enables intelligent and adaptive evaluation logic, significantly reducing evaluation bias caused by incomplete or inaccurate rule formulation, and improving the accuracy and generalization ability of the score.

[0027] As an example, taking the assessment of the reliability of contact address data items as an example, suppose the standardized contact address data item 1 of data source 1 (core policy system) is "No. 1, C Street, B District, City A, Postal Code: 100000". By extracting features from data item 1, we can obtain the first feature F1 corresponding to the current data item 1. Based on the first feature F1, we can predict the credibility of data item 1 and obtain the credibility score S1 of data item 1. Similarly, the standardized contact address data item 2 of data source 2 (claims history system) is "[empty]". By extracting features from data item 2, we can obtain the first feature F2 corresponding to the current data item 2. Based on the first feature F2, we can predict the credibility of data item 2 and obtain the credibility score S2 of data item 2. The standardized contact address data item 3 of data source 3 (customer behavior APP) is "No. 1, C Street, D District, City A, Postal Code: 100000". The postal code is 100102. By extracting features from data item 3, we can obtain the first feature F3 corresponding to the current data item 3. Based on the first feature F3, we can predict the credibility of data item 3 and obtain the credibility score S3. The standardized contact address data item 4 of data source 4 (agent channel system) is "No. 1, Street C, District B, City A, Postal Code: 100010". By extracting features from data item 4, we can obtain the first feature F4 corresponding to the current data item 4. Based on the first feature F4, we can predict the credibility of data item 4 and obtain the credibility score S4.

[0028] In some embodiments, credibility prediction is obtained through a trained rating prediction model. Before feature extraction is performed on the data items to obtain the first feature corresponding to the data items, the following processing may also be performed: feature extraction is performed on each data item of the sample data to obtain the second feature corresponding to each data item; credibility prediction is performed on the data items based on the second feature using the initialized rating prediction model to obtain the credibility prediction score of the data items; the model loss is determined based on the difference between the credibility prediction score of the data items and the credibility reference score of the data items, and the initialized rating prediction model is updated based on the model loss to obtain the trained rating prediction model.

[0029] Thus, by introducing a supervised learning loop, the internal logic of the prediction model no longer relies on rules or weights subjectively defined by experts. Instead, it autonomously discovers mathematical mapping relationships that best fit objective facts through statistical learning of a large amount of real, labeled data. This ensures that the final credibility score has a solid objective data foundation, eliminates systematic errors caused by personal experience or bias, and provides a highly reliable and neutral evaluation standard for subsequent data fusion decisions. Furthermore, through an optimization mechanism centered on model loss, a clear and quantifiable optimization direction is provided for the model training process. This ensures that each iteration of the model is effective and efficient, avoids blind parameter adjustments, and enables the model's predictive ability to quickly converge to a local or global optimum, thereby significantly improving the accuracy of the credibility score.

[0030] It should be noted that the model loss can be calculated using the cross-entropy loss function, log loss function, and focus loss function, and no specific limitation is made here.

[0031] As an example, suppose data item 1 of the sample data is "No. 1, Street C, District B, City A, Postal Code: 100000". By extracting features from data item 1, we can obtain the second feature F corresponding to the current data item 1. 21 And through the initialized rating prediction model, based on the second feature F 21 By predicting the credibility of data item 1, we can obtain the credibility prediction score S for data item 1. 21 Then, the confidence prediction score S is calculated using the cross-entropy loss function. 21 Difference between S and credibility reference score 20 The model loss is obtained, which represents the degree of difference between the model prediction and the actual situation. Finally, the model parameters are updated by backpropagation based on the model loss, thus obtaining the trained rating prediction model.

[0032] Here's an explanation of backpropagation: The data items of the sample data are input into the input layer of the neural network model (classification model), pass through the hidden layer, and finally reach the output layer to output the result. This is the forward propagation process of the neural network model. Since there is an error between the output result of the neural network model and the actual result, the error between the result and the actual value is calculated and propagated back from the output layer to the hidden layer until it reaches the input layer. During the backpropagation process, the values ​​of the model parameters are adjusted according to the error. The above process is iterated continuously until convergence.

[0033] Taking the cross-entropy loss function in this embodiment as an example, the server determines the model loss based on the cross-entropy loss function, backpropagates the model loss from the output layer of the scoring prediction model, and backpropagates the model loss layer by layer. When the model loss reaches each layer, the gradient (that is, the partial derivative of the loss function with respect to the parameters of each layer) is solved in combination with the propagated model loss, and the corresponding gradient value of the parameters of each layer is updated.

[0034] Step 103: In response to the data fusion operation of the received target data, for each data item of the target data, determine the target data source of the data item based on the credibility score of the data item in each data source.

[0035] In some embodiments, determining the target data source for the data item based on the credibility score of the data item in each of the data sources in step 103 above can be achieved by comparing the credibility scores of the data item in each of the data sources and taking the data source with the highest credibility score as the target data source for the data item.

[0036] In this way, by directly comparing the credibility scores of versions from different data sources and taking the data source with the highest score as the target data source, it can be ensured that each data fusion is the optimal choice based on all available information under a given evaluation system. It can intelligently filter out the best version of each data item from multiple data sources of varying quality, making full use of the local advantages of the data sources, thereby improving the quality of subsequent data fusion.

[0037] As an example, suppose that through credibility assessment, data item 1 has a credibility score of 85 in data source 1, a credibility score of 80 in data source 2, a credibility score of 95 in data source 3, and a credibility score of 60 in data source 3. By comparing the credibility scores, it can be seen that data item 1 has the highest credibility score in data source 3, so data source 3 is taken as the target data source for data item 1.

[0038] In some embodiments, the following processing may also be performed: in response to the data item having a credibility score less than a scoring threshold in each of the data sources, a prompt message corresponding to the data item is generated to prompt the user to update the data item.

[0039] In this way, by setting a monitoring mechanism with scoring thresholds, proactive risk warnings for data quality can be achieved. Alarms can be issued in time before low-quality data is adopted and pollutes downstream business systems, and potential risk scenarios of unreliable data items can be discovered in a timely manner to ensure the quality of subsequent data fusion.

[0040] It should be noted that the scoring threshold can be a preset value or it can be dynamically determined based on the actual situation; no specific limitation is made here.

[0041] As an example, suppose the system has obtained user A's data item M from the core policy system (data source 1), the customer behavior APP (data source 2), and the agent channel system (data source 3), and calculated the credibility score of data item M in each data source. The credibility score of data item M in data source 1 is 75, the credibility score of data item M in data source 2 is 60, and the credibility score of data item M in data source 3 is 50. The scoring threshold is 80. By comparison, it can be seen that the credibility score of data item M in each data source is less than the scoring threshold. At this time, a prompt message corresponding to the current data item M is generated to prompt the user to update the data item M. For example, the prompt message could be "User A's data item M has a credibility score lower than the scoring threshold in all data sources. Please update the information of data item M in time."

[0042] In some embodiments, after determining the credibility score of each data item of the standardized data in the data source, the following processing may also be performed: storing the credibility score of each data item of the standardized data in each data source and the evaluation time in a database; in response to the interval between the current time and the evaluation time being longer than a duration threshold, re-determining the credibility score of each data item of the standardized data in each data source, and updating the credibility score and evaluation time in the database.

[0043] Thus, by introducing a monitoring mechanism of "evaluation time" and "duration threshold", it can be ensured that the score can reflect the latest status of the data source and avoid making wrong fusion decisions due to reliance on outdated evaluation results. In particular, for data items that are sensitive to timeliness, it can significantly improve the accuracy and reliability of fusion results in dynamic business environments, thereby enhancing the real-time value and availability of data assets.

[0044] As an example, suppose data item 1 (user account: user12345) in standardized data has a credibility score of 85 in data source A, and the evaluation time is 2023-10-01 10:00:00. Data item 1, along with the corresponding credibility score and evaluation time in data source A, is stored in the database. The update rule is that the credibility score needs to be re-evaluated every 7 days (duration threshold). The current time is 2023-10-08 10:05:00. Calculation shows that the interval between the current time and the evaluation time of data item 1 in data source A is 7 days and 5 minutes, which is greater than 7 days (duration threshold). Therefore, the credibility of data item 1 in data source A needs to be re-evaluated. The new credibility score is 72, and the new evaluation time is 2023-10-08 10:05:00. The new credibility score and evaluation time are then used to replace the previous credibility score and evaluation time of data item 1 in data source A in the database.

[0045] Step 104: Merge the data items corresponding to each target data source to obtain merged data.

[0046] As an example, suppose we need to obtain information about VIP customer Ms. Wang. We extract Ms. Wang's name (data item) from the e-commerce website (target data source), resulting in "Ms. Wang"; we extract her mobile phone number (data item) from the customer service system (target data source), resulting in "13812345678"; we extract her savings card balance (data item) from the e-commerce website (target data source), resulting in "580" yuan; we extract her preference tags (data item) from the official APP (target data source), resulting in "Clothing, Accessories"; we extract her last purchase date (data item) from the customer service system (target data source), resulting in "2023-05-18"; and we extract her membership level (data item) from the core management system (target data source), resulting in "VIP". Then, by merging these data items, we obtain the following merged data: "Name: Ms. Wang, Mobile Phone Number: 13812345678, Savings Card Balance: 580, Preference Tags: Clothing, Accessories, Last Purchase Date: 2023-05-18, Membership Level: VIP".

[0047] In some embodiments, after performing step 104, the following processing may also be performed: generating a fusion record of the fused data based on the fusion time, each of the target data sources, and the credibility score of the data item corresponding to each target data source.

[0048] In this way, by atomically recording key decision-making elements, the fusion process can be made completely transparent, providing a fully transparent and traceable operational view for data governance. Furthermore, the fusion record can accurately record the specific process of data fusion, which can accelerate the location and attribution of data problems, thereby providing a reliable reference for accurate remediation and responsibility definition.

[0049] As an example, suppose the task ID for the current data fusion is TASK-20230519-001, the fusion object ID is CUS-WANG-NV-SHI, and the fusion time is 2023-05-19 14:30:00. The record information for the name data item is "Data Item: Name, Fusion Value: Ms. Wang, Target Data Source: E-commerce Website, Credibility Score: 99". The record information for the mobile phone number data item is "Data Item: Mobile Phone Number, Fusion Value: 13812345678, Target Data Source: Customer Service System, Credibility Score: 98". The record information for the savings card balance data item is "Data Item: Savings Card Balance, Fusion Value: 580, Target Data Source: E-commerce Website, Credibility Score: 99". The last consumption date data item corresponds to... The record information is as follows: "Data Item: Last Consumption Date, Fusion Value: 2023-05-18, Target Data Source: Customer Service System, Credibility Score: 96". The record information corresponding to the Membership Level data item is as follows: "Data Item: Membership Level, Fusion Value: VIP, Target Data Source: Core Management System, Credibility Score: 95". Based on the above record information, the corresponding fusion record generated is as follows: "Task ID: TASK-20230519-001; Fusion Object: CUS-WANG-NV-SHI; Fusion Time: 2023-05-19". 14:30:00; Data items: Name, Integration value: Ms. Wang, Target data source: E-commerce website, Credibility score: 99; Data items: Mobile phone number, Integration value: 13812345678, Target data source: Customer service system, Credibility score: 98; Data items: Savings card balance, Integration value: 580, Target data source: E-commerce website, Credibility score: 99; Data items: Last consumption date, Integration value: 2023-05-18, Target data source: Customer service system, Credibility score: 96; Data items: Membership level, Integration value: VIP, Target data source: Core management system, Credibility score: 95.

[0050] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0051] In some embodiments, assume that user A has had a traffic accident and submitted a claim application to the insurance company. To accurately assess the risk, the system needs to quickly generate a latest and most accurate information view of user A. The user information is mainly stored in three independent systems within the company, mainly including the core policy system (recording the basic information of user A when applying for insurance, accurate but possibly outdated), the claim history system (recording the past claim situations of user A, with a medium update frequency), and the customer behavior APP system (the information independently updated by user A on the mobile APP, the most timely but possibly incomplete); First, obtain the original data of user A from the three systems respectively. The original data 1 obtained from the core policy system is "Name: Mr. Li, Common vehicle: Beijing N8XXXX, Contact address: No. 15, Q Road, N District, M City", the original data 2 obtained from the claim history system is "Name: Li Xian Sheng, Common vehicle: Beijing N8XXXX, Contact address: [empty]", and the original data 3 obtained from the customer behavior APP is "Name: Mr. Li, Common vehicle: [empty], Contact address: W District, M City, Zip code: 100102"; Subsequently, perform standardization processing on the data, and the standardized data 1 corresponding to the original data 1 can be obtained as "Name: Mr. Li, Common vehicle: Beijing N8XXXX, Contact address: No. 15, Q Road, N District, M City, Zip code: 100083", the standardized data 2 corresponding to the original data 2 is "Name: Mr. Li, Common vehicle: Beijing N8XXXX, Contact address: [empty]", and the standardized data 3 corresponding to the original data 3 is "Name: Mr. Li, Common vehicle: [empty], Contact address: W District, M City, Zip code: 100102"; Then, conduct a credibility assessment for each data item of each data source. The credibility scores corresponding to each data item in the standardized data 1 can be obtained as "Name: Mr. Li; Common vehicle: Beijing N8XXXX, Credibility score: 85; Contact address: No. 15, Q Road, N District, M City, Zip code: 100083, Credibility score: 75", the credibility scores corresponding to each data item in the standardized data 2 are "Name: Mr. Li; Common vehicle: Beijing N8XXXX, Credibility score: 88; Contact address: [empty], Credibility score: 0"; The credibility scores corresponding to each data item in the standardized data 3 are "Name: Mr. Li; Common vehicle: [empty], Credibility score: 20; Contact address: W District, M City, Zip code: 100102, Credibility score: 98"; Then, for each data item, compare the credibility scores corresponding to different data sources, and the data source with the highest credibility score for the common vehicle data item can be obtained as the claim history system (target data source), and the data source with the highest credibility score for the contact address data item is the customer behavior APP (target data source); Finally, extract data from each target data source and fuse them to obtain a final and high-quality fused data as "Name: Mr. Li; Common vehicle: Beijing N8XXXX, Target data source: Claim history system (Credibility score: 88);Contact address: M City, W District, Postcode 100102; Target data source: Customer behavior APP (credibility score: 98).

[0052] The multi-source data fusion method based on credibility configuration provided in this application achieves data fusion by configuring the credibility of data items. It can eliminate the dependence on manual intervention, build a full-link fusion history, facilitate traceability, and enable timed iteration of credibility without frequent modification of fusion rule code, thereby reducing operation and maintenance costs.

[0053] The above are embodiments of the method proposed in this application. Based on the same inventive concept, embodiments of this application also provide a multi-source data fusion device based on trustworthiness configuration, the structure of which is as follows: Figure 2 As shown.

[0054] Figure 2 This is a schematic diagram of the internal structure of a device provided in an embodiment of this application. Figure 2 As shown, the device includes: At least one processor 201; And a memory 202 that is communicatively connected to at least one processor; The memory 202 stores instructions that can be executed by at least one processor. The instructions are executed by at least one processor 201 to enable at least one processor 201 to perform the steps of the method corresponding to any of the above embodiments.

[0055] Some embodiments of this application provide corresponding to Figure 1 A non-volatile computer storage medium stores computer-executable instructions, which are configured to perform the steps of the method corresponding to any of the above embodiments.

[0056] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments for IoT devices and media are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0057] The systems, media, and methods provided in this application are one-to-one correspondences. Therefore, the systems and media also have similar beneficial technical effects as their corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the systems and media will not be repeated here.

[0058] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0059] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0060] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0061] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0062] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0063] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0064] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0065] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0066] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A multi-source data fusion method based on trustworthiness configuration, characterized in that, The method includes: The raw data from each data source is standardized to obtain standardized data for each data source. For each of the data sources, determine the credibility score of each data item of the standardized data in that data source; In response to the data fusion operation of the received target data, for each data item of the target data, the target data source of the data item is determined based on the confidence score of the data item in each data source; The data items corresponding to each target data source are merged to obtain merged data.

2. The method according to claim 1, characterized in that, Determining the credibility score of each data item in the standardized data from the data source includes: For each data item in the standardized data, the following processing is performed: Based on the first rule, the accuracy of the data item is evaluated to obtain the accuracy score of the data item; Based on the second rule, the integrity of the data item is assessed to obtain the integrity score of the data item; Based on the third rule, the timeliness of the data item is evaluated to obtain a timeliness score for the data item; The accuracy score, the completeness score, and the timeliness score are combined to obtain the credibility score of the data item.

3. The method according to claim 1, characterized in that, Determining the credibility score of each data item in the standardized data from the data source includes: For each data item in the standardized data, the following processing is performed: Feature extraction is performed on the data item to obtain the first feature corresponding to the data item; Based on the first feature, the credibility of the data item is predicted to obtain the credibility score of the data item.

4. The method according to claim 3, characterized in that, The credibility prediction is obtained through a trained rating prediction model; the method further includes: For each data item in the sample data, feature extraction is performed to obtain the second feature corresponding to each data item; Based on the second feature, the credibility prediction of the data item is obtained by using the initialized rating prediction model to predict the credibility of the data item. Based on the difference between the confidence prediction score of the data item and the confidence reference score of the data item, the model loss is determined, and the initialized rating prediction model is updated based on the model loss to obtain the trained rating prediction model.

5. The method according to claim 1, characterized in that, The step of determining the target data source for a data item based on its credibility score in each data source includes: The credibility scores of the data items are compared in each of the data sources, and the data source with the highest credibility score is selected as the target data source for the data item.

6. The method according to claim 5, characterized in that, The method further includes: In response to the fact that the credibility score of the data item in each of the data sources is less than the scoring threshold, a prompt message is generated for the corresponding data item to prompt the user to update the data item.

7. The method according to claim 1, characterized in that, The method further includes: The credibility score and evaluation time of each data item of the standardized data in each of the data sources are stored in the database; In response to the interval between the current time and the evaluation time being longer than a duration threshold, the credibility score of each data item of the standardized data in each data source is re-determined, and the credibility score and evaluation time in the database are updated.

8. The method according to claim 1, characterized in that, The method further includes: Based on the fusion time, each of the target data sources, and the credibility score of the data items corresponding to each target data source, a fusion record of the fused data is generated.

9. A multi-source data fusion device based on trust configuration, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-8.

10. A computer storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, they implement the method as described in any one of claims 1-8.

Images