A multi-source data fusion management method and system based on behavior points

Abstract

The application discloses a kind of multi-source data fusion governance method and system based on behavior integral, it is related to multi-source data fusion governance technical field, first adaptation multi-source heterogeneous data of multiple collection protocols access each business channel, verify the subject compliance qualification of data source provider, only allow the subject access collection link to pass through qualification verification;Collect the behavior data related to data source, single data in whole link, build double-layer dimension behavior integral calculation model, respectively calculate the subject credit integral of each data source subject and the data quality integral of each single data, weighted to obtain the comprehensive behavior integral of corresponding data.The application can dynamically adapt to the quality change of data source, enrich the evaluation dimension of conflict collection, greatly improve the accuracy and management efficiency of multi-source data fusion, reduce the workload of manual intervention, reduce the repeated inflow of low-quality data from the source, reduce the whole-link management cost.

Description

Technical Field

[0001] This invention relates to the field of multi-source data fusion governance technology, and in particular to a multi-source data fusion governance method and system based on behavior integrals. Background Technology

[0002] With the rapid development of digital government, inclusive finance, and industrial digitalization, the integration and governance of multi-source heterogeneous data has become a core foundation for data assetization applications. Currently, various industries need to integrate structured, semi-structured, and unstructured data from different entities such as government departments, commercial service providers, and publicly available collection channels to eliminate data silos, resolve data conflicts, and generate unified data assets with high consistency and reliable quality to support upper-level applications such as business decision-making, public services, and risk prevention and control. The industry's demand for the accuracy, governance efficiency, and quality controllability of multi-source data integration continues to increase.

[0003] Currently, mainstream multi-source data fusion governance solutions fall into two main categories. The first category is based on fixed rules, where the core logic involves pre-setting fixed data source priorities and field validation rules. If data conflicts occur within the same dimension during fusion, the data from the data source with the higher preset priority is directly adopted, or the data deemed compliant by the preset rules is matched. The advantage of this solution is its simple implementation logic and clear rules, allowing for rapid adaptation to small-scale fusion scenarios with a limited number of data sources and standardized data. It was widely used in early data integration within a single industry. However, this solution has significant drawbacks: the fixed rule configuration cannot dynamically adapt to changes in data source quality. If the quality of subsequent data from a previously high-priority data source continues to decline, it will still be prioritized, reducing the accuracy of the fusion results. Furthermore, fixed rules cannot cover all complex conflict scenarios, requiring manual intervention for many unknown conflicts, resulting in extremely low governance efficiency.

[0004] The second type is a fusion governance solution based on single-data-item quality assessment. Its core logic involves quantitatively scoring only the completeness, format compliance, and value range reasonableness of each individual piece of data to be merged, with higher-scoring data pieces being prioritized for acceptance. The advantage of this solution is its greater flexibility compared to fixed-rule solutions, allowing for differentiated processing based on the quality differences of individual data pieces and adapting to scenarios with complex data source categories. However, this solution only assesses the surface quality of a single data piece and does not consider the long-term historical behavior of the data source provider. If a data source provider has been uploading low-quality or false data for a long time, even if only a single data piece meets the surface verification rules, it may still be mistakenly accepted. Furthermore, the lack of a constraint mechanism on the data source provider means that providers of low-quality data are not restricted and can repeatedly upload low-quality data, significantly increasing the overall governance cost.

[0005] Neither of the two existing solutions has established a full-link behavioral constraint and dynamic feedback mechanism for the data source entity. The basis for conflict acceptance lacks comprehensive support, resulting in insufficient accuracy of the fusion results, repeated inflow of low-quality data, and high governance costs. They cannot meet the needs of scenarios with extremely high data quality requirements, such as financial risk control and government services. The industry urgently needs a more efficient and reliable multi-source data fusion governance solution. Summary of the Invention

[0006] This invention proposes a multi-source data fusion governance method and system based on behavior integrals to solve the problems mentioned in the prior art, such as the rigidity of existing multi-source data fusion rules, lack of long-term constraints on data source subjects, single dimension of conflict acceptance criteria, low fusion accuracy, and repeated inflow of low-quality data.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: a multi-source data fusion governance method based on behavior integrals, applied to the governance, fusion, and database management scenarios of multi-source structured, semi-structured, and unstructured data, comprising the following steps: S1. By adapting to various collection protocols such as JDBC, RESTful API, FTP, and web crawlers, multi-source heterogeneous data from various business channels is collected. The multi-source heterogeneous data specifically includes relational structured tables generated by business systems, semi-structured log data, and unstructured data such as audio, video, documents, and images. The entity identity information of each data source provider is collected synchronously. The entity identity information includes the unified social credit code, data access filing number, and entity qualification certificate number. Entity qualification verification is completed before collection, and only entities with legal data provision qualifications are allowed to access the data collection link. S2. Perform preliminary cleaning and deduplication processing on the collected multi-source heterogeneous data, specifically including null value filtering, illegal format verification, and deduplication based on SimHash algorithm for duplicate data fingerprint comparison. Extract the metadata tags corresponding to each data, which include the data's domain, generation time, attribute dimension, and security level. Synchronously collect end-to-end behavioral data, which includes the data source's historical data upload timeliness records, data compliance verification results, historical data error rate, data user feedback, cumulative data call frequency, cross-data source cross-validation matching results, and data security compliance records. S3. Construct a two-dimensional behavioral score calculation model. Based on the collected full-link behavioral data, calculate the subject credit score of each data source subject and the data quality score of each individual data. Calculate the comprehensive behavioral score value of each data according to the weighted average of the subject credit score (60%) and the individual data quality score (40%). S4. Based on the comprehensive behavior score of each data, assign a corresponding data fusion weight to each data. The fusion weight is the comprehensive behavior score of a single data divided by the sum of the comprehensive behavior scores of all data to be fused under the same attribute dimension. Perform fusion calibration on multi-source heterogeneous data according to the fusion weight. If there is a data conflict of the same attribute dimension from different data sources, the data corresponding to the data source with the higher comprehensive behavior score is adopted first. Before adoption, a second cross-verification is performed on the high score data. After the verification is passed, the attribute value is stored in the database. S5. After the multi-source data after fusion calibration is standardized by performing field mapping and format conversion according to the preset unified metadata standard, it is stored in a unified distributed database. At the same time, based on the new behavioral data generated in this fusion governance process, including data conflict records, user feedback records, and cross-validation results, the subject credit score of the corresponding data source subject and the data quality score of the corresponding single data are dynamically updated. The updated comprehensive behavioral score value is synchronized to the fusion governance cache pool in real time and directly applied to all subsequent multi-source data fusion governance processes.

[0008] Furthermore, the main credit score in step S3 is calculated using a lifetime cumulative system. The initial score is uniformly set to 100 points, with an upper limit of 200 points and a lower limit of 0 points. For each instance of data field error, non-compliant data content, or data update delay exceeding the agreed time limit, 5 to 30 points are deducted. The deduction value is positively correlated with the severity of the error. For each instance of full data verification passing, valid positive feedback from data users, or 100% matching of cross-data source cross-validation, 2 to 5 points are added. The points are accumulated to the upper limit and then no longer added.

[0009] Furthermore, when data conflicts occur in the same attribute dimension in step S4, after the high-scoring data is accepted and the low-scoring data is marked as discarded, a point deduction instruction is generated simultaneously to deduct the credit score of the data source subject that submitted the conflicting low-quality data. The deduction value is positively correlated with the degree of impact of the fusion error caused by the low-quality data. If the low-quality data belongs to a core business field, 10 to 25 points are deducted; if it belongs to a non-core auxiliary field, 2 to 8 points are deducted. The deduction record is simultaneously stored in the credit file of the corresponding subject as the basis for subsequent qualification review.

[0010] Furthermore, before performing multi-source data fusion calibration in step S4, entity alignment is first completed based on knowledge graph entity linking technology. Through attribute matching and semantic similarity calculation, all multi-source attribute data belonging to the same natural person, legal person, and item entity are collected. Then, based on the comprehensive behavioral score corresponding to each attribute data, the final attribute value of the entity is calculated by combining the business weight of each attribute.

[0011] Furthermore, the dynamic update of behavior points in step S5 adopts an event-driven real-time triggering mechanism. It asynchronously receives full-link behavior events through a message queue. When data from any data source subject is called, feedback is provided for error correction, or cross-validation is performed to check for mismatch, a high-priority points update task is immediately generated. There is no need to wait for fixed-period batch settlement. The update result is synchronized to all associated fusion task caches within 1 second.

[0012] Furthermore, it also includes a credit score warning step, which monitors the credit scores of all data source entities in real time. When the credit score of an entity falls below a preset security threshold, which can be customized according to the business scenario (the default setting is 80 points for financial scenarios and 70 points for government scenarios), the entity's access to all data sources is immediately suspended until the entity submits an explanation of the data error, a rectification plan, a test rectification dataset, and passes the full data compliance verification. In this case, a 7-day trial period is granted. If no data errors occur during the trial period, the entity's formal access rights and the initial credit score of 100 points are restored.

[0013] Furthermore, a multi-source data fusion governance system based on behavior integrals includes: The multi-source data acquisition module is used to adapt to various data acquisition protocols and access multi-source heterogeneous data from various business channels. It has a built-in entity qualification verification unit to verify the entity identity information and compliance qualifications of the data source provider, and only allows entities that have passed the qualification verification to access the data acquisition link. The behavior data acquisition module is used to connect to the full-link log system, user feedback system, and cross-source cross-validation system. It automatically pulls all full-link behavior data related to the data source and individual data, performs deduplication and normalization processing on the behavior data, and stores it in the behavior database. The points calculation module is used to build a two-dimensional behavioral points calculation model. It has a built-in configurable points rule engine and supports the customization of points calculation rules for different industries. Based on the full-link behavioral data, it calculates the main credit score of each data source entity and the data quality score of each individual data, and calculates the comprehensive behavioral points value of each data based on the weighted calculation. The fusion governance module is used to dynamically allocate data fusion weights based on the comprehensive behavior score of each data point, and perform fusion calibration on the collected multi-source heterogeneous data according to the fusion weights. It has built-in conflict handling logic. If there is a conflict between data of the same attribute dimension from different data sources, the data corresponding to the data source with the higher comprehensive behavior score will be given priority. The storage update module has a built-in standardized mapping engine that supports custom unified metadata standards. It is used to perform standardized processing on multi-source data after fusion calibration and store it in a unified distributed database. It also has a built-in points update unit that receives new behavioral data generated during the fusion governance process in real time, dynamically updates the subject credit score of the corresponding data source subject and the data quality score of the corresponding single data, and synchronizes the update results to the fusion governance cache pool in real time.

[0014] Furthermore, the points calculation module includes a main credit points calculation unit and a single data quality points calculation unit. The main credit points calculation unit is connected to the main credit file and calculates all points changes of the main entity since access throughout its life. It supports full-link backtracking of points change history. The single data quality points calculation unit scores a single data item based on four dimensions: completeness, accuracy, timeliness, and compliance, each with a weight of 25%, to obtain the quality points of the single data item.

[0015] Furthermore, the integrated governance module has a built-in conflict handling unit. When the conflict handling unit detects a data conflict of the same attribute dimension, it first determines the importance level of the conflict field. After completing the acceptance of high-scoring data, it generates a corresponding score deduction instruction based on the importance level of the conflict field and sends it to the score calculation module to execute the credit score deduction of the corresponding subject. At the same time, it generates a conflict tracing record and stores it in the data governance log for subsequent audit and verification.

[0016] Furthermore, it also includes a credit score early warning module. The credit score early warning module has a built-in configurable threshold setting unit, which supports the customization of security thresholds for different business scenarios and different security levels. It monitors the credit score of all data source entities in real time. When the entity's credit score is lower than the preset security threshold, it immediately sends a pause access command to the multi-source data acquisition module and pushes an early warning notification to the administrator. After the entity submits rectification materials and passes the verification, it automatically sends a restore access command to the multi-source data acquisition module.

[0017] Compared with existing technologies, the beneficial effects of this invention are: This invention constructs a comprehensive behavioral score system that uses a dual-weighted approach, combining entity credit scores and individual data quality scores. It quantitatively assesses data credibility based on end-to-end behavioral data, dynamically assigns corresponding fusion weights to each piece of data to be fused, and prioritizes data with higher comprehensive behavioral scores when data conflicts occur within the same dimension. This effectively solves the problems of existing fixed-rule fusion schemes, such as rigid rules, inability to adapt to dynamic changes in data source quality, and a single dimension for conflict acceptance. It significantly improves the consistency and accuracy of multi-source data fusion results, reduces the workload of manual intervention in unknown conflict scenarios, and enhances overall governance efficiency.

[0018] This invention establishes a dynamic feedback and adjustment mechanism for end-to-end behavioral data and scores. It integrates behavioral data such as data conflict records, user feedback evaluations, and cross-validation results generated throughout the governance process. This mechanism can update the credit score of the corresponding data source entity and the quality score of the corresponding single data item in real time. Entities that submit low-quality conflicting data will have their corresponding credit score deducted. When the score falls below a preset security threshold, their data access permissions will be automatically suspended. This effectively solves the problem that existing single data quality assessment schemes do not establish a long-term constraint mechanism for entities and that low-quality data repeatedly flows into the governance process. It controls the entry of low-quality data from the source of data access and significantly reduces the cost of repetitive governance throughout the entire process.

[0019] The scoring rules, weight allocation ratios, and early warning thresholds of this invention all support custom configuration, which can flexibly adapt to the integrated governance needs of different industries and different security levels. It does not require large-scale modification of the upload links of existing data sources. It can be quickly deployed by simply connecting to the full-link behavioral data collection interface. It can be widely used in multiple scenarios such as government data sharing, financial risk control data integration, and industrial data assetization, and has high industry promotion value. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating the overall process of multi-source data fusion governance based on behavior integrals proposed in this invention. Figure 2 This is a flowchart of the full-link calculation and real-time dynamic update of the behavior integral of the present invention; Figure 3 This is a flowchart of the multi-source heterogeneous data entity alignment and conflict degradation processing of the present invention; Figure 4 This is a flowchart illustrating the credit score early warning and data access permission control process of the main body of this invention; Figure 5 This is a diagram showing the architecture and module interaction of the multi-source data fusion governance system of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] Reference Figures 1 to 5 This invention discloses a multi-source data fusion and governance method based on behavior integrals, applicable to the governance, fusion, and database management of multi-source structured, semi-structured, and unstructured data. The specific implementation steps include: S1. Through a pre-encapsulated multi-protocol adaptation layer, it adapts to various collection protocols such as JDBC, RESTful API, FTP, SFTP, WebSocket, and WebMagic crawler framework to collect multi-source heterogeneous data from various business channels. The structured data specifically consists of business table data stored in relational databases such as MySQL and Oracle. The semi-structured data specifically consists of system operation logs and JSON format interface return data. The unstructured data specifically consists of enterprise certificate images, PDF documents, and audio and video evidence files. The system also collects the entity identity information of the corresponding data source providers. The entity identity information includes the unified social credit code, data access filing number, and entity qualification certificate number. Before the collection link starts, the entity qualification verification is completed. The public API of the National Enterprise Credit Information Publicity System is called to verify whether the entity is in normal operation and whether there are any data compliance-related credit records. Only entities that pass the qualification verification are allowed to access the data collection link. Entities that fail the verification are directly rejected from accessing the link.

[0023] S2. Perform preliminary cleaning and deduplication processing on the collected multi-source heterogeneous data, specifically including null value filtering, illegal format verification of regular expression rules, and deduplication based on SimHash algorithm for duplicate data fingerprint comparison. The implementation logic of SimHash deduplication is as follows: after segmenting text data, extract weighted feature vectors, calculate and generate 64-bit SimHash fingerprints, compare the Hamming distance of two data fingerprints, and determine that data with a Hamming distance less than 3 is duplicate data, and directly filter and retain the latest one. After cleaning, the system automatically extracts metadata tags corresponding to each data point. These metadata tags include the data's domain, generation time, attribute dimension, security level, and whether it is a required core field. Simultaneously, the system collects end-to-end behavioral data, including historical data upload timeliness records of the data source entity, data compliance verification results, historical data error rate, user feedback, cumulative data call frequency, cross-data source cross-validation matching results, and data security compliance records. Behavioral data collection interfaces are connected to the open interfaces of the data verification system, user feedback system, cross-source cross-validation system, and data call log system. Event-based behavioral data is retrieved in real time, while batch statistical behavioral data is synchronized hourly.

[0024] S3. Construct a two-layer behavioral score calculation model. Based on the collected end-to-end behavioral data, calculate the credit score of each data source entity and the data quality score of each individual data point. Then, calculate the comprehensive behavioral score value for each data point by weighting them according to preset weights. The core calculation formula is: ;in The integral of the comprehensive behavior of a single data point is expressed in units of points. This is the credit score of the entity corresponding to the data source, measured in points; This is the quality score for each data point, measured in points. The weighting can be customized according to business scenarios; in government scenarios, it can be adjusted so that the entity's credit score accounts for 70% and the individual data point quality score accounts for 30%.

[0025] S4. Based on the comprehensive behavioral score corresponding to each data point, assign a corresponding data fusion weight to each data point. The core calculation formula for the fusion weight is as follows: ;in Let be the fusion weight of the i-th data to be fused. Let n be the comprehensive behavioral integral of the i-th data point, and n be the total number of data points to be fused under the same attribute dimension. Multi-source heterogeneous data are fused and calibrated according to the fusion weight. If there is a conflict between data of the same attribute dimension from different data sources, the data corresponding to the data source with the higher comprehensive behavioral score is adopted first. Before adoption, a second cross-verification is performed on the high score data. The accuracy of the data is verified by calling an authoritative public database or a preset verification rule. After the verification is passed, the attribute value is stored in the database. If the verification fails, the data of the next level of score is adopted in turn.

[0026] S5. After the multi-source data after fusion and calibration is standardized according to the preset unified metadata standard, field mapping, format conversion and sensitive data desensitization are performed and stored in the HBase distributed database. At the same time, based on the new behavioral data generated in this fusion governance process, including data conflict records, user feedback records and cross-validation results, the subject credit score of the corresponding data source subject and the data quality score of the corresponding single data are dynamically updated. The updated comprehensive behavioral score value is synchronized to the Redis fusion governance cache pool in real time and directly applied to all subsequent multi-source data fusion governance processes without waiting for batch updates.

[0027] In this invention, the main credit score is calculated using a lifetime cumulative system. The initial score is uniformly set to 100 points, with an upper limit of 200 points and a lower limit of 0 points. For each instance of data field error, non-compliant data content, or data update delay exceeding the agreed time limit, 5 to 30 points are deducted. The deduction is positively correlated with the severity of the error. For core fields, such as the unified social credit code and ID number, errors result in a deduction of 20-30 points. For auxiliary fields, such as contact phone number and address, errors result in a deduction of 5-10 points. Data update delays exceeding 24 hours result in a deduction of 5 points, and delays exceeding 72 hours result in a deduction of 15 points. For each instance of full data verification passing, valid user reviews, or 100% cross-data source cross-validation matching, 2 to 5 points are added. Specifically, 100 consecutive data verifications passing all times add 5 points, 5-star valid user reviews add 2 points, and full cross-validation matching adds 3 points. Points accumulate to 200 points and then stop being added.

[0028] In this invention, when data conflicts occur within the same attribute dimension, after accepting high-scoring data and marking low-scoring data as unusable, a point deduction instruction is generated synchronously to deduct the credit score of the data source entity that submitted the conflicting low-quality data. The deduction value is positively correlated with the degree of impact of the fusion error caused by the low-quality data. If the low-quality data belongs to a core business field and has already flowed to the downstream business system causing a call error, 10 to 25 points are deducted. If it belongs to a non-core auxiliary field and is only discovered during the fusion process without flowing to the downstream, 2 to 8 points are deducted. The deduction record is synchronously stored in the entity's credit profile stored in MongoDB as the basis for subsequent qualification review and service pricing.

[0029] In this invention, before performing multi-source data fusion calibration, entity alignment is first completed based on knowledge graph entity linking technology. Through attribute matching and semantic similarity calculation of the BERT pre-trained model, all multi-source attribute data belonging to the same natural person, legal person, and item entity are aggregated. The specific implementation logic is as follows: The entity identifier attributes of each data point are input into the pre-trained BERT-base model, which outputs a 768-dimensional semantic vector. The cosine similarity between two entity vectors is calculated. Entities with a cosine similarity greater than 0.95 are considered to be the same entity and are grouped under the same globally unique entity ID. Then, based on the comprehensive behavioral integral value corresponding to each attribute data and the business weight of each attribute, the final attribute value of the entity is calculated. For numerical attributes, the average value is calculated by weighting the attributes, and for text attributes, the text content corresponding to the highest weight is adopted.

[0030] In this invention, the dynamic update of behavior points adopts an event-driven real-time triggering mechanism. It asynchronously receives end-to-end behavior events via a RocketMQ message queue. The message queue topic is set to "behavior_event". The event message carries the event type, associated subject ID, associated data ID, and event priority fields. When data from any data source subject is accessed, receives feedback for error correction, or is cross-validated for a mismatch, a high-priority points update task with priority 1 is immediately generated. Upon receiving the high-priority event, the consumer immediately executes the points update, without waiting for fixed-period batch settlement. The update result is synchronized to the cache of all associated fusion tasks within 1 second. The core implementation code snippet is as follows: @RocketMQMessageListener(topic = "behavior_event", consumerGroup = "score_update_group") public class ScoreUpdateConsumer implements RocketMQListener <behaviorevent>{ @Resource private ScoreService scoreService; @Resource private RedisTemplate<String, Object> redisTemplate; @Override public void onMessage(BehaviorEvent event) { High-priority events are processed immediately, while low-priority events are processed in batches. if(event.getPriority() == 1){ / / Asynchronous update of database points scoreService.updateScoreAsync(event.getSubId(), event.getDataId(), event.getEventType()); / / Synchronously update Redis cache to ensure that the fusion task retrieves the latest points. Integer newScore = scoreService.getSubScore(event.getSubId()); redisTemplate.opsForValue().set("score:sub:"+event.getSubId(), newScore); } } } This invention also includes a credit score warning step, which monitors the credit scores of all data source entities in real time. When an entity's credit score falls below a preset security threshold (which can be customized according to the business scenario; the default setting is 80 points for financial scenarios, 70 points for government scenarios, and 60 points for public service scenarios), the disable method of the data collection interface is immediately invoked to suspend access to all data sources for that entity. At the same time, a warning notification is sent to the administrator and the person in charge of the entity. The entity is granted a 7-day trial period after submitting an explanation of the data error, a rectification plan, and a test rectification dataset that passes the full data compliance verification. If no data errors occur during the trial period, the entity's formal access permission and initial credit score of 100 points are restored. If data errors occur during the trial period, access is suspended again, and rectification is required.

[0031] This invention discloses a multi-source data fusion governance system based on behavior integrals, which specifically includes the following modules: The multi-source data acquisition module is used to adapt to various data acquisition protocols and access multi-source heterogeneous data from various business channels. It has a built-in entity qualification verification unit to verify the entity identity information and compliance qualifications of the data source provider. Only entities that have passed the qualification verification are allowed to access the data acquisition link. It supports users to visually configure acquisition tasks, set acquisition frequency, acquisition field range, and data anonymization rules.

[0032] The behavior data acquisition module is used to connect to the end-to-end log system, user feedback system, and cross-source cross-validation system. It automatically pulls all end-to-end behavior data related to the data source and individual data. After performing deduplication and normalization on the behavior data, it stores it in the behavior database. The behavior data deduplication uses "data ID + event ID" as the unique primary key, and duplicate events are directly filtered.

[0033] The points calculation module is used to build a two-dimensional behavioral points calculation model. It has a built-in configurable points rule engine based on Drools, which supports custom configuration of points calculation rules, weights and thresholds for different industries. After the rules are modified, they will take effect without restarting the system. Based on the full-link behavioral data, it calculates the main credit score of each data source entity and the data quality score of each individual data, and calculates the comprehensive behavioral points value of each data by weighting.

[0034] The data fusion governance module is used to dynamically allocate data fusion weights based on the comprehensive behavior score of each data. It performs fusion calibration on the collected multi-source heterogeneous data according to the fusion weights. It has built-in conflict handling logic. If there is a conflict between data of the same attribute dimension from different data sources, the data corresponding to the data source with the higher comprehensive behavior score will be given priority. Conflicts in confidential fields will automatically trigger a manual review process. The data will be stored in the database only after the review is approved.

[0035] The storage update module has a built-in standardized mapping engine that supports custom unified metadata standards and automatic mapping from formats such as XML, JSON, and Excel to the unified metadata format. It is used to perform standardized processing on multi-source data after fusion calibration and store it in a unified distributed database. At the same time, it has a built-in points update unit that receives new behavioral data generated during the fusion governance process in real time, dynamically updates the subject credit score of the corresponding data source subject and the data quality score of the corresponding single data, and synchronizes the update results to the fusion governance cache pool in real time.

[0036] In this invention, the points calculation module includes a main credit points calculation unit and a single data quality points calculation unit. The main credit points calculation unit connects to the main credit file and calculates all points changes of the main entity since its access throughout its life. It supports full-link backtracking of points change history and can filter and query historical records by time range, event type, and deduction value. The single data quality points calculation unit scores a single data record based on four dimensions: completeness, accuracy, timeliness, and compliance, each with a weight of 25%. Completeness is 25 points for all required fields not being empty, and 5 points are deducted for each missing required field. Accuracy is 25 points for all fields meeting the validation rules, and 3 points are deducted for each non-compliant field. Timeliness is 25 points for data generated within the validity period, and points are deducted based on the length of time beyond the validity period. Compliance is 25 points for no sensitive information leakage and compliance with data security regulations, and 0 points for the presence of un-de-identified sensitive information.

[0037] In this invention, the integrated governance module has a built-in conflict handling unit. When the conflict handling unit detects a data conflict of the same attribute dimension, it first determines the importance level of the conflicting field. After accepting the high-scoring data, it generates a corresponding score deduction instruction based on the importance level of the conflicting field and the impact range of the low-quality data. This instruction is then sent to the score calculation module to execute the credit score deduction for the corresponding entity. At the same time, a conflict tracing record is generated and stored in Elasticsearch. The conflict tracing record includes the conflict time, conflicting field, conflicting data content, corresponding data source entity ID, acceptance result, and deduction score. It supports full-text search for subsequent audit and verification. The deduction record is automatically synchronized to the person in charge of the corresponding data source entity to facilitate targeted rectification.

[0038] In this invention, the system also includes a credit score early warning module. The credit score early warning module has a built-in configurable threshold setting unit, which supports custom configuration of security thresholds for different business scenarios, different subject types, and different security levels. For example, the threshold for government data sources can be set to 60 points, and the threshold for commercial data sources can be set to 80 points. The credit score early warning module monitors the credit score of all data source subjects in real time. When the subject's credit score is lower than the preset security threshold, it immediately sends a pause access command to the multi-source data acquisition module and simultaneously pushes an early warning notification to the administrator containing the subject's name, current score, reason for deduction, and rectification requirements. After the subject submits rectification materials and passes full verification, it automatically sends a restore access command to the multi-source data acquisition module. Example

[0039] A provincial government service platform needs to integrate enterprise data from four government departments—industry and commerce, taxation, social security, and housing and construction—to generate a unified enterprise credit profile to support the "one-stop online government service" business. Previously, during the integration process, there were frequent problems such as data conflicts between different departments on the same dimension and repeated inflow of low-quality data, with no clear basis for acceptance.

[0040] Four government departments act as the main data sources, uploading data through the RESTful API interface of the government data sharing and exchange platform. The configured scoring rules are as follows: the main credit score accounts for 70%, the quality score of a single data entry accounts for 30%, the security threshold is set at 70 points, and 20 points are deducted for errors in core fields.

[0041] Through end-to-end behavioral scoring, the historical data error rate of the industry and commerce department was only 0.02%, with a main credit score of 185 points; the error rate of the tax department was 0.05%, with a score of 172 points; the error rate of the social security department was 0.1%, with a score of 156 points; and the error rate of the housing and construction department was 0.12%, with a score of 148 points. When data conflicts occur within the same dimension, the higher-scoring data from the industry and commerce department is prioritized, resolving the previous issue of unsubstantiated conflicts. If a department uploads the enterprise address field incorrectly three times consecutively, its score drops to 68 points, triggering an alert and suspending access. After rectification, 1,000 test data entries are uploaded, and access is restored once the error rate is below 0.01%, effectively preventing the repeated inflow of low-quality data.

[0042] The four departments first access the data collection link through entity qualification verification. The collected multi-source data is first cleaned, deduplicated, and tagged with metadata. The historical upload records and verification results of each department are collected to calculate the entity credit score and the quality score of a single data item. All attribute data of the same enterprise are aggregated through entity alignment and fused according to the fusion weight. In case of conflict, the data with higher score is given priority. The standardized data after fusion is stored in the government sharing database. The score is updated in real time for each fusion of behavioral data, and an abnormal score will automatically trigger an early warning. Example

[0043] A financial risk control service provider needs to integrate enterprise operating data from 12 commercial data sources to provide input data for banks to build risk control models for small and micro enterprises. Previously, the quality of commercial data sources varied greatly, and some data sources uploaded false data in order to increase the data volume, resulting in the accuracy of the risk control model being less than 70%, and the quality of the data sources was uncontrollable.

[0044] Twelve commercial data sources serve as the main entities, uploading encrypted business data via FTP protocol. The configured scoring rules are as follows: main entity credit score accounts for 60%, single data quality score accounts for 40%, security threshold is set at 80 points, core field errors deduct 30 points, and user-reported data errors deduct 15 points each time.

[0045] By dynamically adjusting behavioral scores, data sources that upload false data will be repeatedly reported as incorrect by bank users, causing their scores to drop rapidly to below 80 points, at which point access will be automatically suspended, thus solving the problem of uncontrollable data source quality. During data integration, data from data sources with high scores are prioritized, increasing the accuracy of input data for the risk control model to over 90%, effectively reducing the bank's bad debt risk.

[0046] When calculating the quality score of a single data point, the risk control rule engine is called to verify whether the data meets the requirements of the risk control field, such as whether the company's revenue data is within the reasonable range of the industry. After calculating the comprehensive behavioral score, there are 5 data points to be integrated for the same company's revenue field, with comprehensive scores of 190, 170, 150, 120, and 90 respectively. According to the integration weight formula, the weights of each data point are calculated to be 0.26, 0.24, 0.21, 0.17, and 0.12 respectively. The weighted calculation yields the company's final revenue value, which is used as the input field of the risk control model.

[0047] Reference Figure 1 This diagram macroscopically illustrates the complete closed-loop lifecycle of multi-source data fusion governance based on behavioral scores. The process begins with the comprehensive collection of heterogeneous data from multiple channels and sources, along with corresponding entity identity information. This is followed by initial cleaning, deduplication, and extraction of metadata tags. Building upon this, the system simultaneously captures end-to-end behavioral data to construct a score calculation model, calculating entity credit scores and individual data quality scores separately. In the core fusion governance phase, the system transforms these behavioral scores into data fusion weights, thereby performing weighted fusion and calibration / error correction on massive amounts of heterogeneous multi-source data. Finally, the standardized, high-quality fusion data is stored in a database. Simultaneously, new behavioral features generated in this round of governance directly feed back into the score model, driving dynamic iterative updates of corresponding entity and data scores, forming a virtuous cycle of optimized governance.

[0048] Reference Figure 2 This diagram provides an in-depth analysis of the end-to-end calculation mechanism and real-time dynamic update logic of behavioral points. The points calculation model relies on a continuously aggregated end-to-end behavioral data source, specifically covering multi-dimensional indicators such as historical upload records, data verification results, error rate, and call frequency. For data providers, the system uses a cumulative system to calculate the entity's credit score, awarding positive incentives for data compliance and efficient flow, and deducting penalties for data delays or errors. Simultaneously, it incorporates user feedback to calculate individual data quality scores. The dynamic points update mechanism employs a real-time trigger mode; once critical business actions such as calls, error correction, or cross-validation discrepancies occur, the system immediately and synchronously updates the corresponding entity's credit status, completely eliminating the latency effects of traditional fixed-period batch processing.

[0049] Reference Figure 3 This diagram highlights the entity alignment mechanism in complex multi-source environments and the degradation resolution strategy for data conflicts within the same dimension. Before entering the substantive multi-source data fusion and calibration stage, the system primarily relies on the metadata tags of each data point to perform efficient entity-level alignment, consolidating all multi-source attribute features scattered across the same logical entity. In the subsequent attribute value calculation, the final attribute value of the entity is derived based on a preset integral weighting model. When physical conflicts are detected between data of the same dimension from different data sources, the arbitration engine prioritizes the data source information with the higher behavioral integral value to ensure absolute data quality. Simultaneously, the system triggers a penalty mechanism to accurately measure the depth of the fusion error caused by low-quality data and deduct the credit score of the entity submitting the conflicting data accordingly.

[0050] Reference Figure 4 This diagram clearly illustrates the security alert and data source access permission blocking and recovery process centered on the entity's credit score. The system's deployed credit score alert module monitors the credit score status of all entities using a high-frequency polling method in real time. When any entity's credit score falls below the preset system security threshold, the defense mechanism immediately responds, issuing a pause access command to the data collection front-end and completely blocking all heterogeneous data import channels for that entity. During the permission silence period, the restricted entity must submit detailed rectification materials to the system and proactively apply for a re-inspection. Only after passing a rigorous data compliance verification process can the system lift the blockade, restore data access permissions for the corresponding channel, and reset its initial credit score, thereby constructing a robust data compliance defense.

[0051] Reference Figure 5 This diagram comprehensively deconstructs the underlying hardware and software logical architecture of the system implementing the behavioral points data governance method, as well as the interactive and collaborative relationships between various functional modules. The system's top-level deployment includes two fundamental modules: multi-source data acquisition and behavioral data acquisition, responsible for processing heterogeneous detailed data and end-to-end behavioral characteristics. Data flows into the points calculation module, where internally nested main credit points calculation units and single data quality points calculation units output quantified points in parallel. The core fusion governance module receives points instructions and schedules the built-in conflict handling unit to perform precise weighting and conflict arbitration. The tail-end storage and update module takes over the standardized and cleaned finished data for storage and updates the points matrix accordingly. Furthermore, an external points early warning module continuously monitors the overall credit level and works closely with the acquisition module to execute permission circuit breaking and recovery operations.

[0052] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.< / behaviorevent>

Claims

1. A multi-source data fusion governance method based on behavior integrals, characterized in that, Includes the following steps: S1. Collect multi-source heterogeneous data from various channels, including structured data, semi-structured data and unstructured data, and simultaneously collect the identity information of the subject corresponding to each data source. S2. Perform preliminary cleaning and deduplication on the collected multi-source heterogeneous data, extract the metadata tags corresponding to each data, and synchronously collect full-link behavioral data. The full-link behavioral data includes the historical data upload records of the data source entity, data verification results, historical data error rate, feedback evaluation of data users, data call frequency, and cross-validation matching results. S3. Construct a behavior score calculation model. Based on the collected full-link behavior data, calculate the subject credit score of each data source subject and the data quality score of each individual data to obtain the behavior score value of each data. S4. Based on the behavioral score of each data point, assign a corresponding data fusion weight to each data point, and perform fusion calibration on the multi-source heterogeneous data according to the fusion weight. If there is a data conflict of the same dimension from different data sources, the data corresponding to the data source with the higher behavioral score will be given priority. S5. After standardizing the fused and calibrated multi-source data, store it in a unified database. At the same time, based on the new behavioral data generated in this fusion governance process, dynamically update the subject credit score of the corresponding data source subject and the data quality score of the corresponding single data. The updated behavioral score value is directly applied to the subsequent multi-source data fusion governance process.

2. The multi-source data fusion governance method based on behavior integrals according to claim 1, characterized in that, The main credit score in step S3 is calculated using a cumulative system. The initial score is a preset fixed value. Each time there is a data error, non-compliance, or untimely data update, the corresponding score is deducted. Each time there is a full pass of data verification, positive user reviews, or 100% cross-verification match, the corresponding score is increased.

3. The multi-source data fusion governance method based on behavior integrals according to claim 1, characterized in that, When a data conflict occurs in the same dimension in step S4, after the high-scoring data is accepted, the credit score of the data source subject that submitted the conflicting low-quality data is deducted simultaneously. The deduction value is positively correlated with the degree of impact of the fusion error caused by the low-quality data.

4. The multi-source data fusion governance method based on behavior integrals according to claim 1, characterized in that, Before performing multi-source data fusion calibration in step S4, entity alignment is first performed based on the metadata tags of each data. All multi-source attribute data belonging to the same entity are aggregated, and then the final attribute value of the entity is calculated by weighting the behavior integral values ​​corresponding to each attribute data.

5. The multi-source data fusion governance method based on behavior integrals according to claim 1, characterized in that, In step S5, the dynamic update of behavior points is triggered in real time. When data from any data source is called, feedback is provided for error correction, or cross-validation is performed to check for mismatch, the corresponding points update operation is immediately triggered without waiting for fixed-period batch settlement.

6. The multi-source data fusion governance method based on behavior integrals according to claim 1, characterized in that, It also includes a credit score warning step, which monitors the credit scores of all data source entities in real time. When the credit score of an entity falls below a preset security threshold, the entity’s access to all data sources is suspended until the entity submits rectification materials and passes the data compliance verification, at which point its access permissions and corresponding initial score value are restored.

7. A multi-source data fusion governance system based on behavior integrals, used to implement the multi-source data fusion governance method based on behavior integrals as described in any one of claims 1-6, characterized in that, include: The multi-source data acquisition module is used to collect heterogeneous data from various channels and the identity information of the entities corresponding to each data source. The behavioral data acquisition module is used to collect full-link behavioral data of the data source entity. The full-link behavioral data includes historical data upload records, data verification results, historical data error rate, feedback and evaluation from data users, data call frequency, and cross-validation matching results. The points calculation module is used to build a behavior points calculation model. Based on the full-link behavior data, it calculates the subject credit score of each data source subject and the data quality score of each individual data, and obtains the behavior points value of each data. The fusion governance module is used to assign data fusion weights based on the behavioral score values ​​corresponding to each data point, and to perform fusion calibration on multi-source heterogeneous data according to the fusion weights. If there is a data conflict of the same dimension from different data sources, the data corresponding to the data source with the higher behavioral score value will be given priority. The storage update module is used to standardize the multi-source data after fusion calibration and store it in a unified database. At the same time, based on the new behavioral data generated in this fusion governance process, it dynamically updates the subject credit score of the corresponding data source subject and the data quality score of the corresponding individual data.

8. The multi-source data fusion governance system based on behavior integrals according to claim 7, characterized in that, The points calculation module includes a main credit points calculation unit and a single data quality points calculation unit. The main credit points calculation unit calculates the main credit points using a cumulative system, while the single data quality points calculation unit calculates the corresponding data quality points based on the end-to-end quality indicators of a single data item.

9. A multi-source data fusion governance system based on behavior integrals according to claim 7, characterized in that, The integrated governance module has a built-in conflict handling unit. When the conflict handling unit detects a data conflict in the same dimension, it sends a point deduction instruction to the point calculation module after completing the acceptance of high-scoring data, and deducts the subject credit score of the data source subject that submitted the conflicting low-quality data.

10. A multi-source data fusion governance system based on behavior integrals according to claim 7, characterized in that, It also includes a credit score early warning module, which monitors the credit score of all data source entities in real time. When the credit score of an entity falls below a preset security threshold, it sends a pause access command to the multi-source data acquisition module to stop collecting data from all data sources of the corresponding entity.

Images