Intelligent market supervision data management system and method based on multi-level data sharing exchange

Abstract

The application discloses a smart market supervision data management system and method based on multi-level data sharing exchange, and relates to the technical fields of data management and market supervision informatization.The application realizes efficient sharing and exchange of market supervision data across departments, regions and levels through a data sharing exchange module, ensures the safety and controllability of the data sharing process in combination with strategy-driven permission management and dynamic resource directory functions, the data standard management module constructs a closed-loop quality management process for standardized data, and improves the reliability of data quality, the data model management module realizes cross-business system correlation analysis based on multi-star data models, efficiently integrates multidimensional data, and provides optimized analysis results for the data asset management module, the data asset management module dynamically adjusts storage strategies using access heat, and optimizes data storage and management efficiency in combination with data link analysis functions, fully tapping the value of data.

Description

Technical Field

[0001] The present invention relates to the field of data governance and market supervision information technology, and specifically to an intelligent market supervision data governance system and method based on multi-level data sharing and exchange. Background Art

[0002] With the rapid development of informatization, the volume and complexity of market supervision data are constantly increasing. The data involves diverse departments, regions, and levels. The decentralized storage and isolated management of data make cross-departmental, cross-regional, and cross-level data sharing and collaboration difficult. At the same time, due to inconsistent data standards, uneven quality, and a lack of efficient analytical models, market supervision data faces the following major problems in its use:

[0003] The data sharing process has problems such as complex authority management, insufficient privacy protection, and low data access efficiency;

[0004] The diversity and heterogeneity of data lead to a lack of unified standards for shared data, which cannot meet the actual needs of market supervision;

[0005] Due to the diversity of data sources and the lack of effective governance mechanisms, the identification, governance, and tracking of problematic data are inefficient, affecting the accuracy of subsequent analysis;

[0006] Cross-system and cross-business data correlation analysis relies on complex model construction, and existing technologies are unable to support multi-dimensional and high-frequency data integration and analysis;

[0007] Storage strategies and access efficiency are not combined with dynamic adjustment mechanisms, and there is a lack of effective means for tracking and change management of data links, making it difficult to fully tap into the value of assets. Summary of the Invention

[0008] To address the above issues, the present invention proposes an intelligent market supervision data governance system and method based on multi-level data sharing and exchange, providing a full-process solution from data sharing, standardization, quality governance to model analysis and asset management.

[0009] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0010] An intelligent market supervision data governance system based on multi-level data sharing and exchange, comprising:

[0011] The data sharing and exchange module is used to share and exchange market supervision data across departments, regions, and hierarchies. It implements data access control and dynamic resource directory management in combination with policy-driven methods. The data resource directory and shared data generated by the data sharing and exchange module are used by other modules.

[0012] The data standards management module is used to standardize the management of market regulatory data throughout its life cycle. It interfaces with the data sharing and exchange module to ensure that shared data meets standardization requirements. It optimizes the standardization process through dynamic updates of data standards and version difference analysis, and provides standardized data for processing by the data quality management module.

[0013] The data quality management module is used to manage the quality of market regulatory data throughout its life cycle. It receives standardized data and forms a closed-loop data quality management process through the identification, allocation, management, and re-testing of problematic data, ensuring high-quality data for use in the data model management module.

[0014] The data model management module is used to optimize and analyze multidimensional data based on the multi-satellite data model. It receives high-quality data to build correlation models across business systems, implements data integration and analysis, and provides the analysis results to the data asset management module, thereby improving data processing efficiency through scheduling optimization.

[0015] The data asset management module is used to comprehensively manage data assets and store, optimize, and display the analysis results output by the data model management module. This includes dynamically adjusting data storage strategies based on access popularity, improving data usage efficiency through data link analysis, and providing data change impact analysis capabilities.

[0016] As a preferred solution of the present invention, the data sharing and exchange module includes:

[0017] The policy-driven unit is used to define data sharing policies, including:

[0018] Data access strategy: Using a Byzantine fault-tolerant consensus protocol, an independent private blockchain ledger is maintained to record cross-departmental, cross-regional, and cross-level data operations. A lightweight proof-of-work mechanism is used to maintain a cross-domain public blockchain ledger, recording only checkpoint information for cross-domain operations. Smart contracts are used within the blockchain network to automatically define data access permissions, privacy protection constraints, and data integrity verification rules, enabling automated permission control for cross-domain data sharing.

[0019] Data processing strategy: Privacy-preserving data modeling is achieved through a federated learning module, with collaborative modeling performed across multiple institutions. This module adjusts model update weights based on the amount of data and data quality scores of participating parties. It uses a clustering algorithm to group similarly distributed data, improving the model's adaptability to non-IID data. It combines differential privacy techniques with a compressed parameter transmission mechanism to reduce the communication cost of federated learning model updates. Furthermore, it adopts a compute-to-data model, allowing computational tasks to be executed at the data source and outputting only aggregated statistical results to ensure data security.

[0020] Data retention policy: used to limit the retention period of market regulatory data and automatically delete or archive the data after the period expires;

[0021] Approval and verification strategy: used to define the approval process and rules before data sharing. After verifying the sharing request through the rule engine, the approval process is generated;

[0022] The optimization unit is used to build the data resource directory into a graph structure based on the graph database and graph algorithm, optimize the data retrieval path using the shortest path algorithm, dynamically optimize the data exchange rules using the DQN algorithm based on the dynamic rule adjustment function of reinforcement learning, and achieve real-time load balancing.

[0023] As a preferred solution of the present invention, the lightweight proof-of-work mechanism specifically includes:

[0024] Each node in the network organizes the current transaction records into a Merkle tree and calculates the root hash value as the basis for the checkpoint record;

[0025] The difficulty of each proof-of-work task is dynamically adjusted according to the following formula:

[0026]

[0027] Where D is the difficulty of the current task; D min and D max are the lower and upper limits of difficulty respectively; D prev is the difficulty of the last task; T target is the target time, the default value is 1 minute; T actual is the actual time to complete the previous task;

[0028] Nodes attempt to calculate a hash value that meets the difficulty requirement. The first node to complete the calculation broadcasts the checkpoint record and proof result. Other nodes check whether the hash value meets the difficulty requirement through batch verification and use multi-threaded asynchronous processing. If the verification passes, the checkpoint record is stored on the public blockchain, and only the operation summary information is recorded;

[0029] In cross-domain operations, nodes in the local domain quickly reach consensus through Byzantine fault-tolerant consensus, and then submit cross-domain checkpoints to the public blockchain, forming a hierarchical consensus process;

[0030] Based on smart contracts, only checkpoint records with high access frequency are retained. The high access frequency is judged by setting a time window to count the number of accesses and setting a threshold. The statistical results are updated every 30 minutes, sorted by the PageRank algorithm, and the storage strategy is dynamically adjusted to reduce redundant data.

[0031] As a preferred solution of the present invention, the calculation to data mode is specifically:

[0032] The requester sends the task to the data source through a secure channel, including the calculation logic, input parameters, and privacy protection requirements;

[0033] The data source uses a dynamic scheduling algorithm to allocate tasks based on the current resource load, supports multi-task parallel execution, prioritizes high-priority tasks, and runs tasks in an isolated environment;

[0034] Use DBSCAN or adaptive density clustering algorithm to automatically select clustering parameters according to local data distribution, including the maximum distance ε between sample points and the minimum number of sample points MinPts. The minimum number of sample points MinPts is set to twice the sample dimension in the dataset. The maximum distance ε between sample points is estimated by the following formula:

[0035] ε=mean(d)+k×std(d);

[0036] Where d is the distance set between sample points in the data set, mean(d) represents the average value of the distance between all sample points in the data set, std(d) represents the standard deviation of the distance between sample points in the data set; k is the adjustment coefficient;

[0037] The intermediate output data of the task is perturbed using differential privacy technology to add noise to prevent the leakage of sensitive information. The calculation formula for the noise intensity b is: Where △f is the function sensitivity and δ is the privacy budget;

[0038] Prune or quantize the task results and encrypt them for transmission using homomorphic encryption technology that supports addition and multiplication.

[0039] After receiving the result, the requester decrypts and verifies the data correctness. If the verification passes, the result is stored in the data asset library for subsequent analysis.

[0040] The data source cleans or archives temporary data according to the data retention policy to optimize resource utilization efficiency and prevent sensitive data from being retained for a long time.

[0041] As a preferred solution of the present invention, the data standard management module includes:

[0042] The dynamic standard maintenance unit is used to achieve real-time optimization and dynamic maintenance of data standards. It records the version information of each update through process management based on the smallest data unit, generates standard optimization reports based on version difference analysis, and automatically identifies and reports deviations from data standards and market regulatory requirements.

[0043] The non-standard data detection unit is used to perform automated detection of metadata and data content according to preset standards and specifications, generate a conformity analysis report based on the detection results, and provide improvement suggestions for problematic data by analyzing the differences with the current standards;

[0044] The master data management unit is used to identify and track multi-departmental master data through centralized coding and dynamic management strategies, automatically assess the impact of master data changes on related data and business processes, and output optimization plans to ensure consistency and efficient sharing of cross-departmental data;

[0045] The standard fit analysis unit is used to generate a scoring report by comparing the degree of fit between data table instances and data standards, dynamically adjust the standard content based on the scoring results and actual usage feedback, and generate actionable improvement suggestions for weak links in standard implementation.

[0046] As a preferred solution of the present invention, the data quality management module includes:

[0047] The problem data identification unit is used to perform automatic detection on the data in the system according to the preset data quality rules. It dynamically adjusts the detection rule weights by combining the improved multi-level rule engine algorithm and constructs a hierarchical weighted model with data characteristics as factors, which can be expressed as:

[0048]

[0049] Where W i is the detection rule weight; α i is the priority value of the data feature, n is the total number of data features; β i is the rule adaptation;

[0050] The data quality closed-loop governance unit is used to manage problem data throughout its lifecycle. It introduces a dynamic progress prediction algorithm P = f(Q, T, R), where P is the expected governance completion time, Q is the data quality score, T is the data volume, R is the resource allocation efficiency, and f represents the function mapping relationship. This optimizes the governance task allocation process to ensure that problem data is governed in the shortest possible time. It also adjusts governance priorities in real time during the re-testing phase, forming a closed-loop governance process.

[0051] The data quality scoring unit is used to quantitatively score data quality based on the defined data detection rules and governance results. It uses a dynamic fuzzy comprehensive evaluation algorithm to combine data characteristics with governance results and generate a quality detection report. The dynamic fuzzy comprehensive evaluation algorithm is expressed as:

[0052]

[0053] Where S is the total score of data quality; μ iis the weight of data feature i; τ ij is the scoring result of data feature i under rule j; w ij is the weight of rule j on data feature i; m is the total number of evaluation rules;

[0054] The manual data governance support unit is used to assign problematic data to specific personnel for verification and processing based on the data's region of origin, business type, or custom label, combined with a workload balancing algorithm. Where L is the workload per person, W is the total task volume, and N is the number of people. Tasks are dynamically allocated to avoid resource waste, and batch export of problem data and monitoring of governance progress are supported.

[0055] As a preferred solution of the present invention, the data model management module includes:

[0056] The multi-star data model design unit is used to build a unified model across business systems based on multiple fact tables and shared dimension tables. It supports multi-dimensional data analysis and integration, expands the model's semantic association capabilities through knowledge graph technology, and combines relationship inference algorithms to achieve efficient integration of heterogeneous data.

[0057] The model conversion and optimization unit is used to dynamically switch between the star model and the snowflake model according to specific business needs, optimize data query performance through an adaptive scheduling mechanism, and improve data processing efficiency by combining dynamic partitioning strategies;

[0058] A scenario-driven analysis unit is used to perform risk prediction, trend analysis, and hotspot detection based on an inert data model. The inert data model is implemented through a lazy loading mechanism. Specifically, the required data fragments are dynamically loaded only when a specific analysis task is received. Unrequested data remains in storage and does not occupy memory resources. After loading, a caching strategy is used to avoid repeated loading. Generative adversarial network technology is also used to simulate business boundary scenarios, evaluate the model's adaptability under extreme conditions, and provide decision support for market supervision through analysis results.

[0059] The data lineage analysis and early warning unit is used to trace data sources and change links in real time during data processing. It displays the data processing path through visualization, monitors the quality of model output in combination with dynamic rules, and triggers early warning mechanisms when anomalies occur, providing feedback through system announcements, SMS reminders, and email notifications.

[0060] The data processing configuration and scheduling unit is used to provide a browser-based visual interface, combined with a reinforcement learning-driven automatic scheduling mechanism, to dynamically adjust task order and resource allocation according to real-time task load and priority.

[0061] As a preferred solution of the present invention, the multi-star data model design unit introduces a dynamic semantic expansion technology based on deep learning, optimizes the semantic association strength of the knowledge graph in real time through the attention mechanism, and combines multi-hop path analysis to mine implicit relationships, thereby achieving deep association of cross-domain data;

[0062] Using knowledge graph embedding technology, nodes and relationships are mapped into a low-dimensional vector space. Efficient relationship inference is achieved through a distributed computing framework, supporting semantic integration of heterogeneous data sources.

[0063] A reinforcement learning algorithm is used to dynamically adjust semantic conflicts generated during the integration process. This adjustment is based on the DQN algorithm, which identifies conflicts by building a conflict discriminant function. The reinforcement learning model updates the reward function during each adjustment to maximize the consistency of the association relationship, ensuring the rationality and consistency of the semantic association results and generating optimized high-order semantic relationships.

[0064] Through the dynamic adjustment mechanism of semantic priority and real-time expansion function, the adaptability and integration efficiency of the model in complex business scenarios are improved.

[0065] As a preferred solution of the present invention, the data asset management module includes:

[0066] The data resource directory unit is used to form a hierarchical directory through the metadata description of the data resource to support the retrieval and acquisition of the data resource;

[0067] Data asset heat analysis unit, used to dynamically calculate data heat based on data access frequency, citation counts, and page views, and adjust storage and access strategies to optimize system performance;

[0068] The data link analysis unit is used to combine SQL logs and stored procedure analysis to automatically generate a link relationship map of metadata. The update cycle of the link relationship map is dynamically adjusted according to the system log traffic. The optimized analysis process is achieved through distributed log analysis technology, and version change notification and difference analysis functions are provided.

[0069] A data governance method for an intelligent market supervision data governance system based on multi-level data sharing and exchange, the method comprising:

[0070] Share and exchange market regulatory data across departments, regions, and hierarchies, and implement data access control and dynamic resource catalog management through policy-driven approaches.

[0071] Conduct standardized management of market regulatory data throughout its life cycle to ensure shared data complies with standardization requirements, and optimize standardization processes through dynamic updates of data standards and version difference analysis;

[0072] Conduct quality management for the entire life cycle of market regulatory data, receive standardized data, and form a closed-loop data quality management process through the identification, allocation, management, and re-testing of problematic data;

[0073] Optimize and analyze multidimensional data based on multi-star data models, receive high-quality data to build correlation models across business systems, and achieve data integration and analysis;

[0074] Comprehensively manage data assets, store, optimize, and display analysis results of data models, including dynamically adjusting data storage strategies based on access popularity, and improving data usage efficiency through data link analysis.

[0075] The beneficial effects of the present invention are as follows: through the data sharing and exchange module, efficient sharing and exchange of market supervision data across departments, regions and levels is achieved, and combined with policy-driven permission management and dynamic resource directory functions, the security and controllability of the data sharing process are ensured, and a unified data call basis is provided for other modules; the data standard management module ensures that shared data meets unified standard requirements by dynamically updating data standards and version difference analysis, provides high-quality, standardized data input for the data quality management module, and builds a closed-loop quality governance process for standardized data, including the identification, governance and re-detection of problem data, thereby improving the reliability of data quality; the data model management module realizes correlation analysis across business systems based on the multi-star data model, efficiently integrates multi-dimensional data, and provides optimized analysis results for use by the data asset management module; the data asset management module uses access popularity to dynamically adjust the storage strategy, and combines the data link analysis function to optimize data storage and management efficiency, and fully tap the value of data. Through the collaborative work between modules, the system achieves full process coverage from data sharing to governance, analysis and management, improves the governance capacity and intelligence level of market supervision data, and provides accurate and reliable decision support for supervision. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0077] in:

[0078] Figure 1 It is a modular structure diagram of the system of the present invention;

[0079] Figure 2 Schematic diagram of the process of data quality management in an embodiment of the present invention;

[0080] Figure 3 This is a schematic diagram of the interface for data asset heat analysis in an embodiment of the present invention;

[0081] Figure 4 4 is a flow chart of a method in an embodiment of the present invention. DETAILED DESCRIPTION

[0082] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of the present invention.

[0083] like Figure 1-Figure 3 FIG. 1 is an embodiment of the present invention, which provides an intelligent market supervision data governance system based on multi-level data sharing and exchange, including:

[0084] (1) Data sharing and exchange module

[0085] It is used to share and exchange market supervision data across departments, regions and levels, and implements data access control and dynamic resource directory management in combination with policy-driven. The data resource directory and shared data generated by the data sharing and exchange module are used by other modules to achieve safe and efficient data flow.

[0086] In one embodiment, the data sharing exchange module includes:

[0087] Policy-driven unit, used to define data sharing policies, including data access policies, data processing policies, data retention policies, approval and verification policies, etc.

[0088] Data access policies are used to dynamically define the access scope, purpose, and access targets of data sharing, and to control data access based on permission rules. Specifically, independent private blockchain ledgers are maintained through a Byzantine Fault Tolerant consensus protocol to record cross-departmental, cross-regional, and cross-level data operations. Lightweight proof-of-work is used to maintain a cross-domain public blockchain ledger, recording only checkpoint information for cross-domain operations. In addition, smart contracts are used within the blockchain network to define data access rights, privacy protection constraints, and data integrity verification rules, enabling automated permission control for cross-domain data sharing.

[0089] The data processing strategy is used to anonymize, desensitize, or aggregate data during data sharing to protect data privacy. Specifically, it implements privacy-preserving data modeling through a federated learning module and conducts collaborative modeling across multiple institutions.

[0090] The federated learning module adjusts the model update weight based on the data volume and data quality scores of the participants, groups data with similar distributions through clustering algorithms, improves the model's adaptability to non-independent and identically distributed data, combines differential privacy technology and compressed parameter transmission mechanism to reduce the communication cost of updating the federated learning model, and adopts a compute-to-data mode, allowing computing tasks to be executed at the data source and only outputting aggregated statistical results to ensure data security.

[0091] Data retention and deletion policies are used to limit the retention period of data and automatically delete or archive the data after the expiration of the period. For example, transaction records may only be retained for six months and must be deleted after the expiration.

[0092] Approval and verification policies define the approval process and rules for data sharing. Sharing requests are verified by the rules engine and then the approval process is generated. For example, detailed financial data of a company can only be accessed after joint approval from the local market supervision department and the State Administration for Market Regulation.

[0093] Data sharing policies are dynamically configured by the system administrator or data owner and stored in the system's policy management library. When data is shared or processed, the system automatically calls the data sharing policy to ensure that the operation complies with predefined rules.

[0094] The optimization unit is used to construct the data resource directory into a graph structure based on the graph database and graph algorithm, optimize the data retrieval path using the shortest path algorithm, dynamically optimize the data exchange rules using the DQN (Deep Q-Network) algorithm based on the dynamic rule adjustment function of reinforcement learning, and achieve real-time load balancing.

[0095] Furthermore, the lightweight proof-of-work mechanism specifically includes:

[0096] Each node in the network organizes the current transaction records into a Merkle tree and calculates the root hash value as the basis for the checkpoint record;

[0097] The difficulty of each proof-of-work task is dynamically adjusted according to the following formula:

[0098]

[0099] Where D is the difficulty of the current task; D min and D max are the lower and upper limits of difficulty respectively; D prev is the difficulty of the last task; T target is the target time, the default value is 1 minute; T actual is the actual time to complete the previous task;

[0100] Nodes attempt to calculate a hash value that meets the difficulty requirement. The first node to complete the calculation broadcasts the checkpoint record and proof result. Other nodes check whether the hash value meets the difficulty requirement through batch verification and use multi-threaded asynchronous processing. If the verification passes, the checkpoint record is stored on the public blockchain, and only the operation summary information is recorded;

[0101] In cross-domain operations, nodes in the local domain quickly reach consensus through Byzantine fault-tolerant consensus, and then submit cross-domain checkpoints to the public blockchain, forming a hierarchical consensus process;

[0102] Based on smart contracts, only checkpoint records with high access frequency are retained. The high access frequency is judged by setting a time window to count the number of accesses and setting a threshold. The statistical results are updated every 30 minutes, sorted by the PageRank algorithm, and the storage strategy is dynamically adjusted to reduce redundant data.

[0103] The data model is calculated as follows:

[0104] The requester sends the task to the data source through a secure channel, including the calculation logic, input parameters, and privacy protection requirements;

[0105] The data source uses a dynamic scheduling algorithm to allocate tasks based on the current resource load, supports multi-task parallel execution, prioritizes high-priority tasks, and runs tasks in an isolated environment;

[0106] Use DBSCAN or adaptive density clustering algorithm to automatically select clustering parameters according to local data distribution, including the maximum distance ε between sample points and the minimum number of sample points MinPts. The minimum number of sample points MinPts is set to twice the sample dimension in the dataset. The maximum distance ε between sample points is estimated by the following formula:

[0107] ε=mean(d)+k×std(d);

[0108] Where d is the distance set between sample points in the data set, mean(d) represents the average value of the distance between all sample points in the data set, reflecting the average size of the distance between the overall samples; std(d) represents the standard deviation of the distance between sample points in the data set, measuring the degree of dispersion of the distance distribution between samples; k is the adjustment coefficient, which is used to adjust the size of ε. By increasing or decreasing k, the adaptability of the clustering radius to the sample points can be adjusted.

[0109] By calculating the mean and standard deviation of the distance between sample points and combining it with the adjustment coefficient k, the value of ε is dynamically estimated, so that the clustering algorithm can better adapt to the distribution characteristics of the data;

[0110] The intermediate output data of the task is perturbed using differential privacy technology to add noise to prevent the leakage of sensitive information. The calculation formula for the noise intensity b is: Where △f is the function sensitivity, which refers to the maximum change in the output of the objective function. When the input data changes by one sample, the maximum change in the output result may occur; δ is the privacy budget, which is a key parameter in differential privacy protection and reflects the trade-off between privacy protection and data accuracy. A smaller δ provides stronger privacy protection;

[0111] Prune or quantize the task results and encrypt them for transmission using homomorphic encryption technology that supports addition and multiplication.

[0112] After receiving the result, the requester decrypts and verifies the data correctness. If the verification passes, the result is stored in the data asset library for subsequent analysis.

[0113] The data source cleans or archives temporary data according to the data retention policy to optimize resource utilization efficiency and prevent sensitive data from being retained for a long time.

[0114] (2) Data Standard Management Module

[0115] It is used to conduct standardized management of market supervision data throughout its life cycle, and to connect with the data sharing and exchange module to ensure that shared data meets standardization requirements. It optimizes the standardization process through dynamic updates of data standards and version difference analysis, and provides standardized data for processing by the data quality management module.

[0116] Specifically, the data standard management module includes:

[0117] The dynamic standard maintenance unit is used to achieve real-time optimization and dynamic maintenance of data standards. It records the version information of each update through process management based on the smallest data unit, generates standard optimization reports based on version difference analysis, and automatically identifies and reports deviations between data standards and market regulatory requirements, ensuring the dynamic adaptability and pertinence of the standards.

[0118] The non-standard data detection unit is used to perform automated detection of metadata and data content according to preset standards and specifications, generate a conformity analysis report based on the detection results, and provide improvement suggestions for problematic data through analysis of differences with current standards, supporting subsequent standardization governance and quality improvement;

[0119] The master data management unit is used to identify and track multi-departmental master data through centralized coding and dynamic management strategies, automatically assess the impact of master data changes on related data and business processes, and output optimization plans to ensure consistency and efficient sharing of cross-departmental data;

[0120] The standard conformity analysis unit is used to generate a scoring report by comparing the degree of conformity between data table instances and data standards, dynamically adjust the standard content based on the scoring results and actual usage feedback, and generate actionable improvement suggestions for weak links in standard implementation, thereby improving the accuracy and effectiveness of standardization implementation.

[0121] (3) Data quality management module

[0122] It is used to manage the quality of market supervision data throughout its entire life cycle, receive standardized data, and form a closed-loop data quality management process through the identification, allocation, governance and re-testing of problem data, ensuring high-quality data for use by the data model management module and improving data integrity and accuracy.

[0123] like Figure 2 As shown in the figure, it is a data quality management process based on business data, which mainly includes the following steps:

[0124] Data quality requirements analysis: Determine the quality requirements of business data and formulate data quality rules to provide a basis for subsequent testing and governance;

[0125] Data verification and problem data identification: Based on data quality rules, business data is verified to identify problematic data;

[0126] Data quality report generation: Organize the test results into a data quality report, including the distribution of problematic data, scores, and governance recommendations;

[0127] Governance progress monitoring: Allocate and claim problematic data, and combine progress monitoring and feedback mechanisms to ensure efficient completion of data governance tasks;

[0128] Data governance and repair: Store problematic data in the governance repository, perform data corrections automatically or manually, and return the corrected data to the business system.

[0129] Data reuse: The corrected data flows back into the business system to support subsequent business operations.

[0130] In one embodiment, the data quality management module includes:

[0131] The problem data identification unit is used to perform automatic detection on the data in the system according to the preset data quality rules. It dynamically adjusts the detection rule weights by combining the improved multi-level rule engine algorithm and constructs a hierarchical weighted model with data characteristics as factors, which can be expressed as:

[0132]

[0133] Where W iis the detection rule weight, which is used to indicate the importance of the rule in data quality detection; α i is the priority value of the data feature, reflecting the degree of influence of different data features on the importance of the rule; n is the total number of data features; β i Rule adaptability indicates the degree of match between the current rule and the target data characteristics. Dynamically adjust the weight of the detection rule so that it can flexibly change according to the data characteristics and rule matching situation, thus optimizing the detection process.

[0134] The data quality closed-loop governance unit is used to manage problem data throughout its lifecycle. It introduces a dynamic progress prediction algorithm P = f(Q, T, R), where P is the expected governance completion time (used to estimate the total time required for the task), Q is the data quality score (indicating the current quality status of the data), T is the data volume (indicating the scale of the data to be processed), R is the resource allocation efficiency (indicating the efficiency of computing resources allocated to governance tasks), and f represents the function mapping relationship. This optimizes the governance task allocation process to ensure that problem data is governed in the shortest possible time. It also adjusts governance priorities in real time during the re-testing phase, forming a closed-loop governance process.

[0135] The data quality scoring unit is used to quantitatively score data quality based on the defined data detection rules and governance results. It uses a dynamic fuzzy comprehensive evaluation algorithm to combine data characteristics with governance results and generate a quality detection report. The dynamic fuzzy comprehensive evaluation algorithm is expressed as:

[0136]

[0137] Where S is the total score of data quality, which is used to comprehensively evaluate the data quality status; μ i is the weight of data feature i, indicating the importance of different data features in the overall score; τ ij is the scoring result of data feature i under rule j, indicating the quality of data features under specific rules; w ij is the weight of rule j on data feature i, indicating the matching degree of the rule to the feature; m is the total number of evaluation rules;

[0138] The manual data governance support unit is used to assign problematic data to specific personnel for verification and processing based on the data's region of origin, business type, or custom label, combined with a workload balancing algorithm. Where L is the workload per person (indicating the amount of tasks after equalization), W is the total amount of tasks, and N is the number of people. Tasks are dynamically allocated to avoid resource waste, and batch export of problem data and monitoring of governance progress are supported.

[0139] (4) Data model management module

[0140] It is used to optimize and analyze multidimensional data based on the multi-star data model, receive high-quality data to build correlation models across business systems, realize data integration analysis, and provide the analysis results to the data asset management module, thereby improving data processing efficiency through scheduling optimization.

[0141] In one embodiment, the data model management module includes:

[0142] The multi-star data model design unit is used to build a unified model across business systems based on multiple fact tables and shared dimension tables. It supports multi-dimensional data analysis and integration, expands the model's semantic association capabilities through knowledge graph technology, and combines relationship inference algorithms to achieve efficient integration of heterogeneous data.

[0143] Specifically, the multi-star data model design unit introduces dynamic semantic expansion technology based on deep learning. It optimizes the semantic association strength of the knowledge graph in real time through the attention mechanism, and combines multi-hop path analysis to mine implicit relationships, thus achieving deep association of cross-domain data.

[0144] Using knowledge graph embedding technology, nodes and relationships are mapped into a low-dimensional vector space. Efficient relationship inference is achieved through a distributed computing framework, supporting semantic integration of heterogeneous data sources.

[0145] A reinforcement learning algorithm is used to dynamically adjust semantic conflicts generated during the integration process. This adjustment is based on the DQN algorithm, which identifies conflicts by building a conflict discriminant function. The reinforcement learning model updates the reward function during each adjustment to maximize the consistency of the association relationship, ensuring the rationality and consistency of the semantic association results and generating optimized high-order semantic relationships.

[0146] Improve the adaptability and integration efficiency of the model in complex business scenarios through a dynamic adjustment mechanism of semantic priorities and real-time expansion capabilities;

[0147] The model conversion and optimization unit is used to dynamically switch between the star model and the snowflake model according to specific business needs, optimize data query performance through an adaptive scheduling mechanism, and improve data processing efficiency by combining dynamic partitioning strategies;

[0148] A scenario-driven analysis unit is used to perform risk prediction, trend analysis, and hotspot detection based on an inert data model. The inert data model is implemented through a lazy loading mechanism. Specifically, the required data fragments are dynamically loaded only when a specific analysis task is received. Unrequested data remains in storage and does not occupy memory resources. After loading, a caching strategy is used to avoid repeated loading. Generative adversarial network technology is also used to simulate business boundary scenarios, evaluate the model's adaptability under extreme conditions, and provide decision support for market supervision through analysis results.

[0149] The data lineage analysis and early warning unit is used to trace data sources and change links in real time during data processing. It displays the data processing path through visualization, monitors the quality of model output in combination with dynamic rules, and triggers early warning mechanisms when anomalies occur, providing feedback through system announcements, SMS reminders, and email notifications.

[0150] The data processing configuration and scheduling unit is used to provide a browser-based visual interface, combined with a reinforcement learning-driven automatic scheduling mechanism, to dynamically adjust task order and resource allocation according to real-time task load and priority.

[0151] (5) Data asset management module

[0152] It is used to comprehensively manage data assets, store, optimize and display the analysis results output by the data model management module, including dynamically adjusting data storage strategies based on access popularity, improving data usage efficiency in combination with data link analysis, and providing data change impact analysis functions.

[0153] The data asset management module includes:

[0154] The data resource directory unit is used to form a hierarchical directory through the metadata description of the data resource to support the retrieval and acquisition of the data resource;

[0155] The data asset heat analysis unit is used to dynamically calculate the heat of data based on the access frequency, citation count and browsing count of the data, and adjust the storage and access strategies to optimize system performance, such as Figure 3 As shown;

[0156] The data link analysis unit is used to combine SQL logs and stored procedure analysis to automatically generate a link relationship map of metadata. The update cycle of the link relationship map is dynamically adjusted according to the system log traffic. The optimized analysis process is achieved through distributed log analysis technology, and version change notification and difference analysis functions are provided.

[0157] like Figure 4 FIG. 1 is another embodiment of the present invention, which provides a data governance method for an intelligent market supervision data governance system based on multi-level data sharing and exchange, including the following steps:

[0158] Share and exchange market regulatory data across departments, regions, and hierarchies, and implement data access control and dynamic resource catalog management through policy-driven approaches.

[0159] Conduct standardized management of market regulatory data throughout its life cycle to ensure shared data complies with standardization requirements, and optimize standardization processes through dynamic updates of data standards and version difference analysis;

[0160] Conduct quality management for the entire life cycle of market regulatory data, receive standardized data, and form a closed-loop data quality management process through the identification, allocation, management, and re-testing of problematic data;

[0161] Optimize and analyze multidimensional data based on multi-star data models, receive high-quality data to build correlation models across business systems, and achieve data integration and analysis;

[0162] Comprehensively manage data assets, store, optimize, and display analysis results of data models, including dynamically adjusting data storage strategies based on access popularity, and improving data usage efficiency through data link analysis.

[0163] In summary, this invention, through the introduction of a data sharing and exchange module, enables efficient sharing and exchange of market regulatory data across departments, regions, and hierarchies. Combined with a policy-driven implementation, the system can flexibly define data access permissions and dynamically manage resource directories, ensuring the security and controllability of data during sharing. The generated data resource directory is further accessible to other modules, establishing a foundation for data sharing between modules and supporting subsequent standardization, quality governance, and model analysis.

[0164] The Data Standards Management Module standardizes the entire lifecycle of market regulatory data and seamlessly integrates with the Data Sharing and Exchange Module, ensuring that shared data adheres to unified standards. By dynamically updating data standards and analyzing version differences to optimize the standardization process, data can quickly adapt to changing market regulatory requirements. Based on standardized data, this module provides high-quality input for the Data Quality Management Module.

[0165] The Data Quality Management module implements quality governance for standardized data throughout its lifecycle, forming a closed-loop management process that includes the identification, allocation, governance, and re-testing of problematic data. This closed-loop process significantly improves the efficiency and accuracy of problem data processing and ensures reliable, high-quality data input for the Data Model Management module, laying a solid foundation for subsequent analysis.

[0166] The Data Model Management module builds a cross-business system association model based on the multi-star data model, enabling efficient integration of multidimensional data and improving data processing efficiency through optimized analysis. The module leverages the association model to integrate and analyze data from multiple systems and efficiently transmits the analysis results to the Data Asset Management module, significantly enhancing the depth and efficiency of data analysis.

[0167] The Data Asset Management module provides comprehensive data asset management services based on analysis results. Through dynamic storage adjustment strategies based on access popularity, the system optimizes data storage and access efficiency. Combined with data link analysis, the system provides impact analysis and dynamic management of data changes, enabling the full exploration and utilization of the value of data assets.

[0168] Through the collaborative work of multiple modules, a complete closed-loop process has been formed from data sharing, standardization, quality governance to model analysis and asset management, which has significantly improved the governance, analysis and intelligence capabilities of market supervision data, and provided accurate, fast and reliable decision-making support for market supervision.

[0169] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various modifications or substitutions within the technical scope disclosed in this application, and such modifications or substitutions should be included within the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. An intelligent market supervision data governance system based on multi-level data sharing and exchange, characterized by: The system includes: a data sharing and exchange module for sharing and exchanging market supervision data across departments, regions, and hierarchies, and combining policy-driven implementation of data access control and dynamic resource directory management. The data resource directory and shared data generated by the data sharing and exchange module are used by other modules; The data standards management module is used to standardize the management of market regulatory data throughout its life cycle. It interfaces with the data sharing and exchange module to ensure that shared data meets standardization requirements. It optimizes the standardization process through dynamic updates of data standards and version difference analysis, and provides standardized data for processing by the data quality management module. The data quality management module is used to manage the quality of market regulatory data throughout its life cycle. It receives standardized data and forms a closed-loop data quality management process through the identification, allocation, management, and re-testing of problematic data, ensuring high-quality data for use in the data model management module. The data model management module is used to optimize and analyze multidimensional data based on the multi-satellite data model. It receives high-quality data to build correlation models across business systems, implements data integration and analysis, and provides the analysis results to the data asset management module, thereby improving data processing efficiency through scheduling optimization. The data asset management module is used to comprehensively manage data assets and store, optimize, and display the analysis results output by the data model management module. This includes dynamically adjusting data storage strategies based on access popularity, improving data usage efficiency through data link analysis, and providing data change impact analysis capabilities. The data sharing and exchange module includes: The policy-driven unit is used to define data sharing policies, including: Data access strategy: Using a Byzantine fault-tolerant consensus protocol, an independent private blockchain ledger is maintained to record cross-departmental, cross-regional, and cross-level data operations. A lightweight proof-of-work mechanism is used to maintain a cross-domain public blockchain ledger, recording only checkpoint information for cross-domain operations. Smart contracts are used within the blockchain network to automatically define data access permissions, privacy protection constraints, and data integrity verification rules, enabling automated permission control for cross-domain data sharing. Data processing strategy: Privacy-preserving data modeling is achieved through a federated learning module, with collaborative modeling performed across multiple institutions. This module adjusts model update weights based on the amount of data and data quality scores of participating parties. It uses a clustering algorithm to group similarly distributed data, improving the model's adaptability to non-IID data. It combines differential privacy techniques with a compressed parameter transmission mechanism to reduce the communication cost of federated learning model updates. Furthermore, it adopts a compute-to-data model, allowing computational tasks to be executed at the data source and outputting only aggregated statistical results to ensure data security. Data retention policy: used to limit the retention period of market regulatory data and automatically delete or archive the data after the period expires; Approval and verification strategy: used to define the approval process and rules before data sharing. After verifying the sharing request through the rule engine, the approval process is generated; The optimization unit is used to build the data resource directory into a graph structure based on the graph database and graph algorithm, optimize the data retrieval path using the shortest path algorithm, dynamically optimize the data exchange rules using the DQN algorithm based on the dynamic rule adjustment function of reinforcement learning, and achieve real-time load balancing.

2. The intelligent market supervision data governance system based on multi-level data sharing and exchange according to claim 1 is characterized in that: The lightweight proof-of-work mechanism specifically includes: Each node in the network organizes the current transaction records into a Merkle tree and calculates the root hash value as the basis for the checkpoint record; The difficulty of each proof-of-work task is dynamically adjusted according to the following formula: Where D is the difficulty of the current task; D min and D max are the lower and upper limits of difficulty respectively; D prev is the difficulty of the last task; T target is the target time, the default value is 1 minute; T actual is the actual time to complete the previous task; Nodes attempt to calculate a hash value that meets the difficulty requirement. The first node to complete the calculation broadcasts the checkpoint record and proof result. Other nodes check whether the hash value meets the difficulty requirement through batch verification and use multi-threaded asynchronous processing. If the verification passes, the checkpoint record is stored on the public blockchain, and only the operation summary information is recorded; In cross-domain operations, nodes in the local domain quickly reach consensus through Byzantine fault-tolerant consensus, and then submit cross-domain checkpoints to the public blockchain, forming a hierarchical consensus process; Based on smart contracts, only checkpoint records with high access frequency are retained. The high access frequency is judged by setting a time window to count the number of accesses and setting a threshold. The statistical results are updated every 30 minutes, sorted by the PageRank algorithm, and the storage strategy is dynamically adjusted to reduce redundant data.

3. The intelligent market supervision data governance system based on multi-level data sharing and exchange according to claim 1 is characterized in that: The calculation to data mode is specifically: The requester sends the task to the data source through a secure channel, including the calculation logic, input parameters, and privacy protection requirements; The data source uses a dynamic scheduling algorithm to allocate tasks based on the current resource load, supports multi-task parallel execution, prioritizes high-priority tasks, and runs tasks in an isolated environment; Use DBSCAN or adaptive density clustering algorithm to automatically select clustering parameters according to local data distribution, including the maximum distance ε between sample points and the minimum number of sample points MinPts. The minimum number of sample points MinPts is set to twice the sample dimension in the dataset. The maximum distance ε between sample points is estimated by the following formula: ε=mean(d)+k×std(d); Where d is the distance set between sample points in the data set, mean(d) represents the average value of the distance between all sample points in the data set, std(d) represents the standard deviation of the distance between sample points in the data set; k is the adjustment coefficient; The intermediate output data of the task is perturbed using differential privacy technology to add noise to prevent the leakage of sensitive information. The calculation formula for the noise intensity b is: Where Δf is the function sensitivity and δ is the privacy budget; Prune or quantize the task results and encrypt them for transmission using homomorphic encryption technology that supports addition and multiplication. After receiving the result, the requester decrypts and verifies the data correctness. If the verification passes, the result is stored in the data asset library for subsequent analysis. The data source cleans or archives temporary data according to the data retention policy to optimize resource utilization efficiency and prevent sensitive data from being retained for a long time.

4. The intelligent market supervision data governance system based on multi-level data sharing and exchange according to claim 1 is characterized in that: The data standard management module includes: The dynamic standard maintenance unit is used to achieve real-time optimization and dynamic maintenance of data standards. It records the version information of each update through process management based on the smallest data unit, generates standard optimization reports based on version difference analysis, and automatically identifies and reports deviations from data standards and market regulatory requirements. The non-standard data detection unit is used to perform automated testing of metadata and data content according to preset standards and specifications, generate a conformance analysis report based on the test results, and provide improvement suggestions for problematic data through analysis of differences with current standards. The master data management unit is used to identify and track multi-departmental master data through centralized coding and dynamic management strategies, automatically assess the impact of master data changes on related data and business processes, and output optimization solutions to ensure consistency and efficient sharing of cross-departmental data. The standard fit analysis unit is used to generate a scoring report by comparing the degree of fit between data table instances and data standards, dynamically adjust the standard content based on the scoring results and actual usage feedback, and generate actionable improvement suggestions for weak links in standard implementation.

5. The intelligent market supervision data governance system based on multi-level data sharing and exchange according to claim 1 is characterized in that: The data quality management module includes: The problem data identification unit is used to perform automatic detection on the data in the system according to the preset data quality rules. It dynamically adjusts the detection rule weights by combining the improved multi-level rule engine algorithm and constructs a hierarchical weighted model with data characteristics as factors, which can be expressed as: Where W i is the detection rule weight; α i is the priority value of the data feature, n is the total number of data features; β i The data quality closed-loop governance unit is used to manage problem data throughout its life cycle. It introduces a dynamic progress prediction algorithm P = f(Q, T, R), where P is the expected completion time of governance, Q is the data quality score, T is the data volume, R is the resource allocation efficiency, and f represents the function mapping relationship. This optimizes the governance task allocation process to ensure that problem data is governed in the shortest possible time. It also adjusts the governance priority in real time during the re-detection phase to form a closed-loop governance process. The data quality scoring unit is used to quantitatively score data quality based on the defined data detection rules and governance results. It uses a dynamic fuzzy comprehensive evaluation algorithm to combine data characteristics with governance results and generate a quality detection report. The dynamic fuzzy comprehensive evaluation algorithm is expressed as: Where S is the total score of data quality; μ i is the weight of data feature i; τ ij is the scoring result of data feature i under rule j; w ij is the weight of rule j on data feature i; m is the total number of evaluation rules; The manual data governance support unit is used to assign problematic data to specific personnel for verification and processing based on the data's region of origin, business type, or custom label, combined with a workload balancing algorithm. Where L is the workload per person, W is the total task volume, and N is the number of people. Tasks are dynamically allocated to avoid resource waste, and batch export of problem data and monitoring of governance progress are supported.

6. The intelligent market supervision data governance system based on multi-level data sharing and exchange according to claim 1 is characterized in that: The data model management module includes: The multi-star data model design unit is used to build a unified model across business systems based on multiple fact tables and shared dimension tables. It supports multi-dimensional data analysis and integration, expands the model's semantic association capabilities through knowledge graph technology, and combines relationship inference algorithms to achieve efficient integration of heterogeneous data. The model conversion and optimization unit is used to dynamically switch between the star model and the snowflake model according to specific business needs, optimize data query performance through an adaptive scheduling mechanism, and improve data processing efficiency by combining dynamic partitioning strategies; A scenario-driven analysis unit is used to perform risk prediction, trend analysis, and hotspot detection based on an inert data model. The inert data model is implemented through a lazy loading mechanism. Specifically, the required data fragments are dynamically loaded only when a specific analysis task is received. Unrequested data remains in storage and does not occupy memory resources. After loading, a caching strategy is used to avoid repeated loading. Generative adversarial network technology is also used to simulate business boundary scenarios, evaluate the model's adaptability under extreme conditions, and provide decision support for market supervision through analysis results. The data lineage analysis and early warning unit is used to trace data sources and change links in real time during data processing. It displays the data processing path through visualization, monitors the quality of model output in combination with dynamic rules, and triggers early warning mechanisms when anomalies occur, providing feedback through system announcements, SMS reminders, and email notifications. The data processing configuration and scheduling unit is used to provide a browser-based visual interface, combined with a reinforcement learning-driven automatic scheduling mechanism, to dynamically adjust task order and resource allocation according to real-time task load and priority.

7. The intelligent market supervision data governance system based on multi-level data sharing and exchange according to claim 6 is characterized in that: In the multi-star data model design unit, dynamic semantic expansion technology based on deep learning is introduced. The semantic association strength of the knowledge graph is optimized in real time through the attention mechanism. Multi-hop path analysis is combined to mine implicit relationships and achieve deep association of cross-domain data. Using knowledge graph embedding technology, nodes and relationships are mapped into a low-dimensional vector space. Efficient relationship inference is achieved through a distributed computing framework, supporting semantic integration of heterogeneous data sources. A reinforcement learning algorithm is used to dynamically adjust semantic conflicts generated during the integration process. This adjustment is based on the DQN algorithm, which identifies conflicts by building a conflict discriminant function. The reinforcement learning model updates the reward function during each adjustment to maximize the consistency of the association relationship, ensuring the rationality and consistency of the semantic association results and generating optimized high-order semantic relationships. Through the dynamic adjustment mechanism of semantic priority and real-time expansion function, the adaptability and integration efficiency of the model in complex business scenarios are improved.

8. The intelligent market supervision data governance system based on multi-level data sharing and exchange according to claim 1 is characterized in that: The data asset management module includes: The data resource directory unit is used to form a hierarchical directory through the metadata description of the data resource to support the retrieval and acquisition of the data resource; Data asset heat analysis unit, used to dynamically calculate data heat based on data access frequency, citation counts, and page views, and adjust storage and access strategies to optimize system performance; The data link analysis unit is used to combine SQL logs and stored procedure analysis to automatically generate a link relationship map of metadata. The update cycle of the link relationship map is dynamically adjusted according to the system log traffic. The optimized analysis process is achieved through distributed log analysis technology, and version change notification and difference analysis functions are provided.

9. The data governance method of the intelligent market supervision data governance system based on multi-level data sharing and exchange according to any one of claims 1 to 8 is characterized in that: The method comprises: Share and exchange market regulatory data across departments, regions, and hierarchies, and implement data access control and dynamic resource catalog management through policy-driven approaches. Conduct standardized management of market regulatory data throughout its life cycle to ensure shared data complies with standardization requirements, and optimize standardization processes through dynamic updates of data standards and version difference analysis; Conduct quality management for the entire life cycle of market regulatory data, receive standardized data, and form a closed-loop data quality management process through the identification, allocation, management, and re-testing of problematic data; Optimize and analyze multidimensional data based on multi-star data models, receive high-quality data to build correlation models across business systems, and achieve data integration and analysis; Comprehensively manage data assets, store, optimize, and display analysis results of data models, including dynamically adjusting data storage strategies based on access popularity, and improving data usage efficiency through data link analysis.

Images