Enterprise data link governance and value management method and system

Abstract

The application discloses an enterprise data link governance and value management method and system, adopts a distributed data collection engine to perform real-time grabbing on multi-source heterogeneous data of various business systems in an enterprise, identifies different formats of original data types through a preset data source adapter, adopts a relational database connector to perform extraction if structured data is detected, enables a document analysis module to perform content extraction if the data is identified as unstructured data, and obtains an initial data set containing metadata tags; according to a pre-established data standardization rule library, performs format conversion and field mapping on the initial data set, removes duplicate records and abnormal values through a data cleaning algorithm, automatically evaluates data quality grades by adopting a naive Bayes classifier, triggers a data repair process if a data quality score is lower than a preset threshold, and obtains standardized data meeting unified standards. The application effectively improves the standardization and value utilization efficiency of data management.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly discloses an enterprise data link governance and value management method and system. BACKGROUND

[0002] With the deepening of digital transformation, enterprise data has become a core production factor and strategic resource, and data link governance and value management have become a key capability that determines the competitive advantage of enterprises.

[0003] In a data-driven business environment, how to effectively manage the whole link process from data collection to application directly affects the decision-making quality and operational efficiency of enterprises. The current data governance solutions on the market have obvious shortcomings. Most solutions only focus on the optimization of a single link, and lack unified planning and coordination mechanisms for the whole link.

[0004] Traditional methods are inefficient in handling massive heterogeneous data, making it difficult to achieve effective inventory and classified management of data assets. At the same time, the existing technical framework has limited capabilities in data security protection and value quantification, and cannot meet the urgent needs of enterprises for fine-grained management of data governance. The core challenge faced by enterprise data governance stems from the difficulty of unified collection and standardized processing of multi-source heterogeneous data.

[0005] The data formats, structures, and quality standards generated by different business systems are different, and traditional data collection techniques cannot achieve efficient cross-system data integration, resulting in serious data island phenomenon. This scattered state of data further exacerbates the complexity of data quality control, with a large amount of dirty data, duplicate data, and inconsistent data, seriously affecting the subsequent data processing and analysis effect. The cumulative effect of data quality problems makes it difficult for enterprises to build a reliable data service system and provide accurate and reliable data support for business decision-making. More critically, the lack of effective data value quantification mechanism makes it difficult for enterprises to accurately assess the cost-effectiveness of data governance investment and develop reasonable data asset management strategies.

[0006] Therefore, how to build a unified management system covering the whole link of data collection, governance, and application, and achieve standardized processing and value quantification evaluation of multi-source data, has become a key problem that needs to be solved in the field of enterprise data governance. SUMMARY

[0007] The present application provides an enterprise data link governance and value management method and system, which aims to solve at least one of the defects in the prior art.

[0008] One aspect of the present application relates to an enterprise data link governance and value management method, comprising the following steps:

[0009] The distributed data acquisition engine is adopted to perform real-time grabbing on multi-source heterogeneous data of various business systems in the enterprise, different formats of original data types are identified through preset data source adapters, if structured data is detected, a relational database connector is adopted for extraction, if unstructured data is identified, a document parsing module is enabled for content extraction, and an initial data set containing metadata tags is obtained;

[0010] According to the pre-established data standardization rule library, the initial data set is subjected to format conversion and field mapping, through a data cleaning algorithm, repeated records and abnormal values are eliminated, a naive Bayes classifier is adopted to automatically evaluate the data quality level, if the data quality score is lower than the preset threshold, a data repair process is triggered, and standardized data conforming to the unified standard is obtained;

[0011] The hierarchical clustering algorithm is adopted to automatically classify the standardized data according to business attributes and usage frequency, a multi-dimensional data asset directory system is established, a hash algorithm is adopted to generate a unique data asset identifier, if the data contains sensitive information, a data desensitization technique is applied for security processing, the security level and access permission of each type of data asset are determined;

[0012] According to the data usage frequency, business correlation degree and processing cost, a value quantification evaluation model is constructed, a weighted scoring algorithm is adopted to calculate the comprehensive value index V = α × U + β × R + γ × C of each data asset, the investment return rate and priority ranking of the data asset are determined, wherein U represents the usage frequency weight, R represents the business correlation degree weight, C represents the processing cost weight, and α, β, γ are the corresponding weight coefficients;

[0013] A unified data service interface is constructed through the API gateway and the service bus, identity authentication and access control are performed according to the data request type and permission level of the business application, if the request passes the security check, the corresponding data is extracted from the data asset library and real-time processing is performed, and the standardized data service response meeting the business demand is obtained.

[0014] Further, the distributed data acquisition engine is adopted to perform real-time grabbing on multi-source heterogeneous data of various business systems in the enterprise, different formats of original data types are identified through preset data source adapters, if structured data is detected, a relational database connector is adopted for extraction, if unstructured data is identified, a document parsing module is enabled for content extraction, and an initial data set containing metadata tags is obtained.

[0015] Multi-source heterogeneous data is obtained from various business systems through distributed nodes, real-time grabbing is performed through a preset acquisition protocol, and a first data set containing original data types is obtained;

[0016] For the first data set, the data source adapter is used to identify the data format, if it is structured data, the fields are extracted through the relational database connector, if it is unstructured data, the content is extracted through the document parsing module, and the second data set containing metadata tags is obtained;

[0017] According to the second data set, the metadata tags are associated with the business system attributes, and the preset mapping rules are used to determine the field mapping relationship;

[0018] Through the field mapping relationship, the second data set is imported into the relational database, and the batch insertion method is used to generate the initial data set containing complete business attributes.

[0019] Further, according to the pre-established data standardization rule library, the initial data set is converted and mapped, the repeated records and abnormal values are removed through the data cleaning algorithm, the data quality level is automatically evaluated using the Naive Bayes classifier, and if the data quality score is lower than the preset threshold, the data repair process is triggered, and the steps of obtaining the standardized data conforming to the unified standard include:

[0020] According to the pre-established data standardization rule library, the initial data set is converted and mapped, the repeated records and abnormal values are removed through the data cleaning algorithm, the data quality level is automatically evaluated using the Naive Bayes classifier, and if the data quality score is lower than the preset threshold, the data repair process is triggered, and the steps of obtaining the standardized data conforming to the unified standard include:

[0021] Through the data cleaning tool, the first data set is processed to remove repeated records and abnormal values, if repeated field values are detected in the first data set, the earliest record is retained and the subsequent repeated items are deleted, and if the value exceeds the preset range, it is marked as abnormal and removed, and the second data set after cleaning is obtained;

[0022] The classification tool is used to evaluate the quality level of the second data set, the quality score of the second data set is calculated, and if the quality score is lower than the preset threshold, it is marked as repairable data, and the third data set entering the subsequent process is determined;

[0023] According to the repairable data in the third data set, the automatic repair process is triggered, the third data set is completed and corrected through the preset rule library, and the standardized data conforming to the unified standard is obtained.

[0024] Further, the hierarchical clustering algorithm is used to automatically classify the standardized data according to business attributes and usage frequency, a multi-dimensional data asset directory system is established, a unique data asset identifier is generated using the hash algorithm, and if the data contains sensitive information, the data desensitization technology is applied for security processing, and the steps of determining the security level and access permission of various data assets include:

[0025] According to the pre-established business classification rule library, the normalized data is grouped to obtain an initial grouping result based on business classification, and the initial grouping result is sorted based on frequency data to obtain a classification set with priority labels;

[0026] A directory construction tool is used to hierarchically divide the classification set with priority labels according to the association rules of the asset directory, to obtain a structured directory framework, and to determine an identified directory set;

[0027] If the identified directory set contains sensitive information, the sensitive information is content-screened by a data desensitization tool to obtain a desensitized security set, and the security set is divided into levels according to security level evaluation rules to determine the security attribution of each data asset;

[0028] A permission management tool is used to control and configure the security set according to access permission rules to obtain a final set after permission configuration, and to determine a permission management result that meets the business target.

[0029] Further, a value quantification evaluation model is constructed according to data usage frequency, business correlation degree and processing cost, and a weighted scoring algorithm is used to calculate the comprehensive value index V = α × U + β × R + γ × C of each data asset to determine the investment return rate and priority order of the data asset.

[0030] According to the pre-established classification rule library, the data usage frequency, business correlation degree and processing cost are grouped to obtain a preliminary classification set, and an attribute label is added to each data asset by a labeling tool to obtain an asset set with attribute labels;

[0031] A scoring calculation tool is used to calculate the comprehensive value index of each data asset by weighting the preset coefficients of frequency weight, correlation weight and cost weight, to determine a value sequence after sorting;

[0032] If the comprehensive value index of part of the data assets in the value sequence after sorting is lower than a preset threshold, a filtering tool is used to filter the data assets to obtain an asset subset that meets the investment return conditions, and the asset range that needs to be paid attention to is determined;

[0033] A directory generation tool is used to structurally divide the asset range that needs to be paid attention to according to the association rules of business correlation degree and processing cost to obtain a final classification directory, and to determine an asset list that meets the value quantification target.

[0034] Furthermore, by building a unified data service interface through an API gateway and service bus, authentication and access control are performed based on the data request type and permission level of business applications. If the request passes the security verification, the corresponding data is extracted from the data asset repository and processed in real time to obtain a standardized data service response that meets business needs. The steps include:

[0035] The API gateway receives data requests, parses the initiator's identity information based on the data request type and permission level of the business application, and matches it using a pre-established verification rule base. If the match is successful, an authentication token is generated to determine the legitimacy of the request.

[0036] Receive the authentication token, verify the permission level of the authentication token through the service bus, and if it meets the preset rules, extract the original data set from the data asset library to obtain the data content that can be processed.

[0037] For the original dataset, real-time processing tools are used for format conversion and cleaning, and structured data output is generated according to business needs to obtain intermediate processing results;

[0038] Based on the structured data output, it is distributed to the target application through the service bus. The structured data is encapsulated using standardized templates. If it conforms to the output specifications, the final response content is generated to determine the delivery result.

[0039] Another aspect of the present invention relates to an enterprise data link governance and value management system, used to implement the above-described enterprise data link governance and value management method. The enterprise data link governance and value management system includes:

[0040] The first acquisition module is used to capture multi-source heterogeneous data from various business systems within the enterprise in real time using a distributed data acquisition engine. It identifies different formats of raw data types through a preset data source adapter. If structured data is detected, it is extracted using a relational database connector. If it is identified as unstructured data, it is used to extract the content using a document parsing module to obtain an initial data set containing metadata tags.

[0041] The second acquisition module is used to perform format conversion and field mapping on the initial data set according to the pre-established data standardization rule library, remove duplicate records and outliers through data cleaning algorithms, automatically evaluate the data quality level using a Naive Bayes classifier, and trigger the data repair process if the data quality score is lower than the preset threshold to obtain standardized data that conforms to the unified standard.

[0042] The determination module is used to automatically classify standardized data according to business attributes and usage frequency through hierarchical clustering algorithms, establish a multi-dimensional data asset catalog system, generate unique data asset identifiers using hash algorithms, apply data desensitization technology for security processing if the data contains sensitive information, and determine the security level and access permissions of various data assets.

[0043] The judgment module is used to build a value quantification assessment model based on data usage frequency, business relevance and processing cost. It uses a weighted scoring algorithm to calculate the comprehensive value index V=α×U+β×R+γ×C for each data asset, and judges the return on investment and priority ranking of data assets. U represents the usage frequency weight, R represents the business relevance weight, C represents the processing cost weight, and α, β and γ are the corresponding weight coefficients.

[0044] The third acquisition module is used to build a unified data service interface through the API gateway and service bus. It performs authentication and access control based on the data request type and permission level of the business application. If the request passes the security verification, it extracts the corresponding data from the data asset library and processes it in real time to obtain a standardized data service response that meets the business needs.

[0045] Furthermore, the first acquisition module includes:

[0046] The first acquisition unit is used to acquire multi-source heterogeneous data from various business systems through distributed nodes, and to perform real-time capture using a preset acquisition protocol to obtain a first data set containing the original data types.

[0047] The second acquisition unit is used to identify the data format of the first data set using a data source adapter. If it is structured data, it extracts fields through a relational database connector; if it is unstructured data, it extracts content through a document parsing module to obtain a second data set containing metadata tags.

[0048] The first determining unit is used to determine the field mapping relationship by associating metadata tags with business system attributes based on the second data set and using preset mapping rules.

[0049] The generation unit is used to import the second data set into a relational database through field mapping relationships, and to generate an initial data set containing complete business attributes using a batch insertion method.

[0050] Furthermore, the second acquisition module includes:

[0051] The third acquisition unit is used to perform format conversion and field mapping operations on the initial data set according to the pre-established data standardization rule base, to obtain the first dataset that conforms to the preset format from the initial data set, and to compare the first dataset one by one with the batch processing tool to obtain preliminary results with uniform format and accurate field correspondence.

[0052] The fourth acquisition unit is used to remove duplicate records and outliers from the first dataset using data cleaning tools. If duplicate field values ​​are detected in the first dataset, the earliest record is retained and subsequent duplicates are deleted. If the value exceeds the preset range, it is marked as an anomaly and removed, and the cleaned second dataset is obtained.

[0053] The second determining unit is used to evaluate the quality level of the second dataset using a classification tool, calculate a quality score for the second dataset, and mark it as data to be repaired if the quality score is lower than a preset threshold, thus determining the third dataset to enter the subsequent process.

[0054] The fifth acquisition unit is used to trigger an automated repair process based on the data to be repaired in the third dataset, and to complete and correct the third dataset through a preset rule base to obtain standardized data that conforms to a unified standard.

[0055] Furthermore, the identified modules include:

[0056] The sixth acquisition unit is used to group the standardized data according to the pre-established business classification rule base, obtain the initial grouping results based on business classification, and sort them in combination with usage frequency data to obtain a classification set with priority labels.

[0057] The third determining unit is used to use a directory building tool to hierarchically divide the classification set with priority tags according to the association rules of the asset directory, obtain a structured directory framework, and determine the identified directory set.

[0058] The judgment unit is used to mask sensitive information using data desensitization tools if the identifiable directory set contains sensitive information, obtain the desensitized security set, classify the security level according to the security level assessment rules, and determine the security ownership of each data asset.

[0059] The fourth determining unit is used to control and configure access permissions according to the access permission rules based on the level classification results in the security set through the permission management tool, obtain the final set of permission configuration from the security set, and determine the permission management result that meets the business objectives.

[0060] The beneficial effects achieved by this invention are as follows:

[0061] This invention provides a method and system for enterprise data link governance and value management, offering an integrated solution to the problem of difficult unified management and value assessment of multi-source heterogeneous data within enterprises. First, this invention uses a distributed data acquisition engine to capture data from various business systems in real time. Data source adapters identify data types; structured data is extracted through a relational database connector, while unstructured data is processed by a document parsing module, forming an initial data set containing metadata. Next, the initial data undergoes format conversion and field mapping based on a data standardization rule base. Data cleaning algorithms remove duplicate and abnormal records, and a Naive Bayes classifier automatically assesses data quality. If quality fails to meet standards, a repair process is triggered to ensure data standardization. Then, a hierarchical clustering algorithm classifies the standardized data according to business attributes and usage frequency, establishing a multi-dimensional data asset catalog. A hash algorithm generates unique identifiers, and sensitive data is anonymized to determine security levels and access permissions. Finally, a value quantification assessment model is constructed, using a weighted scoring algorithm to calculate the comprehensive value index of data assets, determining return on investment and priority. A unified data service interface is provided through an API gateway and service bus to achieve authentication and access control, ensuring data security and efficiency to meet business needs. This invention effectively improves the standardization and value utilization efficiency of data management, providing a strong guarantee for the efficient management and decision support of enterprise data assets. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating an embodiment of the enterprise data link governance and value management method of the present invention. Detailed Implementation

[0063] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0064] like Figure 1 As shown, the first embodiment of the present invention proposes an enterprise data link governance and value management method, including the following steps:

[0065] Step S100: Use a distributed data acquisition engine to capture multi-source heterogeneous data from various business systems within the enterprise in real time. Identify different formats of raw data types through a preset data source adapter. If structured data is detected, extract it using a relational database connector. If unstructured data is detected, enable the document parsing module to extract the content and obtain an initial data set containing metadata tags.

[0066] A distributed data acquisition engine is a software tool or system based on a distributed computing architecture. It is used to acquire data in real-time or in batches from multiple heterogeneous data sources (such as databases, file systems, API interfaces, and IoT devices), and transmit the acquired data to a data storage or processing platform. Distributed data acquisition engines achieve efficient acquisition, transmission, and management of large-scale data through distributed node collaboration, overcoming the bottlenecks in scalability, reliability, and performance of traditional centralized acquisition models.

[0067] Multi-source heterogeneous data refers to a collection of data that comes from multiple different data sources and differs in data structure, format, protocol, and semantics. Its core characteristics are "multi-source" and "heterogeneousness," and it is widely found in enterprise information systems, Internet applications, the Internet of Things, and other scenarios, representing a typical manifestation of data diversity in the digital age.

[0068] The data source adapter is a key component of the data acquisition layer. It parses the protocols, formats, and structures of different data sources, identifies the original data types, and performs standardized conversions to ensure that heterogeneous data can be processed uniformly by subsequent systems. This process is similar to a "translator," matching data source characteristics through preset rules or algorithms to achieve automated identification and adaptation of structured, unstructured, and semi-structured data.

[0069] Structured data refers to data with a defined data model (such as data type, format, length, constraints, etc.) that is logically expressed and stored in a two-dimensional table format. The characteristics of structured data are fixed format, clear structure, and ease of storage, management, and retrieval using relational databases or Structured Query Language (SQL).

[0070] Unstructured data refers to data without a predefined data model or unified organizational structure. It is flexible in format and diverse in content, making it impossible to store and manage directly using two-dimensional tables or the row / column structure of relational databases. The structure of this type of data is implicit in its content, requiring techniques such as text analysis, image recognition, and natural language processing (NLP) to extract valuable information.

[0071] A relational database connector is a tool or component used to enable data interaction and integration between different systems and relational databases (such as MySQL, Oracle, SQL Server, etc.). Its core functions are to establish communication links, parse database protocols, and encapsulate operation interfaces, enabling external applications, data platforms, or analysis tools to securely and efficiently access and manage structured data in relational databases.

[0072] A document parsing module is a software component or system that uses text processing, pattern recognition, machine learning, and other technologies to extract content, convert formats, and perform semantic analysis on various unstructured or semi-structured documents. The document parsing module transforms the raw data in a document into structured information that a computer can understand and process, facilitating subsequent data storage, analysis, retrieval, or automated workflow invocation.

[0073] Metadata tags are structured keywords or phrases used to describe data or files, designed to quickly convey the core characteristics, attributes, context, or classification information of the data through concise identification. Metadata tags are a lightweight representation of metadata ("data about data"), typically existing as single words, phrases, or standardized terms, used to assist in data retrieval, classification, management, and understanding.

[0074] Step S200: Based on the pre-established data standardization rule base, the initial data set is formatted and field mapped. Duplicate records and outliers are removed by data cleaning algorithms. The data quality level is automatically evaluated by a Naive Bayes classifier. If the data quality score is lower than the preset threshold, the data repair process is triggered to obtain standardized data that meets the unified standards.

[0075] A data standardization rule base is a structured set of rules used to define unified standards that data must follow during storage, processing, and exchange. It aims to eliminate format differences, semantic ambiguities, and logical conflicts among multi-source data, ensuring data consistency, accuracy, and usability. The data standardization rule base is a core component of the data governance system, providing the basis for execution and quality benchmarks for data cleaning, transformation, and integration operations.

[0076] Format conversion is the process of converting the storage format, data type, and representation of initial data into a standard format specified by a rule base. It is used to resolve data format compatibility issues between different systems (such as date formats, numerical precision, character encoding, etc.).

[0077] Field mapping establishes the correspondence between initial data fields and target fields, ensuring accurate matching of data in terms of structure and semantics. It is used to resolve inconsistencies in field names, meanings, and business logic across multiple data sources (e.g., "User ID" may be named "User_ID" or "Customer Number" in different systems).

[0078] Removing duplicate records and outliers through data cleaning algorithms refers to the process of using specific algorithmic logic to identify and remove duplicate and redundant records and outlier data points that do not conform to business logic or statistical patterns in the original data, thereby improving data quality. This is one of the core steps of data cleaning, aiming to solve consistency problems (such as duplicate records) and accuracy problems (such as outliers) in the data, providing a reliable foundation for subsequent data analysis and modeling.

[0079] By using the Naive Bayes algorithm (a probabilistic classification model based on Bayes' theorem and the assumption of conditional independence of features), multiple evaluation indicators of data quality (such as completeness, accuracy, consistency, timeliness, etc.) are modeled. Through training samples, the probabilistic relationship between each indicator and the data quality level (such as excellent, good, medium, poor) is learned, thereby realizing the automatic classification and judgment of new data quality levels.

[0080] If the data quality score falls below a preset threshold, a data repair process is triggered. Obtaining standardized data that meets unified standards is a closed-loop management mechanism in data quality control. By quantitatively assessing data quality and comparing it with preset standards, repair actions are automatically triggered for data that does not meet the standards, ensuring that the final data complies with business rules and technical specifications. Its core logic is: using data quality scores as the basis for judgment and automated processes as the means of repair, a closed-loop improvement is achieved from 'unqualified' to 'qualified' data.

[0081] Step S300: Automatically classify standardized data according to business attributes and usage frequency using hierarchical clustering algorithm, establish a multi-dimensional data asset catalog system, generate unique data asset identifiers using hash algorithm, apply data desensitization technology for security processing if the data contains sensitive information, and determine the security level and access permissions of various data assets.

[0082] Automatically classifying standardized data according to business attributes and usage frequency using hierarchical clustering algorithms is a data asset classification step in data governance. Hierarchical clustering algorithms perform unsupervised grouping of standardized data, automatically discovering similarities between data and constructing a hierarchical classification system based on the data's business attribute characteristics (such as theme, source, and sensitivity) and usage frequency indicators (such as access count and update frequency), ultimately forming a data asset catalog that conforms to business logic.

[0083] A multi-dimensional data asset catalog system is a framework that uses data assets as its core to structure, classify, describe, and manage data from multiple business perspectives and technical dimensions. Through systematic hierarchical classification, the multi-dimensional data asset catalog system enables the visualization, retrieval, and management of data assets, thereby improving the efficiency and value of data asset utilization.

[0084] The generation of unique data asset identifiers using hash algorithms refers to the process of using the one-way mapping property and collision resistance of hash functions to convert the characteristic information of data assets (such as content, metadata, etc.) into a fixed-length string (hash value), which serves as the unique identifier (UID) for that data asset in the system. This process ensures through algorithms that each data asset corresponds to a unique identifier, and that the identifier can quickly verify the consistency and integrity of the data.

[0085] Applying data anonymization techniques to securely process data containing sensitive information is a crucial step in data security governance. This involves transforming data containing sensitive information using anonymization techniques, ensuring that it cannot be directly linked to real sensitive content while maintaining business logic and data structure, thereby reducing the risk of data leakage. The core logic is: first, identify the sensitive information in the data; then, select an appropriate anonymization strategy based on the sensitivity level and usage scenario to achieve the security goal of 'data usable but invisible'.

[0086] Determining the security level and access permissions for various data assets is a core component of the data security management system. This involves assessing the sensitivity, business value, and compliance risks of data to classify different types of data assets into clear security levels (such as public, internal, sensitive, confidential, etc.), and then developing differentiated access control policies based on these levels to ensure that data is used reasonably under the premise of legality and compliance.

[0087] Step S400: Construct a value quantification assessment model based on data usage frequency, business relevance, and processing cost. Use a weighted scoring algorithm to calculate the comprehensive value index V=α×U+β×R+γ×C for each data asset, and determine the return on investment and priority ranking of the data assets. Here, U represents the usage frequency weight, R represents the business relevance weight, C represents the processing cost weight, and α, β, and γ are the corresponding weight coefficients.

[0088] A value quantification assessment model is a tool that uses a multi-dimensional indicator system and mathematical algorithms to quantitatively evaluate the business value of data assets. This model uses data usage frequency, business relevance, and processing cost as core evaluation dimensions. Through standardized indicator calculations and weight allocation, it transforms data value into measurable quantitative results, providing a scientific basis for prioritizing data assets, allocating resources, and making decisions.

[0089] The weighted scoring algorithm is a quantitative method for calculating the comprehensive value of data assets by assigning different weights to multi-dimensional evaluation indicators. The weighted scoring algorithm transforms various key attributes of data assets (such as usage frequency, business relevance, and processing costs) into standardized scores, combines them with preset weights, and then sums them to obtain a Composite Value Index (CVI). This index is used to intuitively measure the relative value of data assets and support data management decisions.

[0090] Return on investment (ROI) for data assets is a quantitative indicator that measures the benefits of data asset input and output. It assesses the economic value and investment effectiveness by comparing the total cost and business benefits of data assets in the stages of collection, storage, governance, analysis, and application.

[0091] Data asset prioritization is a process of assessing and ranking data assets based on multiple dimensions such as business objectives, value contribution, and risk level. Its core is to establish an assessment model to assign priority labels (such as "strategic," "priority," and "normal") to data assets, guiding the rational allocation of resources (human resources, budget, and technology).

[0092] Step S500: Build a unified data service interface through the API gateway and service bus. Perform authentication and access control based on the data request type and permission level of the business application. If the request passes the security verification, extract the corresponding data from the data asset library and process it in real time to obtain a standardized data service response that meets the business needs.

[0093] An API gateway is the unified entry point for a system. It receives all client requests, forwards them to backend services according to routing rules, and provides functions such as authentication, authorization, rate limiting, and monitoring. Like a "gatekeeper for data services," the API gateway shields the complexity of backend services and exposes a unified access address to the outside world.

[0094] A service bus is the communication hub connecting different services in a distributed system, supporting asynchronous message passing, event-driven operations, and decoupling between services. Like a "data pipeline," the service bus allows different services to exchange data securely and reliably using either a publish-subscribe or point-to-point model.

[0095] The unified data service interface, through the collaboration of API gateway and service bus, encapsulates data capabilities scattered in different systems (such as ERP, CRM, big data platform) into standardized interfaces (such as RESTful API), providing a unified, secure, and manageable data access channel to the outside world.

[0096] Authentication is the process of verifying the legitimacy of a user or application's identity; it answers the question "Who are you?" By verifying credentials (such as usernames / passwords, API keys, digital certificates, etc.), it ensures that only authorized entities can access the system.

[0097] Access control is a mechanism that restricts access to specific resources based on authentication results, user or application permission levels, and data request types; in other words, it answers the question "What can you do?" Its core objective is to implement the "principle of least privilege," ensuring that data is used only within its legal scope.

[0098] If the request passes security verification, retrieving the corresponding data from the data asset repository and processing it in real time is a crucial step in the data service chain. This means that after a user or application's access request passes security verification (authentication, permission review, compliance checks, etc.), the system automatically retrieves the target data from the data asset repository (a unified management platform storing structured / unstructured data), and performs cleaning, transformation, and analysis on the data through a real-time processing engine, ultimately returning the processing result or triggering subsequent business processes. Its core logic is: based on secure access, and using real-time data processing as a means, to achieve on-demand supply and value release of data assets.

[0099] Obtaining a standardized data service response that meets business needs refers to the process by which a data service system, after completing security verification, authentication, permission matching, and data extraction and processing of user requests, returns data results that meet the needs of business applications or users according to predefined unified formats, specifications, and protocols. This process must ensure the accuracy, integrity, timeliness, and availability of the data, and improve the compatibility and usability of the data in different business scenarios through standardized output.

[0100] Furthermore, the enterprise data link governance and value management method provided in this embodiment includes step S100 as follows:

[0101] Step S110: Obtain multi-source heterogeneous data from various business systems through distributed nodes, and perform real-time capture using a preset acquisition protocol to obtain a first data set containing the original data types.

[0102] Imagine a retail enterprise scenario when acquiring heterogeneous data from various business systems via distributed nodes. The retail enterprise's business systems include an inventory management system, an order processing system, and a customer relationship management system. These systems have different data types. The inventory system may generate structured data, such as product IDs, quantities, and prices; the order system generates semi-structured data, such as JSON-formatted order logs; and the customer relationship system contains unstructured data, such as text records of customer feedback.

[0103] Distributed nodes are deployed on servers in different regions to monitor data updates from these systems in real time. The pre-defined data collection protocol is based on HTTP or MQ message queue protocols, ensuring fast and stable data transmission to the central node to generate the initial data set. This initial data set contains raw data types, such as CSV files, JSON logs, and text documents. This approach guarantees the real-time nature and integrity of data collection, laying the foundation for subsequent processing.

[0104] Step S120: For the first data set, the data format is identified using the data source adapter. If it is structured data, the fields are extracted through the relational database connector. If it is unstructured data, the content is extracted through the document parsing module to obtain the second data set containing metadata tags.

[0105] For the first dataset, the data source adapter plays a crucial role. For example, the data source adapter can identify CSV files as structured data, JSON logs as semi-structured data, and text documents as unstructured data. For structured data, the relational database connector extracts fields, such as "Product ID: 1001", "Quantity: 50", and "Price: 99.5" from a CSV file in an inventory system. For unstructured data, the document parsing module processes customer feedback text, extracts key content such as "Positive feedback: Good product quality" and "Negative feedback: Slow delivery speed", and generates metadata tags such as "Data source: Customer Relationship System", "Data type: Text", and "Sentiment: Positive". The resulting second dataset contains metadata tags, clearly identifying the data's source, type, and key attributes, improving data traceability and understandability, and facilitating subsequent correlation analysis.

[0106] Step S130: Based on the second data set, associate the metadata tags with the business system attributes, and determine the field mapping relationship using preset mapping rules.

[0107] When mapping fields based on the second dataset, the association between metadata tags and business system attributes is crucial. For example, if the "Product ID" field in the inventory system needs to correspond to the "Product Number" field in the order system, the default mapping rule can be defined as follows: if the values ​​of both fields are in numeric format and have the same length, they are considered the same field.

[0108] The mapping rules take into account the context of the fields. For example, "Product ID" comes from the inventory system, while "Product Number" comes from the order system; both are related to the core entity "Product" in the retail business. Through this mapping relationship, data in the second dataset is accurately imported into the relational database.

[0109] Step S140: Import the second data set into the relational database through field mapping relationship, and generate an initial data set containing complete business attributes using batch insertion method.

[0110] Batch insertion further improves efficiency. For example, 1000 records are inserted into the database as a batch, generating an initial dataset containing complete business attributes, such as "Order ID: 20001", "Product ID: 1001", "Quantity: 50", "Price: 99.5", and "Customer Feedback: Good product quality". The advantage of this approach is that it not only ensures data consistency and integrity but also significantly improves data import efficiency and reduces system load.

[0111] The deployment of distributed nodes can flexibly adapt to the business needs of retail enterprises. For example, if an enterprise has multiple warehouses across the country, distributed nodes can be deployed on servers in each warehouse to collect inventory change data in real time. It should be noted that this distributed architecture can effectively cope with network latency and single points of failure, ensuring the stability of data collection.

[0112] The preset data acquisition protocol supports priority settings; for example, order data can be collected and transmitted before customer feedback data to meet the needs of real-time order processing. This approach improves system robustness and responsiveness.

[0113] The combined use of the data source adapter and document parsing module enables retail businesses to better handle complex data. For unstructured customer feedback data, the document parsing module incorporates natural language processing technology to extract keywords and sentiment indicators, generating metadata tags. These tags, when associated with business system attributes, help businesses quickly identify customer needs and problems; for example, discovering that "slow delivery" is a frequently reported customer issue allows for optimization of logistics processes.

[0114] Establishing field mapping relationships is a crucial step in data integration. For example, in a retail scenario, mapping rules are optimized based on historical data samples. For instance, analyzing order and inventory data from the past year reveals that the matching rate between "product ID" and "product number" reaches over 95%, thus confirming the reliability of the mapping rules.

[0115] Preferably, the enterprise data link governance and value management method provided in this embodiment includes step S200 as follows:

[0116] Step S210: Based on the pre-established data standardization rule base, perform format conversion and field mapping operations on the initial data set, obtain the first dataset that conforms to the preset format from the initial data set, and use a batch processing tool to compare the first dataset one by one to obtain preliminary results with uniform format and accurate field correspondence.

[0117] In the data processing scenarios of retail enterprises, the format conversion and field mapping operations of the initial dataset are analyzed in detail from multiple perspectives. First, regarding the application of a data standardization rule base, consider a retail enterprise's inventory and sales data that need to be integrated. Inventory data may be stored in tabular form with a field named "Item Number," while sales data has a field named "Product Code." By using a pre-defined rule base, both are mapped to a unified "Product Identifier," ensuring field consistency during data integration. This approach effectively avoids confusion caused by differences in field naming.

[0118] For batch processing tools, a line-by-line comparison is performed using product identifiers extracted from inventory and sales data. Assuming the initial dataset contains 100,000 records, the batch processing tool compares records in batches of 5,000, checking if field values ​​conform to a preset format, such as "product identifier" being a 6-digit number. Records that do not conform to the format, such as a 5-digit number, are marked as abnormal and await further processing.

[0119] Step S220: Use a data cleaning tool to remove duplicate records and outliers from the first dataset. If duplicate field values ​​are detected in the first dataset, retain the earliest record and delete subsequent duplicates. If the value exceeds a preset range, mark it as an anomaly and remove it to obtain the cleaned second dataset.

[0120] In the application of data cleaning tools, the removal of duplicate records and outliers is based on specific scenarios. For example, if the same order number "20001" appears twice in a retail company's sales data, the cleaning tool will retain the earliest record and delete subsequent duplicates. Simultaneously, if a product's price field value is "9999.5," far exceeding the normal range of "10.0 to 500.0," it will be marked as an anomaly and removed to ensure the accuracy of the second dataset.

[0121] Step S230: Use a classification tool to evaluate the quality level of the second dataset, calculate the quality score for the second dataset, and if the quality score is lower than the preset threshold, mark it as data to be repaired and determine the third dataset to enter the subsequent process.

[0122] For the quality assessment of the classification tool, assuming the second dataset contains 80,000 records, a quality score is derived by calculating field completeness and consistency. A preset threshold of 80 points is set; if a batch of data scores 75 points, it is marked as data requiring repair. This assessment method helps to filter out data that needs further processing, ensuring that the third dataset entering subsequent processes is of higher quality.

[0123] Step S240: Based on the data to be repaired in the third dataset, trigger the automated repair process, complete and correct the third dataset through a preset rule base, and obtain standardized data that conforms to a unified standard.

[0124] In the automated repair process, data to be repaired in the third-party dataset can be completed and corrected using a rule base. For example, if a sales record lacks a "product identifier," the system can automatically complete it to "100001" by associating the order number with inventory data. Furthermore, if the price field is empty, the system can complete it to "99.5" based on the historical average price. This automated repair improves data integrity and provides a reliable foundation for subsequent business analysis.

[0125] From an overall process perspective, the aforementioned stages support each other, forming a complete data processing chain from format conversion to the final generation of standardized data. The initial dataset is processed using a standardized rule base and batch processing tools to ensure field consistency and format uniformity; cleaning tools and quality assessments further enhance data reliability; automated repair resolves legacy issues, ultimately generating standardized data that conforms to unified standards. This multi-layered processing approach significantly improves the efficiency and accuracy of data processing for retail enterprises, providing more valuable support for business decisions.

[0126] Furthermore, the enterprise data link governance and value management method provided in this embodiment includes step S300 as follows:

[0127] Step S310: Based on the pre-established business classification rule library, group the standardized data to obtain the initial grouping results based on business classification, and sort them in combination with the usage frequency data to obtain a classification set with priority labels.

[0128] In the data management scenario of retail enterprises, a series of operations such as grouping and processing standardized data and subsequent catalog construction can be carried out for detailed analysis from multiple perspectives, especially in key aspects such as business classification and access management.

[0129] For the application of business classification rule bases, the standardized data of retail enterprises includes multiple categories such as product sales, inventory management, and customer feedback. Through a pre-established rule base, the data is grouped according to business type; for example, sales data is categorized as "transactions," and inventory data as "warehousing." Combined with usage frequency data—for example, if "transactions" data is queried an average of 500 times per day, while "warehousing" data is queried an average of 200 times—the system will mark "transactions" as high priority, forming a set of categories with priority tags. This grouping method helps to quickly identify core business data.

[0130] Step S320: Using a directory building tool, for the category set with priority tags, perform hierarchical division according to the association rules of the asset directory, obtain a structured directory framework, and determine the identifiable directory set.

[0131] When using catalog building tools, categories with priority markers are hierarchically divided according to the association rules of the asset catalog. For example, under "Transactions," further subdivided into "Online Sales" and "Offline Sales," a tree-like catalog framework is formed, with "Online Sales" at the top due to its higher priority. This structured catalog framework facilitates the rapid location and management of data assets.

[0132] Step S330: If the identifiable directory set contains sensitive information, the sensitive information is masked using a data desensitization tool to obtain the desensitized security set, and the security level is classified according to the security level assessment rules to determine the security ownership of each data asset.

[0133] For sensitive information that may exist within a identifiable catalog set, the application of data anonymization tools is particularly important. Suppose a catalog contains customer feedback data, including customer contact information fields such as "13800138000". The system will mask this as "138****8000", forming a secure, anonymized catalog. Simultaneously, based on security level assessment rules, catalogs containing customer information are marked as "high security level," while others, such as inventory data, are marked as "low security level," thus clearly defining the security ownership of data assets.

[0134] Step S340: Using the permission management tool, control configuration is performed according to the access permission rules based on the level classification results in the security set. The final set after permission configuration is obtained from the security set, and the permission management result that meets the business objectives is determined.

[0135] In the application of access control tools, the security set is categorized by level, and access permission rules are configured accordingly. For example, "high security level" customer data may only be accessible to the marketing department manager, while "low security level" inventory data may be accessible to all warehouse personnel. This control configuration forms the final set of permissions, ensuring that data access aligns with business objectives. This hierarchical access control effectively protects sensitive information while guaranteeing the availability of business data.

[0136] The overall logic of the above process, from business classification to permission configuration, is tightly integrated at every stage. Grouping and prioritization lay the foundation for subsequent directory construction, while data masking and security level division provide assurance for access management. This multi-layered approach not only enhances the organization of data management but also achieves a balance between security and access efficiency, providing strong support for achieving the business goals of retail enterprises.

[0137] Preferably, the enterprise data link governance and value management method provided in this embodiment includes step S400 as follows:

[0138] Step S410: Based on the pre-established classification rule library, group the data according to usage frequency, business relevance and processing cost to obtain a preliminary classification set, and add attribute tags to each data asset through the annotation tool to obtain an asset set with attribute tags.

[0139] For data management scenarios in retail enterprises, a pre-established classification rule base groups data based on usage frequency, business relevance, and processing cost. For example, standardized data in a retail enterprise includes product sales data, promotional activity data, and logistics and distribution data. The classification rule base can set a usage frequency threshold of 300 queries per day, business relevance based on its correlation with core sales processes, and processing cost based on data storage and computing resource consumption. Assuming product sales data is queried an average of 600 times per day (high business relevance, medium processing cost), promotional activity data is queried 250 times (medium relevance, low cost), and logistics and distribution data is queried 100 times (low relevance, high cost), after initial grouping, product sales data is categorized into a high-frequency, high-relevance group, promotional activity data into a medium-frequency, medium-relevance group, and logistics and distribution data into a low-frequency, low-relevance group, forming a preliminary classification set.

[0140] The labeling tool adds attribute tags to the data assets in the initial classification set. For example, product sales data is labeled as "core business, high frequency, computationally intensive," promotional activity data is labeled as "auxiliary business, medium frequency, low cost," and logistics and distribution data is labeled as "support business, low frequency, high cost." These tags not only reflect the business attributes of the data but also provide a basis for subsequent value assessment.

[0141] The addition of tags should be combined with the business scenario to ensure that the tags accurately reflect the characteristics of the data. For example, the "high frequency" tag for product sales data is based on its average of 600 queries per day, while "computationally intensive" stems from the need for real-time analysis.

[0142] Step S420: Using a scoring calculation tool, for the asset set with attribute labels, perform weighted calculation based on preset coefficients of frequency weight, association weight and cost weight to obtain the comprehensive value index of each data asset and determine the sorted value sequence.

[0143] The scoring tool calculates the comprehensive value index of data assets based on preset coefficients of frequency weight 0.5, association weight 0.3, and cost weight 0.2.

[0144] For example, the frequency score for product sales data is 600 / 1000*0.5=0.3, the correlation score is 0.8*0.3=0.24, the cost score is 0.6*0.2=0.12, and the total index is 0.66. The total index for promotional activity data is 0.45, and the total index for logistics and delivery data is 0.25.

[0145] Step S430: If the comprehensive value index of some data assets in the sorted value sequence is lower than the preset threshold, the data assets are filtered by the screening tool to obtain a subset of assets that meet the investment return conditions and determine the scope of assets to focus on.

[0146] If the preset threshold is 0.4, then product sales data and promotional activity data are included in the value sequence, while logistics and delivery data are filtered out. This value quantification method helps to prioritize high-value data.

[0147] The screening tool further filters the value sequence, retaining a subset of assets that meet the return on investment criteria. For example, merchandise sales data is prioritized due to its high-frequency query frequency and relevance to core business operations, while promotional activity data is also included in the subset because of its low cost and support for marketing activities. Logistics and distribution data is eliminated due to its low frequency and high cost. This screening ensures that resources are allocated to high-return data.

[0148] Step S440: Using a catalog generation tool, the scope of assets of priority is structurally divided according to the association rules of business relevance and processing cost to obtain the final classification catalog and determine the list of assets that meet the value quantification target.

[0149] The catalog generation tool structures and divides priority asset categories based on business relevance and processing costs. For example, product sales data is subdivided into "online orders" and "in-store transactions," while promotional activity data is subdivided into "holiday promotions" and "daily discounts," forming a tree-like catalog structure. Online orders, due to their high value index, are placed at the top of the catalog for easy and quick location. This structured catalog clearly displays the hierarchy of data assets, supporting efficient management and querying.

[0150] The above process, from grouping to catalog generation, is progressive, ensuring that the classification and value assessment of data assets meet the business needs of retail enterprises. The classification rule base provides the basis for grouping, attribute tags and value indices support filtering, and the structured catalog facilitates data management, thereby improving the overall efficiency of data asset utilization.

[0151] Furthermore, the enterprise data link governance and value management method provided in this embodiment includes step S500 as follows:

[0152] Step S510: Receive data requests through the API gateway, parse the identity information of the initiator according to the data request type and permission level of the business application, match it using a pre-established verification rule base, and generate an authentication token if the match is successful to determine the legitimacy of the request.

[0153] In the data management scenarios of retail enterprises, the process of receiving data requests through API gateways can be analyzed in detail from the entire process of identity verification to data delivery. Combined with business needs and access control, this ensures the security and efficient utilization of data assets.

[0154] This document describes a rule-based identity verification mechanism for the process of receiving data requests and parsing the initiator's identity information through an API gateway. In a retail enterprise, assuming a business application initiates a request to retrieve product sales data, the API gateway first parses the identity information in the request, such as the user ID and request type. A pre-built verification rule base stores the permission levels for different business applications; for example, sales department applications are granted high permissions, while logistics department applications have lower permissions. If a sales department application initiates a request, the rule base matches its identity information and generates an authentication token to confirm the request's legitimacy. This method ensures that only qualified initiators can proceed to the next step.

[0155] Step S520: Receive the authentication token, verify the permission level of the authentication token through the service bus, and if it meets the preset rules, extract the original data set from the data asset library to obtain the data content that can be processed.

[0156] In the process of verifying authentication token permission levels via the service bus, retail enterprises may set rules requiring applications with higher privileges to access core sales data. For example, if a sales department's application token shows a permission level of A, which meets the preset rules, the service bus will allow the extraction of raw data sets from the data asset repository, such as data tables containing product sales records. However, if a logistics department's application token shows a permission level of C, access may be restricted, allowing only access to non-core data. This tiered verification mechanism effectively protects sensitive data.

[0157] Step S530: For the original dataset, use real-time processing tools to perform format conversion and cleaning, generate structured data output according to business needs, and obtain intermediate processing results.

[0158] In retail scenarios, real-time processing tools play a crucial role in the format conversion and cleaning of raw datasets. Suppose the extracted raw data contains product sales records, but the formats are inconsistent; some records are stored in text format, while others contain redundant fields. The processing tool will convert the data to a standard format, remove invalid information, and ultimately generate structured data output, such as sales reports organized by date and product category. This processing facilitates subsequent business analysis.

[0159] Step S540: Based on the structured data output, distribute it to the target application through the service bus, encapsulate the structured data using a standardized template, and generate the final response content if it conforms to the output specifications to determine the delivery result.

[0160] In the stage of distributing and encapsulating data to the target application via the service bus, assuming the structured data output is a sales report and the target application is the sales department's analytics system, the service bus distributes the data to this system and encapsulates it using standardized templates. This ensures that field names and formats conform to output specifications, ultimately generating response content that can be directly used for analysis. This standardized delivery improves the usability of data in business applications.

[0161] Each of these steps is closely aligned with the business needs of retail enterprises, forming a complete chain from identity verification to data delivery. This ensures the efficient flow of data assets under access control, while structuring and standardizing the data to enhance its business applicability. This end-to-end management is of great significance for the utilization of data assets by retail enterprises.

[0162] This invention relates to an enterprise data link governance and value management system, used to implement the aforementioned enterprise data link governance and value management method. The enterprise data link governance and value management system includes a first acquisition module, a second acquisition module, a determination module, a judgment module, and a third acquisition module. The first acquisition module uses a distributed data acquisition engine to capture multi-source heterogeneous data from various business systems within the enterprise in real time. It identifies different formats of raw data types through a preset data source adapter. If structured data is detected, it uses a relational database connector for extraction; if unstructured data is identified, it uses a document parsing module for content extraction, obtaining an initial data set containing metadata tags. The second acquisition module performs format conversion and field mapping on the initial data set according to a pre-established data standardization rule base. It removes duplicate records and outliers using a data cleaning algorithm, and automatically evaluates the data quality level using a Naive Bayes classifier. If the data quality score is lower than a preset threshold, it triggers a data repair process to obtain standardized data conforming to a unified standard. The determination module uses... The system automatically classifies standardized data according to business attributes and usage frequency using a hierarchical clustering algorithm, establishing a multi-dimensional data asset catalog system. A hash algorithm is used to generate unique data asset identifiers. If the data contains sensitive information, data anonymization technology is applied for security processing, determining the security level and access permissions for each type of data asset. A judgment module is used to construct a value quantification assessment model based on data usage frequency, business relevance, and processing cost. A weighted scoring algorithm is used to calculate the comprehensive value index V=α×U+β×R+γ×C for each data asset, determining the return on investment and priority ranking of the data assets. Here, U represents the usage frequency weight, R represents the business relevance weight, C represents the processing cost weight, and α, β, and γ are the corresponding weight coefficients. A third acquisition module is used to build a unified data service interface through an API gateway and service bus. Authentication and access control are performed based on the data request type and permission level of the business application. If the request passes security verification, the corresponding data is extracted from the data asset repository and processed in real time to obtain a standardized data service response that meets business needs.

[0163] Furthermore, the enterprise data link governance and value management system provided in this embodiment includes a first acquisition module comprising a first acquisition unit, a second acquisition unit, a first determination unit, and a generation unit. The first acquisition unit is used to acquire multi-source heterogeneous data from various business systems through distributed nodes, and performs real-time capture using a preset acquisition protocol to obtain a first data set containing the original data types. The second acquisition unit is used to identify the data format of the first data set using a data source adapter; if the data is structured, it extracts fields through a relational database connector; if the data is unstructured, it extracts content through a document parsing module to obtain a second data set containing metadata tags. The first determination unit is used to associate the second data set with business system attributes through metadata tags and determine field mapping relationships using preset mapping rules. The generation unit is used to import the second data set into a relational database through the field mapping relationships and generate an initial data set containing complete business attributes using a batch insertion method.

[0164] Preferably, the enterprise data link governance and value management system provided in this embodiment includes a second acquisition module comprising a third acquisition unit, a fourth acquisition unit, a second determination unit, and a fifth acquisition unit. The third acquisition unit performs format conversion and field mapping operations on an initial dataset based on a pre-established data standardization rule base, acquiring a first dataset conforming to a preset format from the initial dataset. A batch processing tool is used to compare each item in the first dataset to obtain preliminary results with uniform format and accurate field correspondence. The fourth acquisition unit uses a data cleaning tool to remove duplicate records and outliers from the first dataset. If duplicate field values ​​are detected in the first dataset, the earliest record is retained and subsequent duplicates are deleted. If the value exceeds a preset range, it is marked as an anomaly and removed, acquiring a cleaned second dataset. The second determination unit uses a classification tool to evaluate the quality level of the second dataset, calculating a quality score. If the quality score is lower than a preset threshold, it is marked as data to be repaired, determining a third dataset to enter the subsequent process. The fifth acquisition unit triggers an automated repair process based on the data to be repaired in the third dataset, completing and correcting the third dataset using a preset rule base to obtain standardized data conforming to a unified standard.

[0165] Furthermore, the enterprise data link governance and value management system provided in this embodiment includes a sixth acquisition unit, a third determination unit, a judgment unit, and a fourth determination unit. The sixth acquisition unit is used to group standardized data according to a pre-established business classification rule base, obtain initial grouping results based on business classification, and sort them using usage frequency data to obtain a classification set with priority tags. The third determination unit uses a directory building tool to hierarchically divide the classification set with priority tags according to the association rules of the asset directory, obtain a structured directory framework, and determine an identifiable directory set. The judgment unit, if the identifiable directory set contains sensitive information, uses a data desensitization tool to mask the sensitive information, obtain a desensitized security set, and classifies it according to security level assessment rules to determine the security ownership of each data asset. The fourth determination unit uses a permission management tool to control and configure the permission configuration results in the security set according to access permission rules, obtain the final set after permission configuration from the security set, and determine the permission management result that meets the business objectives.

[0166] The enterprise data link governance and value management method and system provided in this embodiment have the following specific advantages compared with the prior art:

[0167] I. Data Acquisition and Preprocessing: Full Compatibility and Quality Assurance

[0168] 1. Seamless integration of multi-source heterogeneous data

[0169] By using a distributed data acquisition engine and data source adapter architecture, we can achieve differentiated processing of structured data (relational databases) and unstructured data (documents, logs, etc.), solve the problem of data silos in enterprise internal systems, and ensure that all data is included in the governance scope.

[0170] 2. Automated data quality control

[0171] The quality assessment model based on the Naive Bayes classifier can automatically identify data integrity and consistency defects. Combined with data cleaning algorithms (duplicate removal and outlier handling) and repair processes, it can improve data accuracy to over 95% and reduce manual verification costs.

[0172] II. Data Assetization: Categorized Management and Security Protection

[0173] 1. Dynamic classification and asset catalog system

[0174] By using hierarchical clustering algorithms to classify data according to business attributes (such as marketing and production) and usage frequency (high frequency / low frequency), a multi-dimensional asset catalog is constructed, enabling business personnel to quickly retrieve target data and shorten the data acquisition cycle.

[0175] 2. Sensitive data tiered protection

[0176] For data containing sensitive information (such as user privacy and financial data), data anonymization techniques (such as masking and generalization) are used to achieve "usable but not visible," and combined with security levels and access control, the risk of data leakage is reduced.

[0177] III. Value Quantification and Decision Optimization

[0178] 1. Quantitative evaluation model for data assets

[0179] Based on the value formula V=α×U+β×R+γ×C, data usage frequency (U), business relevance (R), and processing cost (C) are transformed into a measurable comprehensive value index, helping enterprises identify high-value data assets (such as frequently used customer tag data) and optimize data investment direction.

[0180] 2. Data-driven business decisions

[0181] The quantitative assessment results can be directly used for the lifecycle management of data assets (such as archiving low-frequency data to reduce storage costs) or to guide the development of data products (such as prioritizing the release of high-value data APIs), driving the transformation of data from a "cost center" to a "value center".

[0182] IV. Data as a Service: Efficient Sharing and Secure Control

[0183] 1. Unified data service interface system

[0184] By building a standardized data service layer through API gateway and service bus, it supports multiple interface protocols such as RESTful and GraphQL, realizes real-time data response and on-demand supply (such as front-end applications calling user profile data), and improves data service efficiency (response time is reduced from hours to seconds).

[0185] 2. Agile support for business innovation

[0186] Standardized data services can be quickly integrated with new business scenarios (such as AI modeling and BI analysis), avoiding redundant development of data interfaces and shortening the business launch cycle. For example, the risk control department can directly call the "Customer Credit Data API" for real-time risk assessment without waiting for the data department to manually extract the data.

[0187] In summary, the enterprise data link governance and value management method and system provided in this embodiment realizes closed-loop management of data from "collection-governance-assessment-service", helping enterprises build an efficient, secure and quantifiable data value system to support the implementation of digital transformation strategies.

[0188] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A method for enterprise data link governance and value management, characterized in that, Includes the following steps: A distributed data acquisition engine is used to capture multi-source heterogeneous data from various business systems within the enterprise in real time. The preset data source adapter identifies raw data types in different formats. If structured data is detected, a relational database connector is used for extraction. If unstructured data is detected, a document parsing module is enabled for content extraction to obtain an initial data set containing metadata tags. The initial dataset is format-converted and field-mapped according to a pre-established data standardization rule base. Duplicate records and outliers are removed by data cleaning algorithms. The data quality level is automatically evaluated by a Naive Bayes classifier. If the data quality score is lower than a preset threshold, a data repair process is triggered to obtain standardized data that conforms to a unified standard. The normalized data is automatically classified according to business attributes and usage frequency using a hierarchical clustering algorithm to establish a multi-dimensional data asset catalog system. A hash algorithm is used to generate unique data asset identifiers. If the data contains sensitive information, data desensitization technology is applied for security processing to determine the security level and access permissions of various data assets. A value quantification assessment model is constructed based on data usage frequency, business relevance, and processing cost. A weighted scoring algorithm is used to calculate the comprehensive value index V=α×U+β×R+γ×C for each data asset to determine the return on investment and priority ranking of the data assets. U represents the usage frequency weight, R represents the business relevance weight, C represents the processing cost weight, and α, β, and γ are the corresponding weight coefficients. A unified data service interface is built through API gateway and service bus. Authentication and access control are performed according to the data request type and permission level of business applications. If the request passes the security verification, the corresponding data is extracted from the data asset library and processed in real time to obtain a standardized data service response that meets business needs. The steps of using a distributed data acquisition engine to capture multi-source heterogeneous data from various business systems within the enterprise in real time, identifying different formats of raw data types through a preset data source adapter, extracting structured data using a relational database connector, and extracting content using a document parsing module if unstructured data is detected, to obtain an initial data set containing metadata tags include: Multi-source heterogeneous data is obtained from various business systems through distributed nodes, and real-time capture is performed using a preset acquisition protocol to obtain a first data set containing the original data types. For the first data set, a data source adapter is used to identify the data format. If it is structured data, the fields are extracted through a relational database connector. If it is unstructured data, the content is extracted through a document parsing module to obtain a second data set containing metadata tags. Based on the second data set, the field mapping relationship is determined by associating metadata tags with business system attributes and using preset mapping rules. Using the field mapping relationship, the second data set is imported into a relational database, and an initial data set containing complete business attributes is generated using a batch insertion method. The steps of automatically classifying the standardized data according to business attributes and usage frequency using a hierarchical clustering algorithm to establish a multi-dimensional data asset catalog system, generating unique data asset identifiers using a hash algorithm, and applying data desensitization technology for security processing if the data contains sensitive information, and determining the security level and access permissions for various data assets, include: Based on the pre-established business classification rule base, the normalized data is grouped to obtain the initial grouping results based on business classification, and then sorted in combination with usage frequency data to obtain a classification set with priority labels. Using a directory building tool, the categories with priority tags are hierarchically divided according to the association rules of the asset directory to obtain a structured directory framework and determine the identifiable directory set. If the identifiable directory set contains sensitive information, the sensitive information is masked using a data de-identification tool to obtain a de-identified security set. The set is then classified according to security level assessment rules to determine the security ownership of each data asset. Using the access control tool, the access control configuration is performed according to the access permission rules based on the level classification results in the security set. The final set of access permissions after configuration is obtained from the security set, and the access control result that meets the business objectives is determined.

2. The enterprise data link governance and value management method as described in claim 1, characterized in that, The steps of format conversion and field mapping of the initial dataset based on a pre-established data standardization rule base, removal of duplicate records and outliers using data cleaning algorithms, automatic evaluation of data quality level using a Naive Bayes classifier, and triggering a data repair process if the data quality score is lower than a preset threshold to obtain standardized data conforming to a unified standard include: Based on a pre-established data standardization rule base, format conversion and field mapping operations are performed on the initial data set. A first dataset conforming to the preset format is obtained from the initial data set. A batch processing tool is used to compare the first dataset item by item to obtain a preliminary result with uniform format and accurate field correspondence. The first dataset is cleaned by a data cleaning tool to remove duplicate records and outliers. If duplicate field values ​​are detected in the first dataset, the earliest record is retained and subsequent duplicates are deleted. If the value exceeds a preset range, it is marked as an anomaly and removed. The cleaned second dataset is then obtained. The quality level of the second dataset is evaluated using a classification tool. A quality score is calculated for the second dataset. If the quality score is lower than a preset threshold, it is marked as data to be repaired, and the third dataset is determined to enter the subsequent process. Based on the data to be repaired in the third dataset, an automated repair process is triggered. The third dataset is then completed and corrected using a preset rule base to obtain standardized data that conforms to a unified standard.

3. The enterprise data link governance and value management method as described in claim 1, characterized in that, The steps of constructing a value quantification assessment model based on data usage frequency, business relevance, and processing cost, and using a weighted scoring algorithm to calculate the comprehensive value index V=α×U+β×R+γ×C for each data asset, and determining the return on investment and priority ranking of data assets, include: Based on a pre-established classification rule library, the data is grouped according to usage frequency, business relevance, and processing cost to obtain a preliminary classification set. Then, attribute tags are added to each data asset using a labeling tool to obtain an asset set with attribute tags. Using a scoring calculation tool, a weighted calculation is performed on an asset set with attribute tags based on preset coefficients for frequency weight, association weight, and cost weight to obtain the comprehensive value index of each data asset and determine the sorted value sequence. If the comprehensive value index of some data assets in the sorted value sequence is lower than the preset threshold, the data assets are filtered through a screening tool to obtain a subset of assets that meet the investment return conditions and determine the scope of assets to focus on. By using a catalog generation tool, the scope of assets of priority is structurally divided according to the association rules of business relevance and processing cost, and a final classification catalog is obtained to determine the list of assets that meet the value quantification target.

4. The enterprise data link governance and value management method as described in claim 1, characterized in that, The steps of building a unified data service interface through an API gateway and service bus, performing authentication and access control based on the data request type and permission level of business applications, and retrieving the corresponding data from the data asset repository and processing it in real time if the request passes the security verification to obtain a standardized data service response that meets business needs include: The API gateway receives data requests, parses the initiator's identity information based on the data request type and permission level of the business application, and matches it using a pre-established verification rule base. If the match is successful, an authentication token is generated to determine the legitimacy of the request. The system receives the authentication token, verifies the permission level of the authentication token through the service bus, and if it meets the preset rules, extracts the original data set from the data asset library to obtain the data content that can be processed. For the original dataset, real-time processing tools are used for format conversion and cleaning, and structured data output is generated according to business needs to obtain intermediate processing results; Based on the structured data output, it is distributed to the target application via the service bus. The structured data is encapsulated using a standardized template. If it conforms to the output specifications, the final response content is generated, and the delivery result is determined.

5. An enterprise data link governance and value management system, used to implement the enterprise data link governance and value management method as described in any one of claims 1 to 4, characterized in that, The enterprise data link governance and value management system includes: The first acquisition module is used to capture multi-source heterogeneous data from various business systems within the enterprise in real time using a distributed data acquisition engine. It identifies different formats of raw data types through a preset data source adapter. If structured data is detected, it is extracted using a relational database connector. If it is identified as unstructured data, it is used to extract the content using a document parsing module to obtain an initial data set containing metadata tags. The second acquisition module is used to perform format conversion and field mapping on the initial data set according to the pre-established data standardization rule library, remove duplicate records and outliers through data cleaning algorithms, automatically evaluate the data quality level using a Naive Bayes classifier, and trigger the data repair process if the data quality score is lower than a preset threshold to obtain standardized data that conforms to a unified standard. The determination module is used to automatically classify the standardized data according to business attributes and usage frequency through hierarchical clustering algorithm, establish a multi-dimensional data asset catalog system, generate unique data asset identifiers using hash algorithm, apply data desensitization technology for security processing if the data contains sensitive information, and determine the security level and access permissions of various data assets. The judgment module is used to build a value quantification assessment model based on data usage frequency, business relevance and processing cost. It uses a weighted scoring algorithm to calculate the comprehensive value index V=α×U+β×R+γ×C for each data asset, and judges the return on investment and priority ranking of data assets. U represents the usage frequency weight, R represents the business relevance weight, C represents the processing cost weight, and α, β and γ are the corresponding weight coefficients. The third acquisition module is used to build a unified data service interface through the API gateway and service bus. It performs authentication and access control based on the data request type and permission level of the business application. If the request passes the security verification, it extracts the corresponding data from the data asset library and processes it in real time to obtain a standardized data service response that meets the business needs.

6. The enterprise data link governance and value management system as described in claim 5, characterized in that, The first acquisition module includes: The first acquisition unit is used to acquire multi-source heterogeneous data from various business systems through distributed nodes, and to perform real-time capture using a preset acquisition protocol to obtain a first data set containing the original data types. The second acquisition unit is used to identify the data format of the first data set using a data source adapter. If it is structured data, it extracts fields through a relational database connector. If it is unstructured data, it extracts content through a document parsing module to obtain a second data set containing metadata tags. The first determining unit is used to determine the field mapping relationship based on the second data set by associating metadata tags with business system attributes and using preset mapping rules. The generation unit is used to import the second data set into a relational database through the field mapping relationship and generate an initial data set containing complete business attributes using a batch insertion method.

7. The enterprise data link governance and value management system as described in claim 5, characterized in that, The second acquisition module includes: The third acquisition unit is used to perform format conversion and field mapping operations on the initial data set according to the pre-established data standardization rule base, to obtain a first dataset that conforms to the preset format from the initial data set, and to compare the first dataset one by one with a batch processing tool to obtain a preliminary result with uniform format and accurate field correspondence. The fourth acquisition unit is used to perform duplicate recording and outlier removal on the first dataset using a data cleaning tool. If duplicate field values ​​are detected in the first dataset, the earliest record is retained and subsequent duplicates are deleted. If the value exceeds a preset range, it is marked as an anomaly and removed, thereby acquiring the cleaned second dataset. The second determining unit is used to evaluate the quality level of the second dataset using a classification tool, calculate a quality score for the second dataset, and if the quality score is lower than a preset threshold, mark it as data to be repaired and determine the third dataset to enter the subsequent process. The fifth acquisition unit is used to trigger an automated repair process based on the data to be repaired in the third dataset, and to complete and correct the third dataset through a preset rule base to obtain standardized data that conforms to a unified standard.

8. The enterprise data link governance and value management system as described in claim 5, characterized in that, The determining module includes: The sixth acquisition unit is used to group the standardized data according to the pre-established business classification rule library, obtain the initial grouping results based on business classification, and sort them in combination with usage frequency data to obtain a classification set with priority labels. The third determining unit is used to use a directory building tool to hierarchically divide the classification set with priority tags according to the association rules of the asset directory, obtain a structured directory framework, and determine the identified directory set. The judgment unit is used to mask the sensitive information using a data desensitization tool if the identifiable directory set contains sensitive information, obtain the desensitized security set, classify the security level according to the security level assessment rules, and determine the security ownership of each data asset. The fourth determining unit is used to control and configure the access permission rules according to the level classification results in the security set through the permission management tool, obtain the final set after permission configuration from the security set, and determine the permission management result that meets the business objectives.

Images