Enterprise data link treatment and value management method and system

Abstract

The invention discloses an enterprise data link management and value management method and system, and the method comprises the steps: carrying out the real-time capturing of multi-source heterogeneous data of all business systems in an enterprise through a distributed data collection engine, and recognizing the types of original data in different formats through a preset data source adapter; if structured data is detected, a relational database connector is adopted for extraction, and if the structured data is recognized as unstructured data, a document analysis module is started for content extraction, and an initial data set containing metadata tags is obtained; performing format conversion and field mapping on the initial data set according to a pre-established data standardization rule base, eliminating duplicate records and abnormal values through a data cleaning algorithm, performing automatic evaluation on a data quality grade by adopting a naive Bayes classifier, if a data quality score is lower than a preset threshold value, triggering a data recovery process, and if the data quality score is lower than the preset threshold value, performing data recovery. And standardized data meeting a unified standard is obtained. The normativity and value utilization efficiency of data management are effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly discloses an enterprise data link governance and value management method and system. BACKGROUND

[0002] With the deepening of digital transformation, enterprise data has become a core production factor and strategic resource, and data link governance and value management have become a key capability that determines the competitive advantage of enterprises.

[0003] In a data-driven business environment, how to effectively manage the whole link process from data collection to application directly affects the decision-making quality and operational efficiency of enterprises. The current data governance solutions in the market have obvious shortcomings. Most solutions only focus on the optimization of a single link, and lack unified planning and coordination mechanisms for the whole link.

[0004] Traditional methods are inefficient in handling massive heterogeneous data, making it difficult to achieve effective inventory and classified management of data assets. At the same time, the existing technical framework has limited capabilities in data security protection and value quantification, and cannot meet the urgent needs of enterprises for fine-grained management of data governance. The core challenge faced by enterprise data governance lies in the unified collection and standardized processing of multi-source heterogeneous data.

[0005] The data formats, structures, and quality standards generated by different business systems are different, and traditional data collection techniques cannot achieve efficient cross-system data integration, resulting in serious data island phenomenon. This scattered state of data further exacerbates the complexity of data quality control, with a large amount of dirty data, duplicate data, and inconsistent data, seriously affecting the subsequent data processing and analysis effect. The cumulative effect of data quality problems makes it difficult for enterprises to build a reliable data service system and provide accurate and reliable data support for business decision-making. More critically, the lack of effective data value quantification mechanism makes it difficult for enterprises to accurately assess the cost-effectiveness of data governance investment and develop reasonable data asset management strategies.

[0006] Therefore, how to build a unified management system covering the whole link of data collection, governance, and application, and achieve standardized processing and value quantification evaluation of multi-source data, has become a key problem that needs to be solved in the field of enterprise data governance. SUMMARY

[0007] The present application provides an enterprise data link governance and value management method and system, which aims to solve at least one of the defects in the prior art.

[0008] One aspect of the present application relates to an enterprise data link governance and value management method, comprising the following steps: The distributed data acquisition engine is used to collect multi-source heterogeneous data of various business systems in the enterprise in real time, different formats of raw data types are identified through the preset data source adapter, if structured data is detected, a relational database connector is used for extraction, if unstructured data is identified, a document parsing module is enabled for content extraction, and an initial data set containing metadata tags is obtained; According to the pre-established data standardization rule library, the initial data set is converted and field mapped, the repeated records and abnormal values are removed through the data cleaning algorithm, the data quality level is automatically evaluated by using the naive Bayes classifier, if the data quality score is lower than the preset threshold, the data repair process is triggered, and the standardized data meeting the unified standard is obtained; The hierarchical clustering algorithm is used to automatically classify the standardized data according to business attributes and usage frequency, a multi-dimensional data asset directory system is established, a unique data asset identifier is generated by using the hash algorithm, if the data contains sensitive information, the data desensitization technology is applied for security processing, the security level and access permission of each type of data asset are determined; According to the data usage frequency, business correlation degree and processing cost, a value quantification evaluation model is constructed, a weighted scoring algorithm is used to calculate the comprehensive value index V=α×U+β×R+γ×C of each data asset, and the investment return rate and priority ranking of the data asset are judged, wherein U represents the usage frequency weight, R represents the business correlation degree weight, C represents the processing cost weight, and α, β and γ are the corresponding weight coefficients; A unified data service interface is constructed through the API gateway and the service bus, identity authentication and access control are performed according to the data request type and permission level of the business application, if the request passes the security check, the corresponding data is extracted from the data asset library and real-time processing is performed, and the standardized data service response meeting the business demand is obtained.

[0009] Further, the distributed data acquisition engine is used to collect multi-source heterogeneous data of various business systems in the enterprise in real time, different formats of raw data types are identified through the preset data source adapter, if structured data is detected, a relational database connector is used for extraction, if unstructured data is identified, a document parsing module is enabled for content extraction, and an initial data set containing metadata tags is obtained. Multi-source heterogeneous data is obtained from various business systems through distributed nodes, real-time collection is performed by using a preset collection protocol, and a first data set containing raw data types is obtained; For the first data set, a data source adapter is used to identify the data format, if it is structured data, a relational database connector is used to extract the field, if it is unstructured data, a document parsing module is used to extract the content, and a second data set containing metadata tags is obtained; According to the second data set, the metadata tags are associated with the business system attributes, a preset mapping rule is used to determine the field mapping relationship; The second data set is imported into the relational database through the field mapping relationship, and an initial data set containing complete business attributes is generated by using a batch insertion method.

[0010] Further, the initial data set is subjected to format conversion and field mapping according to a pre-established data standardization rule library, repeated records and abnormal values are removed through a data cleaning algorithm, the data quality level is automatically evaluated using a naive Bayes classifier, and if the data quality score is lower than a preset threshold, a data repair process is triggered to obtain standardized data conforming to the unified standard. The steps include: According to the pre-established data standardization rule library, format conversion and field mapping operations are performed on the initial data set, and a first data set conforming to the preset format is obtained from the initial data set. A batch processing tool is used to compare the first data set piece by piece to obtain a preliminary result with uniform format and accurate field correspondence; The first data set is subjected to repeated record and abnormal value removal through a data cleaning tool. If repeated field values are detected in the first data set, the earliest record is retained and subsequent repeated items are deleted. Meanwhile, if the value exceeds the preset range, it is marked as abnormal and removed. A second data set after cleaning is obtained; A classification tool is used to evaluate the quality level of the second data set. The quality score of the second data set is calculated. If the quality score is lower than the preset threshold, the data is marked as to-be-repaired data, and a third data set entering the subsequent process is determined; According to the to-be-repaired data in the third data set, an automatic repair process is triggered, the third data set is completed and corrected through a preset rule library, and standardized data conforming to the unified standard is obtained.

[0011] Further, the standardized data is automatically classified according to business attributes and usage frequency through a hierarchical clustering algorithm, a multi-dimensional data asset directory system is established, a unique data asset identifier is generated using a hash algorithm, and if the data contains sensitive information, a data desensitization technology is applied for security processing. The steps of determining the security level and access permission of each type of data asset include: According to the pre-established business classification rule library, the standardized data is subjected to grouping processing, and an initial grouping result based on business classification is obtained therefrom. Combined with the usage frequency data, the classification set with priority markers is obtained; A directory construction tool is used to divide the classification set with priority markers into hierarchical levels according to the association rules of the asset directory, and a structured directory framework is obtained therefrom to determine the identified directory set; If the identified catalog set contains sensitive information, the sensitive information is content-screened by a data desensitization tool, a desensitized security set is obtained, and the security level evaluation rule is used for level division to determine the security attribution of each data asset; Through the permission management tool, the level division result in the security set is controlled and configured according to the access permission rule, the final set after permission configuration is obtained from the security set, and the permission management result meeting the business target is determined.

[0012] Further, a value quantification evaluation model is constructed according to the data usage frequency, business correlation degree and processing cost, and a weighted scoring algorithm is used to calculate the comprehensive value index V = α × U + β × R + γ × C of each data asset, and the steps of judging the investment return rate and priority ranking of the data asset include: According to the pre-established classification rule library, the data usage frequency, business correlation degree and processing cost are grouped and processed to obtain a preliminary classification set, and attribute labels are added to each data asset by using a labeling tool to obtain an asset set with attribute labels; A scoring calculation tool is used to calculate the comprehensive value index of each data asset by weighting the preset coefficients of frequency weight, correlation weight and cost weight according to the asset set with attribute labels, and a value sequence after sorting is determined; If the comprehensive value index of part of the data assets in the value sequence after sorting is lower than the preset threshold, the data assets are filtered by a screening tool to obtain an asset subset meeting the investment return condition, and the asset range to be paid attention to is determined; Through a catalog generation tool, the asset range to be paid attention to is structurally divided according to the association rule of business correlation degree and processing cost to obtain a final classification catalog, and an asset list meeting the value quantification target is determined.

[0013] Further, a unified data service interface is constructed through an API gateway and a service bus, identity verification and access control are performed according to the data request type and permission level of the business application, if the request passes the security check, the corresponding data is extracted from the data asset library and processed in real time to obtain a standardized data service response meeting the business demand, and the steps include: The API gateway receives a data request, parses the identity information of the initiator according to the data request type and permission level of the business application, matches the pre-established verification rule library, generates an identity verification token if the matching passes, and determines the legality of the request; The identity verification token is received, the permission level of the identity verification token is verified through the service bus, and if the preset rule is met, the original data set is extracted from the data asset library to obtain processable data content; For the original data set, a real-time processing tool is used for format conversion and cleaning, structured data output is generated according to business requirements, and intermediate processing results are obtained; According to the structured data output, the service bus is distributed to the target application, the structured data is packaged by using a standardized template, and if the output specification is met, the final response content is generated, and the delivery result is determined.

[0014] Another aspect of the application relates to an enterprise data link governance and value management system for implementing the enterprise data link governance and value management method described above, the enterprise data link governance and value management system comprising: The first acquisition module is configured to use a distributed data collection engine to perform real-time grabbing on multi-source heterogeneous data of various business systems within an enterprise, identify different formats of original data types through a preset data source adapter, extract structured data using a relational database connector, and extract content using a document parsing module if the data is identified as unstructured data, to obtain an initial data set containing metadata tags; The second acquisition module is configured to perform format conversion and field mapping on the initial data set according to a pre-established data standardization rule library, remove duplicate records and outliers through a data cleaning algorithm, automatically evaluate the data quality level using a Naive Bayes classifier, trigger a data repair process if the data quality score is below a preset threshold, and obtain standardized data that meets the unified standard; The determination module is configured to automatically classify the standardized data according to business attributes and usage frequency using a hierarchical clustering algorithm, establish a multi-dimensional data asset directory system, generate a unique data asset identifier using a hash algorithm, and apply data desensitization techniques for security processing if the data contains sensitive information, to determine the security level and access permissions of various data assets; The judgment module is configured to construct a value quantification evaluation model according to data usage frequency, business correlation degree and processing cost, calculate the comprehensive value index V = alpha * U + beta * R + gamma * C of each data asset using a weighted scoring algorithm, and determine the investment return rate and priority ranking of the data asset, wherein U represents the usage frequency weight, R represents the business correlation degree weight, C represents the processing cost weight, and alpha, beta and gamma are the corresponding weight coefficients; The third acquisition module is configured to build a unified data service interface through an API gateway and a service bus, perform identity verification and access control according to the data request type and permission level of the business application, extract the corresponding data from the data asset library and perform real-time processing if the request passes the security check, and obtain a standardized data service response that meets the business requirements.

[0015] Further, the first acquisition module comprises: The first acquisition unit is configured to acquire multi-source heterogeneous data from each service system through a distributed node, perform real-time grabbing by using a preset acquisition protocol, and obtain a first data set containing original data types; The second acquisition unit is configured to identify a data format by using a data source adapter for the first data set, extract fields by using a relational database connector if the data is structured data, extract content by using a document parsing module if the data is unstructured data, and obtain a second data set containing metadata tags; The first determination unit is configured to associate the second data set with service system attributes by using metadata tags, and determine a field mapping relationship by using a preset mapping rule. The generation unit is configured to import the second data set into a relational database by using the field mapping relationship, and generate an initial data set containing complete service attributes by using a batch processing insertion method.

[0016] Further, the second acquisition module includes: The third acquisition unit is configured to perform format conversion and field mapping operations on the initial data set according to a pre-established data standardization rule library, acquire a first data set conforming to a preset format from the initial data set, and compare the first data set by using a batch processing tool to obtain a preliminary result with uniform format and accurate field correspondence. The fourth acquisition unit is configured to perform duplicate record and abnormal value elimination on the first data set by using a data cleaning tool, retain the earliest record and delete subsequent duplicate items if duplicate field values exist in the first data set, and mark and eliminate abnormal values if the values exceed a preset range to obtain a second data set after cleaning. The second determination unit is configured to evaluate the quality level of the second data set by using a classification tool, calculate a quality score for the second data set, mark the data as to-be-repaired data if the quality score is lower than a preset threshold, and determine a third data set entering a subsequent process. The fifth acquisition unit is configured to trigger an automatic repair process according to the to-be-repaired data in the third data set, complete and correct the third data set by using a preset rule library, and obtain standardized data conforming to a unified standard.

[0017] Further, the determination module includes: The sixth acquisition unit is configured to perform grouping processing on the standardized data according to a pre-established service classification rule library, acquire an initial grouping result based on service classification from the standardized data, and sort the initial grouping result in combination with frequency data to obtain a classification set with priority labels. The third determination unit is configured to acquire a structured directory framework from the classification set with priority labels by using a directory construction tool and performing hierarchical division according to association rules of an asset directory, and determine a directory set with identification. A judgment unit is used to, if the identified directory set contains sensitive information, use a data desensitization tool to mask the sensitive information, obtain a desensitized security set, and classify it according to security level assessment rules to determine the security ownership of each data asset; The fourth determination unit is used to use the permission management tool to control the configuration of the level division results in the security set according to the access permission rules, obtain the final set after permission configuration from the security set, and determine the permission management results that meet the business objectives.

[0018] The beneficial effects achieved by the present invention are: The present invention provides an enterprise data link governance and value management method and system, offering an integrated solution to address the challenges of unified management and value assessment of multi-source, heterogeneous data within an enterprise. First, the method uses a distributed data acquisition engine to capture data from various business systems in real time. Data source adapters identify data types, extract structured data using a relational database connector, and process unstructured data using a document parsing module, forming an initial data set containing metadata. Next, the initial data undergoes format conversion and field mapping based on a data standardization rule library. A data cleansing algorithm removes duplicate and abnormal records, and a naive Bayesian classifier automatically assesses data quality. Substandard data triggers a remediation process to ensure data normalization. A hierarchical clustering algorithm then categorizes the normalized data by business attributes and usage frequency, creating a multidimensional data asset catalog. A hashing algorithm is then applied to generate unique identifiers, desensitize sensitive data, and determine security levels and access permissions. Finally, a value quantification model is constructed, using a weighted scoring algorithm to calculate the comprehensive value index of data assets, determine return on investment and priority, and provide a unified data service interface through an API gateway and service bus to implement authentication and access control, ensuring data security and efficient meeting business needs. This invention effectively improves the standardization of data management and the efficiency of value utilization, and provides a strong guarantee for the efficient management and decision-making support of enterprise data assets. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of an embodiment of the enterprise data link governance and value management method of the present invention. DETAILED DESCRIPTION

[0020] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0021] like Figure 1 As shown, the first embodiment of the present invention proposes an enterprise data link governance and value management method, including the following steps: Step S100, using a distributed data collection engine to collect multi-source heterogeneous data of each business system in the enterprise in real time, identifying different formats of raw data types through a preset data source adapter, if structured data is detected, a relational database connector is used for extraction, if it is identified as unstructured data, a document parsing module is enabled for content extraction, and an initial data set containing metadata tags is obtained.

[0022] The distributed data collection engine is a software tool or system based on a distributed computing architecture, used to collect data from multi-source heterogeneous data sources (such as databases, file systems, API interfaces, Internet of Things devices, etc.) in real time or batch, and transmit the collected data to a data storage or processing platform. The distributed data collection engine realizes efficient collection, transmission and management of large-scale data through distributed node cooperation, solving the bottleneck of traditional centralized collection mode in scalability, reliability and performance.

[0023] Multi-source heterogeneous data refers to a collection of data from multiple different data sources with differences in data structure, format, protocol and semantics. Its core features are "multi-source" and "heterogeneity", and it is widely used in enterprise information systems, Internet applications, Internet of Things and other scenarios, and is a typical embodiment of data diversity in the digital era.

[0024] The data source adapter is a key component of the data collection layer. The data source adapter parses the protocols, formats and structures of different data sources, identifies the types of raw data and completes standardized conversion, ensuring that heterogeneous data can be uniformly processed by subsequent systems. This process is similar to a "translator" that matches data source characteristics through preset rules or algorithms to achieve automatic identification and adaptation of structured, unstructured and semi-structured data.

[0025] Structured data refers to data with a clear data model (such as data type, format, length, constraints, etc.) that is logically represented and stored in a two-dimensional table format. The characteristics of structured data are fixed format, clear structure, and easy storage, management and query using a relational database or structured query language (SQL).

[0026] Unstructured data refers to data without a predefined data model or uniform organizational structure, with flexible formats and diverse content, which cannot be directly stored and managed using a two-dimensional table or row / column structure of a relational database. The structure of this type of data is implicit in the content and requires techniques such as text analysis, image recognition and natural language processing (NLP) to extract valuable information.

[0027] A Relational Database Connector is a tool or component used to enable data interaction and integration between different systems and relational databases such as MySQL, Oracle, SQL Server, etc. Its core functions include establishing communication links, parsing database protocols, and encapsulating operation interfaces, allowing external applications, data platforms, or analysis tools to securely and efficiently access and manage structured data in relational databases.

[0028] A Document Parsing Module is a software component or system that uses text processing, pattern recognition, and machine learning techniques to extract content, convert formats, and analyze semantics from various types of unstructured or semi-structured documents. The Document Parsing Module converts raw data in documents into structured information that computers can understand and process, facilitating subsequent data storage, analysis, retrieval, or automated process invocation.

[0029] A Metadata Tag is a structured keyword or phrase used to describe data or files, designed to quickly convey the core characteristics, attributes, context, or classification information of data through concise identification. Metadata tags are lightweight representations of metadata ("data about data") and typically exist as single words, phrases, or standardized terms, assisting in data retrieval, classification, management, and understanding.

[0030] Step S200: Format conversion and field mapping of the initial data set according to the pre-established data standardization rule library, elimination of duplicate records and outliers through data cleaning algorithms, automatic evaluation of data quality level using a Naive Bayes classifier, triggering of data repair procedures if the data quality score is below the preset threshold, and obtaining standardized data conforming to unified standards.

[0031] A data standardization rule library is a structured rule set that defines the unified standards to be followed during data storage, processing, and exchange, aiming to eliminate format differences, semantic ambiguities, and logical conflicts in multi-source data, ensuring data consistency, accuracy, and usability. The data standardization rule library is a core component of the data governance system, providing execution basis and quality benchmarks for data cleaning, conversion, integration, and other operations.

[0032] Format conversion is the process of converting the storage format, data type, and expression form of initial data into the standard format specified by the rule library. It is used to solve data format compatibility issues between different systems, such as date formats, numerical precision, and character encoding.

[0033] The initial data field and the target field are mapped in the field mapping to ensure that the data is accurately matched in structure and semantics. This is used to solve the problem of inconsistent field names, meanings, and business logic in multi-source data (for example, "User ID" may be named "User_ID", "Customer Number", etc.).

[0034] Removing duplicate records and outliers through data cleaning algorithms refers to the process of using specific algorithm logic to identify and remove duplicate and redundant records and abnormal data points that do not conform to business logic or statistical rules in the original data, in order to improve data quality. This is one of the core links of data cleaning, aiming to solve consistency problems (such as duplicate records) and accuracy problems (such as outliers), and to provide a reliable foundation for subsequent data analysis and modeling.

[0035] Using the Naive Bayes algorithm (a probability classification model based on Bayes' theorem and the assumption of conditional independence of features), the data quality evaluation indicators (such as completeness, accuracy, consistency, timeliness, etc.) are modeled, and the probability relationship between each indicator and the data quality level (such as excellent, good, medium, and poor) is learned through training samples, so as to realize the automatic classification and judgment of new data quality level.

[0036] If the data quality score is lower than the preset threshold, the data repair process is triggered, and the standardized data that meets the unified standard is the closed-loop management mechanism in data quality control. By quantitatively evaluating data quality and comparing it with the preset standard, the data that does not meet the standard is automatically triggered for repair, ensuring that the final data meets the business rules and technical specifications. The core logic is: using data quality score as the basis for judgment, using automated process as the repair means, and realizing the closed-loop improvement of data from 'unqualified' to 'qualified'.

[0037] Step S300, automatically classify the standardized data according to business attributes and usage frequency through hierarchical clustering algorithm, establish a multi-dimensional data asset directory system, generate a unique data asset identifier using hash algorithm, and if the data contains sensitive information, apply data desensitization technology for security processing, determine the security level and access permission of each type of data asset.

[0038] The hierarchical clustering algorithm is used to automatically classify the standardized data according to business attributes and usage frequency in the data governance data asset classification link. The hierarchical clustering algorithm (Hierarchical Clustering) is used to unsupervised grouping of the data that has completed the standardization processing, based on the business attribute characteristics (such as theme, source, sensitivity) and usage frequency indicators (such as access frequency, update frequency) of the data, automatically discover the similarity between the data and construct a hierarchical classification system, and finally form a data asset directory that conforms to the business logic.

[0039] Multi-dimensional data asset catalog system is a framework system that organizes, classifies, describes and manages data from multiple business perspectives and technical dimensions with data assets as the core. Through systematic hierarchical classification, multi-dimensional data asset catalog system realizes the visualization, retrievability and manageability of data assets, and improves the efficiency and value of data assets.

[0040] Hash algorithm generates a unique data asset identifier, which refers to using the one-way mapping characteristics and collision resistance of hash function to convert the feature information of data assets (such as content, metadata, etc.) into a fixed-length string (hash value) as the unique identifier (UID, Unique Identifier) of the data asset in the system. This process ensures that each data asset corresponds to a unique identifier through algorithm, and the identifier can quickly verify the consistency and integrity of the data.

[0041] If the data contains sensitive information, applying data desensitization technology for security processing is a key link in data security governance, which refers to transforming data containing sensitive information through data desensitization technology, so that it cannot be directly linked to the real sensitive content while maintaining business logic and data structure, thereby reducing the risk of data leakage. The core logic is: first identify the sensitive information in the data, then select the appropriate desensitization strategy according to the sensitivity level and use scenario, and achieve the security goal of 'data available but invisible'.

[0042] Determining the security level and access permission of various data assets is a core link in the data security management system, which refers to dividing clear security levels (such as public, internal, sensitive, confidential, etc.) for different types of data assets by evaluating the sensitivity, business value, compliance risk, etc. of the data, and formulating differentiated access control strategies based on the level to ensure that the data is used reasonably under the premise of legality and compliance.

[0043] Step S400, according to the data usage frequency, business correlation degree and processing cost, a value quantification evaluation model is constructed, and a weighted scoring algorithm is used to calculate the comprehensive value index V = α × U + β × R + γ × C of each data asset, to judge the investment return rate and priority ranking of the data asset, wherein U represents the usage frequency weight, R represents the business correlation degree weight, C represents the processing cost weight, and α, β, γ are the corresponding weight coefficients.

[0044] The value quantification evaluation model is a tool for quantitatively evaluating the business value of data assets through a multi-dimensional index system and mathematical algorithms. The model takes data usage frequency, business relevance, and processing cost as the core evaluation dimensions, and through standardized index calculation and weight distribution, it converts data value into quantifiable results, providing a scientific basis for data asset prioritization, resource allocation, and decision-making.

[0045] The weighted scoring algorithm is a quantitative method for calculating the comprehensive value of data assets by assigning different weights to multi-dimensional evaluation indicators. The weighted scoring algorithm converts various key attributes of data assets (such as usage frequency, business relevance, processing cost, etc.) into standardized scores, combines pre-set weights for weighted summation, and finally obtains a Composite Value Index (CVI) for intuitively measuring the relative value of data assets, supporting data management decisions.

[0046] The data asset return on investment is a quantitative indicator that measures the input and output efficiency of data assets. By comparing the total cost and business benefits of data assets in the aspects of collection, storage, governance, analysis, and application, it evaluates the economic value and investment effectiveness.

[0047] Data asset prioritization is the process of evaluating and ordering the importance of data assets based on business objectives, value contribution, risk level, and other multi-dimensional factors. Its core is to establish an evaluation model to assign priority labels (such as "strategic level", "priority level", "ordinary level") to data assets, guiding the rational allocation of resources (human resources, budget, technology).

[0048] Step S500, through the API gateway and service bus, a unified data service interface is built, identity verification and access control are performed according to the data request type and permission level of business applications, if the request passes the security check, the corresponding data is extracted from the data asset library and real-time processing is performed, and the standardized data service response meeting the business demand is obtained.

[0049] API Gateway (API Gateway) is the unified entrance of the system, responsible for receiving all client requests, forwarding to the backend service according to the routing rules, and providing authentication, authorization, rate limiting, monitoring, etc. API Gateway is like a "data service guard", shielding the complexity of the backend service and exposing a unified access address to the outside.

[0050] Service Bus is the communication hub that connects different services in a distributed system, supporting asynchronous messaging, event-driven, and decoupling between services. Service Bus is like a "pipeline for data flow", allowing different services to exchange data securely and reliably through the Publish-Subscribe or Point-to-Point mode.

[0051] Unified data service interface encapsulates data capabilities scattered in different systems (such as ERP, CRM, big data platform) into standardized interfaces (such as RESTful API) through the coordination of API gateway and service bus, and provides unified, secure, and manageable data access channels to the outside.

[0052] Authentication is the process of confirming the legitimacy of a user or application identity, i.e. answering the question "Who are you?" By verifying credentials (such as username / password, API key, digital certificate, etc.), it ensures that only authorized subjects can access the system.

[0053] Access Control is a mechanism that restricts access to specific resources based on the user or application's permission level and data request type, based on the authentication result, i.e. answering the question "What can you do?" The core goal is to implement the "least privilege principle" to ensure that data is used within the legal scope.

[0054] If the request passes the security check, the corresponding data is extracted from the data asset library and processed in real time, which is a key link in the data service chain. When the user or application's access request passes the security check (identity verification, permission audit, compliance check, etc.), the system automatically retrieves the target data from the data asset library (a unified management platform for structured / unstructured data), and performs cleaning, conversion, analysis, etc. on the data through the real-time processing engine, and finally returns the processing result or triggers the subsequent business process. The core logic is: taking security access as a prerequisite, taking real-time data processing as a means, and realizing on-demand supply and value release of data assets.

[0055] Obtaining a standardized data service response that meets business needs refers to the process of returning data results that meet the needs of business applications or users in a pre-defined unified format, specification and protocol after the data service system completes the security check, identity verification, permission matching, data extraction and processing, etc. The process needs to ensure the accuracy, integrity, timeliness and availability of data, and improve the compatibility and ease of use of data in different business scenarios through standardized output.

[0056] Further, the enterprise data link governance and value management method provided by the embodiment comprises the following steps: Step S110, obtain multi-source heterogeneous data from each business system through distributed nodes, use the preset collection protocol for real-time grabbing, and obtain a first data set containing original data types.

[0057] When obtaining multi-source heterogeneous data from each business system through distributed nodes, a retail enterprise scenario is assumed. The business systems of the retail enterprise include inventory management systems, order processing systems, and customer relationship management systems. The data types of these systems are different. The inventory system may generate structured data such as product ID, quantity, and price; the order system generates semi-structured data such as JSON format order logs; and the customer relationship system contains unstructured data such as customer feedback text records.

[0058] The distributed nodes are deployed on servers in different regions and monitor the data updates of these systems in real time. The preset collection protocol is based on HTTP or MQ message queue protocol to ensure that the data can be quickly and stably transmitted to the center node to generate the first data set. The first data set contains original data types such as CSV files, JSON logs, and text documents. This way can ensure the real-time and integrity of data collection, laying a foundation for subsequent processing.

[0059] Step S120, for the first data set, use the data source adapter to identify the data format, if it is structured data, then extract the fields through the relational database connector, if it is unstructured data, then extract the content through the document parsing module, to obtain a second data set containing metadata tags.

[0060] For the first data set, the role of the data source adapter is particularly important. For example, the data source adapter can identify CSV files as structured data, JSON logs as semi-structured data, and text documents as unstructured data. For structured data, the relational database connector will extract fields, such as extracting “product ID: 1001”, “quantity: 50”, “price: 99.5” and other fields from the CSV file of the inventory system. For unstructured data, the document parsing module can process customer feedback text and extract key content such as “positive evaluation: good product quality” and “negative evaluation: slow delivery speed”, and generate metadata tags such as “data source: customer relationship system”, “data type: text”, and “emotional tendency: positive”. The second data set obtained in this way contains metadata tags, which can clearly identify the source, type, and key attributes of the data, improve the traceability and understandability of the data, and provide convenience for subsequent correlation analysis.

[0061] Step S130, according to the second data set, associate the metadata tags with the business system attributes, and use the preset mapping rules to determine the field mapping relationship.

[0062] In the field mapping according to the second data set, the association of metadata tags with business system attributes is crucial. Assuming that the "product ID" field of the inventory system needs to correspond to the "product number" field of the order system, the default mapping rule can be defined as: if the value formats of the two fields are both numbers and the lengths are equal, they are considered as the same field.

[0063] The mapping rule takes into account the context of the fields, for example, "product ID" comes from the inventory system, while "product number" comes from the order system, both of which are related to the core entity "product" of the retail business. Through this mapping relationship, the data in the second data set is accurately imported into the relational database.

[0064] Step S140, through the field mapping relationship, the second data set is imported into the relational database, and a batch insertion method is used to generate an initial data set containing complete business attributes.

[0065] The batch insertion method further improves efficiency, for example, every 1000 records is inserted into the database as a batch to generate an initial data set containing complete business attributes, such as "order ID: 20001" "product ID: 1001" "quantity: 50" "price: 99.5" "customer feedback: product quality is good". The advantage of this method is that it not only ensures the consistency and integrity of the data, but also significantly improves the efficiency of data import and reduces the system load.

[0066] The deployment of distributed nodes can flexibly adapt to the business needs of retail enterprises. For example, if the enterprise has multiple warehouses nationwide, distributed nodes can be deployed on the servers of each warehouse to collect inventory change data in real time. It should be noted that this distributed architecture can effectively deal with network latency and single point failure problems, ensuring the stability of data collection.

[0067] The default collection protocol supports priority setting, for example, order data is collected and transmitted before customer feedback data to meet the needs of real-time order processing. The benefit of this approach is to improve the robustness and response speed of the system.

[0068] The combination of data source adapters and document parsing modules enables retail enterprises to better handle complex data. For unstructured data of customer feedback, the document parsing module introduces natural language processing technology to extract keywords and sentiment orientation, generating metadata tags. These tags, associated with business system attributes, can help enterprises quickly identify customer needs and problems, for example, discovering that "slow delivery" is a high-frequency problem in customer feedback, thereby optimizing the logistics process.

[0069] The establishment of the field mapping relationship is a key link of data integration. Exemplarily, in a retail scenario, the mapping rule is optimized according to historical data samples, for example, analyzing the order and inventory data of the past year, it is found that the matching rate of "commodity ID" and "product number" reaches more than 95%, thereby confirming the reliability of the mapping rule.

[0070] Preferably, the enterprise data link governance and value management method provided by the embodiment comprises the following steps: In step S210, according to the pre-established data standardization rule library, format conversion and field mapping operations are performed on the initial data set, the first data set conforming to the preset format is obtained from the initial data set, and the batch processing tool is used to compare the first data set piece by piece to obtain the preliminary result of uniform format and accurate field correspondence.

[0071] In the data processing scene of a retail enterprise, the format conversion and field mapping operation on the initial data set are analyzed in detail from multiple angles. First, for the application of the data standardization rule library, it is assumed that the inventory and sales data of a retail enterprise need to be integrated. The inventory data may be stored in table form, with the field name "item number", while the sales data field name is "product code". Through the preset rule library, the two are mapped to a unified "commodity identification", ensuring that the data is consistent in field when integrated. This way can effectively avoid the confusion caused by the difference in field naming.

[0072] For the piece-by-piece comparison of the batch processing tool, the comparison operation is performed on the commodity identification extracted from the inventory and sales data. It is assumed that the initial data set contains 100,000 records, and the batch processing tool compares 5,000 records per batch to check whether the field value conforms to the preset format, such as "commodity identification" should be 6 digits. If a record is found that does not conform to the format, such as a record with 5 digits, it is marked as abnormal and waits for subsequent processing.

[0073] In step S220, a data cleaning tool is used to remove repeated records and abnormal values from the first data set. If it is detected that there are repeated field values in the first data set, the earliest record is retained and the subsequent repeated items are deleted. At the same time, if the value exceeds the preset range, it is marked as abnormal and removed, and a second data set after cleaning is obtained.

[0074] In the application of the data cleaning tool, the repeated records and abnormal values are removed from the specific scene. It is assumed that the same order number "20001" appears twice in the sales data of a retail enterprise, and the cleaning tool retains the earliest record and deletes the subsequent repeated items. At the same time, if a commodity price field value is "9999.5", which is far beyond the normal range "10.0 to 500.0", it is marked as abnormal and removed, ensuring the accuracy of the second data set.

[0075] Step S230, quality level evaluation of the second data set is performed using a classification tool, a quality score is calculated for the second data set, and if the quality score is lower than a preset threshold, the data is marked as to-be-repaired data, and a third data set entering a subsequent process is determined.

[0076] For quality level evaluation of the classification tool, it is assumed that the second data set contains 80,000 records, and the quality score is calculated by calculating the field completeness and consistency. The preset threshold is set to 80 points, and if the score of a batch of data is 75 points, the data is marked as to-be-repaired data. This evaluation method helps to screen out data that needs further processing, and ensures that the third data set entering the subsequent process has higher quality.

[0077] Step S240, according to the to-be-repaired data in the third data set, an automatic repair process is triggered, and the third data set is completed and corrected through a preset rule library to obtain standardized data meeting unified standards.

[0078] In the automatic repair process, the to-be-repaired data in the third data set can be completed and corrected through the rule library. It is assumed that a sales record lacks a "product identifier", and the system automatically completes it as "100001" according to the order number associated with the inventory data. In addition, if the price field is empty, the system can complete it as "99.5" according to the historical average price. This automatic repair can improve the completeness of the data and provide a reliable basis for subsequent business analysis.

[0079] From the overall process, the above-mentioned various links support each other, from format conversion to final standardized data generation, forming a complete data processing chain. The initial data set is processed through the standardized rule library and batch processing tool to ensure field consistency and format uniformity; the cleaning tool and quality evaluation further improve the reliability of the data; the automatic repair solves the remaining problems, and finally generates standardized data meeting the unified standards. This multi-level processing method can significantly improve the efficiency and accuracy of data processing of retail enterprises, and provide more valuable support for business decision-making.

[0080] Further, the enterprise data link governance and value management method provided in the embodiment comprises the following steps: Step S310, according to the pre-established business classification rule library, the standardized data is grouped and processed to obtain an initial grouping result based on business classification, and the frequency data is combined for sorting to obtain a classification set with priority labels.

[0081] In the data management scene of a retail enterprise, grouping processing and subsequent directory construction and a series of operations on the standardized data can be analyzed in detail from multiple angles, especially in the key links of business classification and permission management.

[0082] For the application of business classification rule library, the standardized data of retail enterprises contains multiple categories such as commodity sales, inventory management and customer feedback. Through the pre-established rule library, the data is grouped according to the business type, such as classifying the sales data as "transaction type" and the inventory data as "warehouse type". Combined with the frequency data, such as the daily query frequency of "transaction type" data is 500 times, and the "warehouse type" is 200 times, the system will mark "transaction type" as high priority to form a classification set with priority marking. This grouping method helps to quickly identify core business data.

[0083] Step S320, using a catalog construction tool, the classification set with priority marking is hierarchically divided according to the association rules of the asset catalog, and a structured catalog framework is obtained from it to determine the identified catalog set.

[0084] In the use of the catalog construction tool, the classification set with priority marking is hierarchically divided according to the association rules of the asset catalog. It is assumed that "transaction type" is further divided into "online sales" and "offline sales", and a tree structure catalog framework is formed, in which "online sales" is located in the upper layer due to high priority. This structured catalog framework facilitates quick positioning and management of data assets.

[0085] Step S330, if the identified catalog set contains sensitive information, the data desensitization tool is used to shield the content of the sensitive information, obtain a security set after desensitization, and perform level division according to the security level evaluation rules to determine the security attribution of each data asset.

[0086] For the sensitive information that may exist in the identified catalog set, the application of the data desensitization tool is particularly important. It is assumed that the catalog contains customer feedback data, which contains the customer's contact information field, such as "13800138000", which will be shielded to "138****8000" to form a security set after desensitization. At the same time, according to the security level evaluation rules, the catalog containing customer information is marked as "high security level", and other data such as inventory data is marked as "low security level", so as to clearly determine the security attribution of data assets.

[0087] Step S340, through the permission management tool, the level division result in the security set is controlled and configured according to the access permission rules to obtain a final set after permission configuration from the security set, and determine the permission management result that meets the business target.

[0088] In the application of the authority management tool, the classification results of the security set are configured according to the access authority rules. Assuming that the customer data of the "high security level" is only allowed to be accessed by the market department manager, and the inventory data of the "low security level" is open to all warehouse personnel. Through this control configuration, the final set after the authority configuration is formed, which ensures that the data access meets the business objectives. This hierarchical authority management effectively protects sensitive information while ensuring the availability of business data.

[0089] The overall logic of the above process is closely linked from business classification to authority configuration. Grouping and priority sorting lay the foundation for subsequent directory construction, while desensitization and security level division provide protection for authority management. This multi-level operation not only improves the orderliness of data management, but also balances security and access efficiency, providing strong support for the realization of business objectives of retail enterprises.

[0090] Preferably, the enterprise data link governance and value management method provided by the embodiment comprises the following steps: Step S410, according to the pre-established classification rule library, grouping processing is performed on the data usage frequency, business correlation degree and processing cost, and the preliminary classification set is obtained, and attribute labels are added to each data asset through the labeling tool to obtain the asset set with attribute labels.

[0091] For the data management scene of a retail enterprise, the pre-established classification rule library performs grouping processing based on the usage frequency, business correlation degree and processing cost of the data. For example, the standardized data of a retail enterprise includes product sales data, promotion activity data and logistics distribution data. The classification rule library can set the usage frequency threshold to 300 daily queries, the business correlation degree is based on the relevance to the core sales process, and the processing cost is based on data storage and computing resource consumption. Assuming that the daily query frequency of product sales data is 600 times, the business correlation degree is high, and the processing cost is medium, the query frequency of promotion activity data is 250 times, the correlation degree is medium, and the cost is low, and the query frequency of logistics distribution data is 100 times, the correlation degree is low, and the cost is high. After preliminary grouping, product sales data is classified as a high-frequency high-correlation group, promotion activity data is classified as a medium-frequency medium-correlation group, and logistics distribution data is classified as a low-frequency low-correlation group, forming a preliminary classification set.

[0092] The labeling tool adds attribute labels to the data assets in the preliminary classification set. For example, product sales data is labeled as "core business, high frequency, and computing intensive", promotion activity data is labeled as "auxiliary business, medium frequency, and low cost", and logistics distribution data is labeled as "support business, low frequency, and high cost". These labels not only reflect the business attributes of the data, but also provide a basis for subsequent value assessment.

[0093] The addition of tags needs to be combined with business scenarios to ensure that the tags accurately reflect the characteristics of the data. For example, the "high frequency" tag of commodity sales data is based on its daily average of 600 queries, and "compute-intensive" is derived from real-time analysis requirements.

[0094] Step S420, using a scoring calculation tool, for the asset set with attribute tags, according to the preset coefficients of frequency weight, correlation weight and cost weight, weighted calculation is carried out to obtain the comprehensive value index of each data asset, and the sorted value sequence is determined.

[0095] The scoring calculation tool calculates the comprehensive value index of the data asset according to the preset coefficients of frequency weight 0.5, correlation weight 0.3 and cost weight 0.2.

[0096] For example, the frequency score of commodity sales data is 600 / 1000*0.5=0.3, the correlation degree score is 0.8*0.3=0.24, and the cost score is 0.6*0.2=0.12. The total index is 0.66. The total index of promotion activity data is 0.45, and the total index of logistics distribution data is 0.25.

[0097] Step S430, if the comprehensive value index of part of the data assets in the sorted value sequence is lower than the preset threshold, then the data assets are filtered by a filtering tool to obtain an asset subset that meets the investment return condition, and the asset range of priority attention is determined.

[0098] If the preset threshold is 0.4, then the commodity sales data and the promotion activity data enter the value sequence, and the logistics distribution data is filtered. This value quantification method helps to prioritize high-value data.

[0099] The filtering tool further filters the value sequence to retain an asset subset that meets the investment return condition. Assuming that commodity sales data is prioritized due to high-frequency queries and core business relevance, and promotion activity data also enters the subset due to low cost and support for marketing activities. The logistics distribution data is excluded due to low frequency and high cost. This selection ensures that resources are allocated to high-return data.

[0100] Step S440, using a catalog generation tool, for the asset range of priority attention, according to the association rules of business relevance and processing cost, the structured division is carried out to obtain the final classification catalog, and the asset list that meets the value quantification target is determined.

[0101] The catalog generation tool structures the asset range of priority concern according to the rules of business relevance and processing cost. For example, commodity sales data is subdivided into "online orders" and "store transactions", and promotion activity data is subdivided into "holiday promotions" and "daily discounts", forming a tree-like catalog structure. Online orders are placed at the upper level of the catalog due to the high value index, facilitating quick positioning. This structured catalog clearly displays the hierarchy of data assets, supporting efficient management and query.

[0102] The above process progresses from grouping to catalog generation, ensuring that the classification and value assessment of data assets meet the business needs of retail enterprises. The classification rule library provides the basis for grouping, attribute tags and value indexes provide support for screening, and the structured catalog facilitates data management, overall improving the utilization efficiency of data assets.

[0103] Further, the enterprise data link governance and value management method provided by the embodiment comprises the following steps: Step S510, receiving a data request through the API gateway, analyzing the identity information of the initiator according to the data request type and permission level of the business application, matching with the pre-established verification rule library, generating an identity verification token if the matching is passed, and determining the legality of the request.

[0104] In the data management scenario of a retail enterprise, the process of receiving a data request through an API gateway can be analyzed in detail from the whole process of identity verification to data delivery, combined with business needs and permission management, to ensure the security and efficient use of data assets.

[0105] For the process of receiving a data request through an API gateway and analyzing the identity information of the initiator, it is a rule-based identity verification mechanism. In a retail enterprise, assuming that a business application initiates a request to obtain commodity sales data, the API gateway first analyzes the identity information in the request, such as user ID and request type. The pre-established verification rule library stores the permission levels of different business applications, such as the application of the sales department being assigned a high permission, and the application of the logistics department having a lower permission. If the application of the sales department initiates a request, the rule library generates an identity verification token after matching its identity information, confirming the legality of the request. This way ensures that only initiators who meet the conditions can proceed to the next step of operation.

[0106] Step S520, receiving the identity verification token, checking the permission level of the identity verification token through the service bus, and if it meets the pre-set rules, extracting the original data set from the data asset library to obtain the processable data content.

[0107] In the step of checking the permission level of the authentication token through the service bus, the retail enterprise can set rules that only high-permission applications can access core sales data. Assuming that the application token of the sales department shows that its permission level is A, which meets the preset rules, the service bus allows the extraction of the original data set from the data asset library, such as a data table containing product sales records. If the application token of the logistics department shows that the permission is C, it may be restricted from accessing and can only obtain non-core data. This hierarchical checking mechanism effectively protects sensitive data.

[0108] Step S530, for the original data set, the real-time processing tool is used for format conversion and cleaning, and structured data output is generated according to business requirements to obtain intermediate processing results.

[0109] For the format conversion and cleaning of the original data set, the real-time processing tool plays a key role in the retail scenario. Assuming that the extracted original data contains product sales records, but the format is not uniform, some records are stored in text form, and some contain redundant fields. The processing tool will convert the data to a standard format and remove invalid information, and finally generate structured data output, such as a sales report arranged by date and product category. This processing facilitates subsequent business analysis.

[0110] Step S540, according to the structured data output, distribute to the target application through the service bus, encapsulate the structured data using a standardized template, generate the final response content if it meets the output specification, and determine the delivery result.

[0111] In the stage of distributing to the target application through the service bus and encapsulating the data, assuming that the structured data output is a sales report and the target application is an analysis system of the sales department. After the service bus distributes the data to the system, it encapsulates the data using a standardized template to ensure that the field name and format meet the output specification, and finally generates a response content that can be directly used for analysis. This standardized delivery improves the usability of data in business applications.

[0112] Each of the above steps is closely related to the business needs of the retail enterprise, forming a complete chain from authentication to data delivery, ensuring efficient flow of data assets under permission control, and improving the business applicability of data through structured and standardized processing. This whole-process management is of great significance to the utilization of data assets of retail enterprises.

[0113] The application relates to an enterprise data link governance and value management system for realizing the enterprise data link governance and value management method, and the enterprise data link governance and value management system comprises a first acquisition module, a second acquisition module, a determination module, a judgment module and a third acquisition module, wherein the first acquisition module is used for adopting a distributed data collection engine to realize real-time grabbing of multi-source heterogeneous data of various business systems in an enterprise, identifying different formats of original data types through a preset data source adapter, adopting a relational database connector to extract structured data, enabling a document analysis module to extract content if the data is unstructured, and obtaining an initial data set containing metadata tags; the second acquisition module is used for performing format conversion and field mapping on the initial data set according to a pre-established data standardization rule library, eliminating repeated records and abnormal values through a data cleaning algorithm, automatically evaluating the data quality level by adopting a naive Bayes classifier, triggering a data repair process if the data quality score is lower than a preset threshold, and obtaining standardized data meeting unified standards; the determination module is used for automatically classifying the standardized data according to business attributes and use frequency through a hierarchical clustering algorithm, establishing a multidimensional data asset directory system, generating a unique data asset identifier by adopting a hash algorithm, applying data desensitization technology for security processing if the data contains sensitive information, and determining the security level and access permission of various data assets; the judgment module is used for constructing a value quantization evaluation model according to data use frequency, business correlation degree and processing cost, calculating the comprehensive value index V of each data asset by adopting a weighted scoring algorithm, and judging the investment return rate and priority order of the data asset, wherein U represents the use frequency weight, R represents the business correlation degree weight, C represents the processing cost weight, and alpha, beta and gamma are corresponding weight coefficients; and the third acquisition module is used for constructing a unified data service interface through an API gateway and a service bus, performing identity authentication and access control according to the data request type and permission level of a business application, extracting corresponding data from a data asset library and performing real-time processing if the request passes the security check, and obtaining a standardized data service response meeting business requirements.

[0114] Furthermore, the enterprise data link governance and value management system provided by this embodiment has a first acquisition module including a first acquisition unit, a second acquisition unit, a first determination unit and a generation unit, wherein the first acquisition unit is used to acquire multi-source heterogeneous data from each business system through distributed nodes, and to perform real-time crawling using a preset acquisition protocol to obtain a first data set containing original data types; the second acquisition unit is used to identify the data format of the first data set using a data source adapter, and if it is structured data, extract the fields through a relational database connector; if it is unstructured data, extract the content through a document parsing module to obtain a second data set containing metadata tags; the first determination unit is used to associate the metadata tags with the business system attributes based on the second data set, and to determine the field mapping relationship using preset mapping rules; the generation unit is used to import the second data set into the relational database through the field mapping relationship, and to generate an initial data set containing complete business attributes using a batch insertion method.

[0115] Preferably, in the enterprise data link governance and value management system provided by this embodiment, the second acquisition module includes a third acquisition unit, a fourth acquisition unit, a second determination unit, and a fifth acquisition unit, wherein the third acquisition unit is configured to perform format conversion and field mapping operations on the initial data set according to a pre-established data standardization rule base, obtain a first data set that conforms to the preset format from the initial data set, and use a batch processing tool to compare the first data set item by item to obtain a preliminary result with a unified format and accurate field correspondence; the fourth acquisition unit is configured to use a data cleaning tool to remove duplicate records and outliers from the first data set. If duplicate field values ​​are detected in the first data set, the earliest record is retained and subsequent duplicates are deleted. If the value exceeds a preset range, it is marked as an anomaly and removed, thereby obtaining a cleaned second data set; the second determination unit is configured to use a classification tool to perform a quality level assessment on the second data set, calculate a quality score for the second data set, and if the quality score is lower than a preset threshold, mark it as data to be repaired, and determine a third data set to enter the subsequent process; the fifth acquisition unit is configured to trigger an automated repair process based on the data to be repaired in the third data set, complete and correct the third data set using the preset rule base, and obtain normalized data that conforms to unified standards.

[0116] Further, the enterprise data link governance and value management system provided by the embodiment determines the module to include a sixth acquisition unit, a third determination unit, a judgment unit, and a fourth determination unit. The sixth acquisition unit is configured to group the normalized data according to a pre-established business classification rule library, to obtain an initial grouping result based on business classification, and to sort the initial grouping result in combination with frequency data to obtain a classification set with priority labels. The third determination unit is configured to use a directory construction tool to perform hierarchical division on the classification set with priority labels according to an association rule of an asset directory, to obtain a structured directory framework, and to determine an identified directory set. The judgment unit is configured to, if the identified directory set contains sensitive information, perform content screening on the sensitive information by using a data desensitization tool to obtain a safe set after desensitization, and to perform grade division according to a safety level evaluation rule to determine the safety attribution of each data asset. The fourth determination unit is configured to use a permission management tool to perform control configuration on the grade division result in the safe set according to an access permission rule, to obtain a final set after permission configuration from the safe set, and to determine a permission management result that meets a business target.

[0117] Compared with the prior art, the enterprise data link governance and value management method and system provided by the embodiment has the beneficial effects as follows. I. Data collection and preprocessing: full compatibility and quality assurance 1. Seamless integration of multi-source heterogeneous data Through the distributed collection engine + data source adapter architecture, differential processing of structured data (relational database) and unstructured data (documents, logs, etc.) is realized, the problem of data island in enterprise internal systems is solved, and full data is ensured to be included in the governance range.

[0118] 2. Automatic data quality control Based on the quality evaluation model of the Naive Bayes classifier, data integrity and consistency defects can be automatically identified, and data accuracy can be improved to more than 95% by combining data cleaning algorithms (de-duplication, outlier processing) and repair processes, thereby reducing manual verification costs.

[0119] II. Data assetization: classification management and security protection 1. Dynamic classification and asset directory system Using hierarchical clustering algorithm to classify data according to business attributes (such as marketing, production) and frequency of use (high frequency / low frequency), a multi-dimensional asset directory is constructed to support business personnel to quickly search for target data and shorten the data acquisition cycle.

[0120] 2. Graded protection of sensitive data For data containing sensitive information (such as user privacy, financial data), through data desensitization technology (such as masking, generalization) to achieve "available but invisible", combined with security level and access control, to reduce the risk of data leakage.

[0121] III. Value Quantification and Decision Optimization 1. Quantitative evaluation model of data assets Based on the value formula V=α×U+β×R+γ×C, the data usage frequency (U), business relevance (R), and processing cost (C) are converted into a measurable comprehensive value index, helping enterprises identify high-value data assets (such as high-frequency customer label data) and optimize data investment direction.

[0122] 2. Data-driven business decisions Quantitative evaluation results can be directly used for lifecycle management of data assets (such as archiving low-frequency data to reduce storage costs), or guiding data product development (such as prioritizing high-value data API), promoting data transformation from "cost center" to "value center".

[0123] IV. Data Service: Efficient Sharing and Safe Control 1. Unified data service interface system Through API gateway + service bus to build a standardized data service layer, supporting multiple interface protocols such as RESTful, GraphQL, etc., to realize real-time response and on-demand supply of data (such as front-end application calling user portrait data), and to improve data service efficiency (response time from hours to seconds).

[0124] 2. Agile support for business innovation Standardized data services can quickly connect to new business scenarios (such as AI modeling, BI analysis), avoiding repeated development of data interfaces and shortening business online cycle. For example, risk control departments can directly call "customer credit data API" for real-time risk assessment without waiting for manual extraction by data departments.

[0125] In summary, the enterprise data link governance and value management method and system provided in this embodiment realizes the full-link closed-loop management of data from "collection-governance-assetization-service", helps enterprises build an efficient, safe, and quantifiable data value system, and supports the landing of digital transformation strategy.

[0126] While the preferred embodiments of the application have been described, additional variations and modifications can be made to these embodiments by those skilled in the art once they have the benefit of the present disclosure without departing from the spirit and scope of the application. Accordingly, it is intended that the appended claims include all such variations and modifications as fall within the scope of the present application. It is apparent that those skilled in the art can modify and adapt the present application in various ways without departing from the spirit and scope of the present application. It is therefore intended that the present application encompass all such modifications and variations as fall within the scope of the claims and their equivalents.

Claims

1. A method for enterprise data link governance and value management, characterized in that: The method comprises the following steps: The distributed data collection engine is used to collect multi-source heterogeneous data of various business systems in the enterprise in real time, different formats of original data types are identified through a preset data source adapter, if structured data is detected, a relational database connector is used for extraction, if unstructured data is identified, a document parsing module is enabled for content extraction, and an initial data set containing metadata tags is obtained; According to the pre-established data standardization rule library, the initial data set is converted in format and field mapping, repeated records and abnormal values are removed through data cleaning algorithm, data quality level is automatically evaluated by using naive Bayes classifier, if the data quality score is lower than the preset threshold, the data repair process is triggered, and the standardized data meeting the unified standard is obtained; The hierarchical clustering algorithm is used to automatically classify the standardized data according to business attributes and usage frequency, a multi-dimensional data asset directory system is established, a unique data asset identifier is generated by using the hash algorithm, if the data contains sensitive information, the data desensitization technology is applied for security processing, and the security level and access permission of each type of data asset are determined; According to the data usage frequency, business correlation degree and processing cost, a value quantization evaluation model is constructed, a weighted scoring algorithm is used to calculate the comprehensive value index V=α×U+β×R+γ×C of each data asset, and the investment return rate and priority ranking of the data asset are judged, wherein U represents the usage frequency weight, R represents the business correlation degree weight, C represents the processing cost weight, and α, β and γ are the corresponding weight coefficients; A unified data service interface is constructed through the API gateway and the service bus, identity authentication and access control are performed according to the data request type and permission level of the business application, if the request passes the security check, the corresponding data is extracted from the data asset library and real-time processing is performed, and the standardized data service response meeting the business demand is obtained.

2. The enterprise data link governance and value management method of claim 1, wherein, The step of using the distributed data collection engine to collect multi-source heterogeneous data of various business systems in the enterprise in real time, identifying different formats of original data types through a preset data source adapter, if structured data is detected, using a relational database connector for extraction, if unstructured data is identified, enabling a document parsing module for content extraction, and obtaining an initial data set containing metadata tags comprises: Multi-source heterogeneous data is obtained from various business systems through distributed nodes, real-time collection is performed by using a preset collection protocol, and a first data set containing original data types is obtained; For the first data set, a data source adapter is used to identify data format, if it is structured data, a relational database connector is used to extract fields, if it is unstructured data, a document parsing module is used to extract content, and a second data set containing metadata tags is obtained; According to the second data set, the metadata tags are associated with the business system attributes, and a preset mapping rule is used to determine the field mapping relationship; Through the field mapping relationship, the second data set is imported into a relational database, and a batch insertion method is used to generate an initial data set containing complete business attributes.

3. The enterprise data link governance and value management method of claim 1, wherein, The step of performing format conversion and field mapping on the initial data set according to the pre-established data standardization rule library, removing duplicate records and abnormal values through a data cleaning algorithm, automatically evaluating the data quality level by using a naive Bayes classifier, triggering a data repair process if the data quality score is lower than a preset threshold, and obtaining standardized data conforming to the unified standard comprises: According to the pre-established data standardization rule library, format conversion and field mapping operations are performed on the initial data set, and a first data set conforming to a preset format is obtained from the initial data set. A batch processing tool is used to compare the first data set piece by piece to obtain a preliminary result with uniform format and accurate field correspondence; Through a data cleaning tool, duplicate records and abnormal values are removed from the first data set. If duplicate field values are detected in the first data set, the earliest record is retained and subsequent duplicates are deleted. If the value exceeds the preset range, it is marked as abnormal and removed. A second data set is obtained after cleaning; A classification tool is used to evaluate the quality level of the second data set. The quality score of the second data set is calculated. If the quality score is lower than a preset threshold, the data is marked as to-be-repaired and a third data set is determined to enter the subsequent process; According to the to-be-repaired data in the third data set, an automated repair process is triggered. The third data set is completed and corrected through a preset rule library to obtain standardized data conforming to the unified standard.

4. The enterprise data link governance and value management method of claim 1, wherein, The step of automatically classifying the standardized data according to business attributes and usage frequency by using a hierarchical clustering algorithm, establishing a multi-dimensional data asset directory system, generating a unique data asset identifier using a hash algorithm, and applying data desensitization technology for security processing if the data contains sensitive information to determine the security level and access permissions of each type of data asset comprises: According to the pre-established business classification rule library, the standardized data is grouped and processed to obtain an initial grouping result based on business classification, and is sorted in combination with usage frequency data to obtain a classification set with priority markers; Using a directory construction tool, the classification set with priority markers is hierarchically divided according to the association rules of the asset directory to obtain a structured directory framework and determine an identified directory set; If the identified directory set contains sensitive information, the sensitive information is content-screened by a data desensitization tool to obtain a desensitized security set, and is divided into levels according to security level evaluation rules to determine the security ownership of each data asset; Through a permission management tool, the level division result in the security set is controlled and configured according to access permission rules to obtain a final set from the security set after permission configuration, and to determine the permission management result conforming to the business target.

5. The enterprise data link governance and value management method of claim 1, wherein, The step of constructing a value quantification evaluation model according to data usage frequency, business correlation degree and processing cost, and calculating the comprehensive value index V = α × U + β × R + γ × C of each data asset by using a weighted scoring algorithm to judge the investment return rate and priority ranking of the data asset comprises: According to the pre-established classification rule library, the data is grouped according to the frequency, business correlation degree and processing cost, a preliminary classification set is obtained, and attribute labels are added to each data asset through a labeling tool to obtain an asset set with attribute labels; A scoring calculation tool is used to calculate the weighted value of each data asset according to the preset coefficients of frequency weight, correlation weight and cost weight, and the comprehensive value index of each data asset is obtained to determine the value sequence after sorting; If the comprehensive value index of part of the data assets in the value sequence after sorting is lower than the preset threshold, the data assets are filtered through a filtering tool to obtain an asset subset that meets the investment return condition, and the asset range to be focused on is determined; A catalog generation tool is used to divide the asset range to be focused on according to the association rules of business correlation degree and processing cost to obtain a final classification catalog and determine the asset list that meets the value quantification target.

6. The enterprise data link governance and value management method of claim 1, wherein, The steps of building a unified data service interface through the API gateway and the service bus, performing identity authentication and access control according to the data request type and permission level of the business application, and obtaining a standardized data service response that meets the business demand if the request passes the security check include: The API gateway receives a data request, parses the identity information of the initiator according to the data request type and permission level of the business application, matches it with a pre-established verification rule library, and generates an identity authentication token if the matching is passed to determine the legality of the request; The identity authentication token is received, and the permission level of the identity authentication token is checked through the service bus to extract the original data set from the data asset library and obtain the processable data content if the preset rule is met; The real-time processing tool is used to convert and clean the original data set according to the business demand to generate structured data output and obtain intermediate processing results; According to the structured data output, the service bus is used to distribute to the target application, the structured data is encapsulated by using a standardized template, and the final response content is generated if the output specification is met to determine the delivery result.

7. An enterprise data link governance and value management system for implementing the enterprise data link governance and value management method according to any one of claims 1 to 6, characterized in that, The enterprise data link governance and value management system comprises: A first acquisition module is used to adopt a distributed data acquisition engine to perform real-time grabbing on multi-source heterogeneous data of various business systems in an enterprise, identify different formats of original data types through a preset data source adapter, extract structured data by using a relational database connector if the structured data is detected, and extract content by using a document analysis module if the data is identified as unstructured data, to obtain an initial data set containing metadata labels; A second acquisition module is used to convert and map the fields of the initial data set according to a pre-established data standardization rule library, remove duplicate records and outliers by using a data cleaning algorithm, automatically evaluate the data quality level by using a Naive Bayes classifier, trigger a data repair process if the data quality score is lower than a preset threshold, and obtain standardized data that meets the unified standard. The determining module is configured to automatically classify the normalized data according to business attributes and usage frequency through a hierarchical clustering algorithm, establish a multi-dimensional data asset directory system, generate a unique data asset identifier using a hash algorithm, apply data desensitization technology for security processing if the data contains sensitive information, and determine the security level and access permission of each type of data asset; The judging module is configured to construct a value quantification evaluation model according to data usage frequency, business correlation degree and processing cost, calculate the comprehensive value index V of each data asset using a weighted scoring algorithm, and judge the investment return rate and priority ranking of the data asset, where U represents the usage frequency weight, R represents the business correlation degree weight, C represents the processing cost weight, and α, β and γ are the corresponding weight coefficients; The third obtaining module is configured to build a unified data service interface through an API gateway and a service bus, perform identity verification and access control according to the data request type and permission level of the business application, extract corresponding data from the data asset library and perform real-time processing if the request passes the security check, and obtain a standardized data service response that meets the business requirements.

8. The enterprise data link governance and value management system of claim 7, wherein, The first obtaining module includes: The first obtaining unit is configured to obtain multi-source heterogeneous data from each business system through a distributed node, perform real-time grabbing using a preset collection protocol, and obtain a first data set containing original data types; The second obtaining unit is configured to identify the data format using a data source adapter for the first data set, extract fields through a relational database connector if the data is structured, extract content through a document parsing module if the data is unstructured, and obtain a second data set containing metadata tags; The first determining unit is configured to associate the second data set with business system attributes through metadata tags and determine the field mapping relationship using a preset mapping rule; The generating unit is configured to import the second data set into a relational database through the field mapping relationship and generate an initial data set containing complete business attributes using a batch insertion method.

9. The enterprise data link governance and value management system of claim 7, wherein, The second obtaining module includes: The third obtaining unit is configured to perform format conversion and field mapping operations on the initial data set according to a pre-established data standardization rule library, obtain a first data set that meets the preset format from the initial data set, and compare the first data set one by one using a batch processing tool to obtain a preliminary result with uniform format and accurate field correspondence; The fourth obtaining unit is configured to perform duplicate record and outlier removal on the first data set using a data cleaning tool, retain the earliest record and delete subsequent duplicates if duplicate field values are detected in the first data set, and mark and remove outliers if the values exceed the preset range, and obtain a second data set after cleaning; The second determining unit is configured to evaluate the quality level of the second data set using a classification tool, calculate the quality score for the second data set, mark the data as to-be-repaired if the quality score is below the preset threshold, and determine a third data set that enters the subsequent process; The fifth acquisition unit is configured to trigger an automatic repair process according to the to-be-repaired data in the third data set, complete and correct the third data set by using a preset rule library, and acquire standardized data conforming to a unified standard.

10. The enterprise data link governance and value management system of claim 7, wherein, The determination module comprises: The sixth acquisition unit is configured to perform grouping processing on the standardized data according to a pre-established service classification rule library, acquire an initial grouping result based on service classification, and perform sorting in combination with frequency data to obtain a classification set with priority labels. The third determination unit is configured to use a directory construction tool to perform hierarchical division on the classification set with priority labels according to an association rule of an asset directory, acquire a structured directory framework, and determine an identified directory set. The judgment unit is configured to, if the identified directory set contains sensitive information, perform content shielding on the sensitive information by using a data desensitization tool, acquire a security set after desensitization, perform grade division according to a security level evaluation rule, and determine the security attribution of each data asset. The fourth determination unit is configured to use a permission management tool to perform control configuration on the grade division result in the security set according to an access permission rule, acquire a final set after permission configuration from the security set, and determine a permission management result conforming to a business target.

Images