Data management method, device, equipment and computer program product

Abstract

The invention provides a data management method, device and equipment and a computer program product, and relates to the technical field of computers and Internet. The method comprises the steps of obtaining asset data of a target machine room, wherein the asset data comprises at least one of order data, asset card data and log data; constructing a to-be-supplemented asset ledger table based on the asset data; extracting information from the asset data through a first large language model, and filling the to-be-filled field to generate an initial asset ledger table; determining a table-level blood relationship path and a field-level blood relationship path of the initial asset ledger table; based on the table-level blood relationship path, the field-level blood relationship path and the initial asset account table, table-level metadata and field-level metadata of the asset account table are generated through a second large language model, and the table-level metadata and the field-level metadata are used for generating a target asset account table containing metadata information in combination with the initial asset account table. According to the embodiment of the invention, data management of the target machine room can be completed intelligently, efficiently and accurately.

Description

Technical Field

[0001] This disclosure relates to the field of computer and Internet technology, and in particular to a data governance method, apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this disclosure as set forth in the claims. The description herein is not intended to be a prior art simply because it is included in this section.

[0003] In data center management, data governance faces severe challenges. Data sources in data centers are complex, with inconsistent formats and chaotic structures, resulting in numerous "data silos" that are difficult to integrate. Current governance efforts rely primarily on manual processing, which is inefficient and prone to errors, leading to inconsistent data quality. This severely restricts operational efficiency and causes a continuous drain on human and material resources.

[0004] Therefore, there is an urgent need for an automated and intelligent data governance method that can efficiently and accurately extract information from multi-source heterogeneous data center data to support efficient data center data governance and value mining. Summary of the Invention

[0005] The purpose of this disclosure is to provide a data governance method, apparatus, electronic device, computer-readable storage medium, and computer program product that can efficiently govern and extract value from data center data.

[0006] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0007] This disclosure provides a data governance method, comprising: acquiring asset data of a target data center, wherein the asset data includes at least one of order data, asset card data, and log data; constructing an asset ledger table to be supplemented based on the asset data, the asset ledger table being used to record multiple equipment assets and containing multiple fields to be filled corresponding to each equipment asset; extracting information from the asset data using a first language model to fill the fields to be filled, thereby generating an initial asset ledger table; determining the table-level lineage path and field-level lineage path of the initial asset ledger table; wherein the table-level lineage path describes the dependency relationship and flow path between data tables, and the field-level lineage path describes the data source and transformation relationship at the field level in the table; and generating table-level metadata and field-level metadata of the asset ledger table using a second language model based on the table-level lineage path, the field-level lineage path, and the initial asset ledger table, wherein the table-level metadata and the field-level metadata are used to combine with the initial asset ledger table to generate a target asset ledger table containing metadata information.

[0008] In some embodiments, the method further includes: retrieving information from an external knowledge base to determine the associated knowledge of the asset data; the external knowledge base includes at least one of historical metadata standards, industry terminology, historical order data, and equipment manuals; wherein, extracting information from the asset data using a first language model to populate the fields to be populated in order to generate an initial asset ledger table includes: the first language model extracting information from the asset data and the associated knowledge to populate the fields to be populated.

[0009] In some embodiments, the first large language model extracts information from the asset data and the associated knowledge to populate the fields to be filled, including: constructing a first prompt word based on the field name of the field to be filled and the associated knowledge retrieved from the external knowledge base; inputting the asset data and the first prompt word into the first large language model to extract information from the asset data and perform preliminary filling of the fields to be filled; if there are still missing fields after filling, the first large language model refers to the associated knowledge to complete the missing fields.

[0010] In some embodiments, based on the table-level lineage path, the field-level lineage path, and the initial asset ledger table, table-level metadata and field-level metadata of the asset ledger table are generated using a second language model. This includes: determining the upstream dependency table and downstream dependency table of the asset ledger table based on the table-level lineage path of the asset ledger table; and determining the table-level metadata of the initial asset ledger table using the second language model, combined with the upstream dependency table and the downstream dependency table. The table-level metadata includes at least one of the following: data flow path description, key processing step annotation, and business definition and description.

[0011] In some embodiments, the initial asset ledger table includes a first field; wherein, based on the field-level lineage path of the first field and the initial asset ledger table, the field-level metadata of the first field is generated through a second major language model, including: determining the field-level lineage path of the first field; and generating, through the second major language model and based on the field-level lineage path of the first field, conversion rules, potential type conversion risks, and type compatibility processing code for the first field during data flow, so as to determine the field-level metadata of the first field based on the conversion rules, potential type conversion risks, and type compatibility processing code, wherein the field-level metadata includes at least one of field description, value description, and sensitivity level.

[0012] In some embodiments, retrieving the asset data from an external knowledge base to determine associated knowledge includes: identifying key information to be retrieved in the asset data; performing word segmentation on the key information to be retrieved to obtain keywords to be retrieved; performing vectorization on the key information to be retrieved to obtain a key vector to be retrieved; performing keyword retrieval in the external knowledge base based on the keywords to be retrieved to obtain first associated knowledge; performing vector similarity retrieval in the external knowledge base based on the key vector to be retrieved to obtain second associated knowledge; and fusing the first associated knowledge and the second associated knowledge to determine the associated knowledge of the asset data.

[0013] In some embodiments, fusing the first associated knowledge and the second associated knowledge to determine the associated knowledge of the asset data includes: determining the ranking of the first associated knowledge in the keyword retrieval results as a first ranking; determining the ranking of the second associated knowledge in the vector similarity retrieval results as a second ranking; adding a constant k to the first ranking and taking its reciprocal to obtain the reciprocal ranking score corresponding to the first associated knowledge, where k is an integer greater than 1; adding a constant k to the second ranking and taking its reciprocal to obtain the reciprocal ranking score corresponding to the second associated knowledge; adding the reciprocal ranking scores of the first and second rankings of each associated knowledge to determine the comprehensive score of each associated knowledge; and determining the associated knowledge of the asset data based on the comprehensive score among the first associated knowledge and the second associated knowledge.

[0014] This disclosure provides a data governance device, including: an asset data acquisition module, a supplementary asset ledger generation module, an initial ledger generation module, a lineage path determination module, and a metadata generation module.

[0015] The asset data acquisition module is used to acquire asset data of the target data center, and the asset data includes at least one of order data, asset card data, and log data. The module for generating the supplementary asset ledger can be used to construct a supplementary asset ledger based on the asset data. The asset ledger records multiple equipment assets and includes multiple fields to be filled for each equipment asset. The module for generating the initial ledger can extract information from the asset data using a first language model to fill the fields to be filled, thereby generating the initial asset ledger. The lineage path determination module can be used to determine the table-level and field-level lineage paths of the initial asset ledger. The table-level lineage path describes the dependencies and flow paths between data tables, while the field-level lineage path describes the data source and transformation relationships at the field level within the table. The metadata generation module can be used to generate table-level and field-level metadata of the asset ledger based on the table-level and field-level lineage paths and the initial asset ledger using a second language model. The table-level and field-level metadata are used in conjunction with the initial asset ledger to generate a target asset ledger containing metadata information.

[0016] This disclosure provides an electronic device comprising: a memory and a processor; the memory for storing computer program instructions; and the processor for calling the computer program instructions stored in the memory to implement the data governance method described above.

[0017] This disclosure provides a computer-readable storage medium storing computer program instructions to implement the data governance method as described in any of the preceding embodiments.

[0018] This disclosure provides a computer program product or computer program that includes computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and the processor executes the computer program instructions to implement the aforementioned data governance method.

[0019] The data governance method, apparatus, electronic device, computer-readable storage medium, and computer program product provided in this disclosure improve the quality and efficiency of data center data governance through a two-stage intelligent processing approach. In the first stage, leveraging the information extraction and completion capabilities of a first-level language model, it directly processes raw data from multiple heterogeneous sources such as orders, asset cards, and logs, automatically identifying and filling key fields in the asset ledger table, thus solving the problems of low efficiency and error-proneness in manual processing. In the second stage, it innovatively introduces data lineage analysis as context, inputting the table-level lineage (data flow) and field-level lineage (data transformation relationships) of the ledger table into a second-level language model, driving it to generate deep metadata (such as business definitions, field association logic, and management attributes) that goes beyond basic descriptions. Ultimately, it outputs a target asset ledger table with a complete structure, rich semantics, and data traceability capabilities, achieving an intelligent leap from "raw data accumulation" to "manageable and understandable data assets."

[0020] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this disclosure. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0022] Figure 1 A schematic diagram of a scenario that can be applied to the data governance method or data governance apparatus of the present disclosure is shown.

[0023] Figure 2 This is a flowchart illustrating a data governance method according to an exemplary embodiment.

[0024] Figure 3 This is a flowchart illustrating a field filling method according to an exemplary embodiment.

[0025] Figure 4 This is a flowchart illustrating a field filling method according to an exemplary embodiment.

[0026] Figure 5 This is a flowchart illustrating a knowledge retrieval method according to an exemplary embodiment.

[0027] Figure 6 This is a flowchart illustrating a method for determining association knowledge according to an exemplary embodiment.

[0028] Figure 7This is a flowchart illustrating a data governance method according to one embodiment.

[0029] Figure 8 This is a block diagram illustrating a data governance apparatus according to an exemplary embodiment.

[0030] Figure 9 A schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present disclosure is shown. Detailed Implementation

[0031] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0032] Those skilled in the art will recognize that embodiments of this disclosure can be a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0033] The features, structures, or characteristics described in this disclosure can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more specific details omitted, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0034] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0035] The accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus omitting repeated descriptions of them. Some block diagrams shown in the drawings do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0036] The flowchart shown in the accompanying drawings is merely illustrative and does not necessarily include all content and steps, nor does it require execution in the described order. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0037] In the description of this disclosure, unless otherwise stated, " / " means "or," for example, A / B can mean A or B. "And / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Furthermore, "at least one" means one or more, and "multiple" means two or more. The terms "first," "second," etc., do not limit the quantity or order of execution, and "first," "second," etc., do not necessarily imply differences; the terms "contains," "includes," and "has" are used to indicate an open-ended meaning of inclusion and refer to the existence of additional elements / components / etc. besides those listed.

[0038] This disclosure embodiment can be implemented by a terminal and / or a server. The terminal can obtain data from a computer device and display that data. The computer device can interact with the terminal, and can be a server hosting the application, or it can belong to the terminal (i.e., the terminal's backend), etc., without limitation.

[0039] The terminal can be a mobile phone, a laptop computer, or a playback device in a vehicle, etc., without limitation. The terminal can be considered a playback device in a vehicle, and it can display the target application. The terminal is only one example of the devices listed; the terminal in this disclosure is not limited to the listed devices. The target application in this disclosure can be any application capable of displaying multimedia information.

[0040] It is understood that the terminal mentioned in the embodiments of this disclosure can be a computer device, including but not limited to a terminal or a server. In other words, the computer device can be a server or a terminal, or a system composed of a server and a terminal. The terminal mentioned above can be an electronic device, including but not limited to mobile phones, tablets, desktop computers, laptops, handheld computers, in-vehicle devices, augmented reality / virtual reality (AR / VR) devices, head-mounted displays, smart TVs, wearable devices, smart speakers, digital cameras, webcams, and other mobile internet devices (MIDs) with network access capabilities, or terminals in scenarios such as trains, ships, and flights.

[0041] The servers mentioned above can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, vehicle-road cooperation, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0042] Optionally, the data involved in the embodiments of this disclosure may be stored in a computer device or may be stored based on cloud storage technology, without limitation.

[0043] To better understand the above-mentioned objectives, features and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0044] The following section will first explain some of the terms used in the embodiments of this disclosure so that those skilled in the art can understand them.

[0045] RAG (Retrieval-Augmented Generation) is a technique that combines information retrieval and natural language generation to improve the accuracy and reliability of AI systems when answering natural language questions.

[0046] Prompt Engineering: Prompt engineering is a technique that guides AI models to generate expected outputs by designing, optimizing, and managing prompts. The core function of prompt engineering is to ensure that AI (Artificial Intelligence) models can respond accurately and efficiently to user commands, generating content that meets requirements.

[0047] The preceding text introduced some terms and concepts involved in the embodiments of this disclosure. The following text introduces the technical features involved in the embodiments of this disclosure.

[0048] With the continuous development of artificial intelligence technology, large models have become one of the core technologies in the field. Large models have a wide range of applications. As they continue to evolve, we can see the rapid development of multimodal and cross-modal applications, no longer limited to traditional unstructured data such as natural language, vision, and speech, but also performing well in tables and Structured Query Language (SML). This application approach enhances the scope and effectiveness of large models, helping to solve more complex problems.

[0049] Currently, when enterprises collect metadata, some rely entirely on manual labor, while others can only collect the most basic technical metadata information (such as database, table names, and field names). They then need to spend significant manpower and time to complete other data information (such as the Chinese name of the table, business definitions, descriptions, and tags; the Chinese name of the field, descriptions, value descriptions, and sensitivity levels). This adds further difficulty to the already challenging governance process, which is difficult to quantify business value. The introduction of AI large-scale models can effectively solve this problem. Compared to traditional, primarily manual data governance, the reasoning capabilities of AI large-scale models can automatically mine and populate other core metadata information from the most basic technical metadata information and business sample data, effectively improving the efficiency and accuracy of metadata work.

[0050] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0051] Figure 1 A schematic diagram of a scenario that can be applied to the data governance method or data governance apparatus of the present disclosure is shown.

[0052] Please refer to Figure 1 The diagram illustrates an implementation environment provided by an exemplary embodiment of this disclosure.

[0053] like Figure 1As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0054] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, desktop computers, wearable devices, virtual reality devices, smart home devices, etc.

[0055] Server 105 can be a server that provides various services, such as a backend management server that supports the devices operated by users using terminal devices 101, 102, and 103. The backend management server can analyze and process received requests and other data, and then feed the processing results back to the terminal devices.

[0056] A server can be a standalone physical server, a server cluster or a distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. This disclosure does not impose any restrictions on this.

[0057] Server 105 may, for example, acquire asset data of the target data center, including at least one of order data, asset card data, and log data; Server 105 may, for example, construct an asset ledger table to be supplemented based on the asset data, the asset ledger table being used to record multiple equipment assets and containing multiple fields to be filled corresponding to each equipment asset; Server 105 may, for example, extract information from the asset data using a first major language model to fill in the fields to be filled, thereby generating an initial asset ledger table; Server 105 may, for example, determine the table-level lineage path and field-level lineage path of the initial asset ledger table; wherein the table-level lineage path describes the dependency relationship and flow path between data tables, and the field-level lineage path describes the data source and transformation relationship at the field level in the table; Server 105 may, for example, generate table-level metadata and field-level metadata of the asset ledger table using a second major language model based on the table-level lineage path, field-level lineage path, and the initial asset ledger table, wherein the table-level metadata and field-level metadata are used to combine with the initial asset ledger table to generate a target asset ledger table containing metadata information.

[0058] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Server 105 can be a single physical server or a combination of multiple servers. Depending on actual needs, it can have any number of terminal devices, networks, and servers.

[0059] Under the above system architecture, this disclosure provides a data governance method that can be executed by any electronic device with computing capabilities.

[0060] Figure 2 This is a flowchart illustrating a data governance method according to an exemplary embodiment. The method provided in this disclosure can be executed by any electronic device with computing power, for example, the method can be implemented by the above-described... Figure 1 The execution can be performed by a server or terminal device in the embodiments, or it can be performed by both a server and a terminal device. In the following embodiments, the server is used as the execution subject for illustration, but this disclosure is not limited to this.

[0061] Reference Figure 2 The data governance method provided in this disclosure may include the following steps.

[0062] Step S202: Obtain asset data of the target data center. The asset data includes at least one of order data, asset card data, and log data.

[0063] In some embodiments, the target data center may be an IDC (Internet Data Center).

[0064] In some embodiments, asset data of the target data center can be obtained through various data channels, such as order data (e.g., procurement system API, Excel reports), asset cards (CMDB (Configuration Management Database) database, scanned PDF documents), and log files (server access logs). Specific steps may include the following.

[0065] Structured data is synchronized periodically using Airflow (an open-source platform for orchestrating, scheduling, and monitoring workflows); unstructured text is scripted using Python (a programming language); and real-time logs are collected using Flume (an open-source, distributed, highly reliable system for collecting, aggregating, and transmitting massive amounts of logs). Data is transmitted via SFTP (Secure File Transfer Protocol), and sensitive fields (such as IP address and client ID) are encrypted using AES (Advanced Encryption Standard) to avoid plaintext storage.

[0066] In some embodiments, for structured data, a daily task can be set to automatically pull the latest data from system A into the database; for unstructured text, a Python script can be written to parse PDF file data and extract text information such as "IDC data center model: xxxx" and "purchase date: xxxx-xx-xx"; for log data, Flumeagent (an independent process) can be deployed on the log server to collect logs into a message queue and then store them in HDFS (Hadoop Distributed File System).

[0067] In some embodiments, a rule base can be defined to purify the data.

[0068] For example, a regular expression can be defined to match invalid order numbers `^IDC-\d{8}` Standardize dates to ISO8601 format (`YYYY-MM-DDHH:MM:SS`), convert IP addresses to IPv6 standard, and standardize units (e.g., TB→GB conversion).

[0069] In some embodiments, natural language recognition technology can also be used to identify device models, data center numbers (such as "BJ-IDC-03") in the text and associate them with existing asset databases.

[0070] In some embodiments, the following data cleaning effects can be achieved through the above steps.

[0071] a) Order Number Verification: The system checks whether all order numbers conform to ^IDC-\d{8} Rules. An invalid order number, such as IDC-123, will be flagged and await manual review.

[0072] b) Date standardization: Convert the extracted "December 12, 2021" into ISO format xxxx-xx-xx00:00:00.

[0073] c) Entity recognition: Using StanfordNLP (an open-source tool and library for natural language processing) to analyze text, such as “server 740 in data center BJ-IDC-03”, the system automatically identifies: Data center number: BJ-IDC-03; Device model: 740.

[0074] Step S204: Based on the asset data, construct an asset ledger table to be supplemented. The asset ledger table is used to record multiple equipment assets and contains multiple fields to be filled for each equipment asset.

[0075] To construct the asset ledger to be supplemented, a blank structured table can first be created based on a predefined standardized table structure framework (including core fields such as unique equipment identifier, type, configuration, location, and status). Subsequently, raw asset data collected from multiple sources (such as procurement information in orders, physical attributes of asset cards, and operational status in logs) will be automatically parsed. Through data source identification, key information extraction, and preliminary classification, these information fragments will be mapped to the corresponding fields to be filled in the ledger table. At this stage, the information may be incomplete or conflicting. Therefore, the system will establish a connection between each field and the original data source, forming a structurally complete but partially missing or unverified "to be supplemented" asset ledger table. This table serves as the basic input for subsequent intelligent filling and integration by a large language model.

[0076] Step S206: Extract information from asset data using the first major language model and populate the fields to be populated to generate an initial asset ledger table.

[0077] Step S208: Determine the table-level lineage path and field-level lineage path of the initial asset ledger table; where the table-level lineage path describes the dependency relationship and flow path between data tables, and the field-level lineage path describes the data source and transformation relationship at the field level in the table.

[0078] Table-level lineage path describes the dependencies and flow paths of the entire data table as a whole, including which upstream tables its data originates from and which downstream tables consume its data.

[0079] Field-level lineage paths are detailed graphs describing the data source, transformation logic, and derived relationships of a specific field in a table. They delve into the table's internal structure, tracing the "genetic" inheritance between fields.

[0080] Step S210: Based on the table-level lineage path, field-level lineage path, and initial asset ledger table, generate table-level metadata and field-level metadata of the asset ledger table through the second major language model. The table-level metadata and field-level metadata are used to combine with the initial asset ledger table to generate a target asset ledger table containing metadata information.

[0081] Table-level metadata describes the table itself. Field-level metadata describes the field.

[0082] In some embodiments, table-level metadata can be a "panoramic archive" describing the overall business positioning, flow relationships, and management attributes of a data table; field-level metadata can be a "detailed specification" defining the specific business meaning, source rules, and security level of each field in the table.

[0083] In some embodiments, table-level lineage paths (revealing the table's upstream and downstream dependencies in the data flow), field-level lineage paths (revealing the source and transformation logic of each field), and the initial asset ledger table (containing basic technical fields) can be integrated into a structured contextual information package, serving as the second major language model for prompt input. The model infers based on this deep context: combining the upstream and downstream relationships of the tables to generate table-level business definitions, data flow descriptions, and core positioning; simultaneously, inferring the business meaning, processing rules, quality constraints, and sensitivity levels of each field based on its lineage path, thereby automatically generating table-level and field-level metadata rich in business semantics and management attributes. Finally, this intelligently generated metadata is merged with the initial ledger table to form a complete, traceable, and business-understandable target asset ledger table.

[0084] In some embodiments, a full-chain data lineage view can be constructed. Table-level lineage relationships are obtained through apache_atlas (a metadata management and governance platform), and the names of upstream and downstream entities are obtained by parsing the lineage data. Field-level lineage paths (lineages) are obtained based on GUIDs (Globally Unique Identifiers).

[0085] In some embodiments, based on the table-level lineage path, field-level lineage path, and the initial asset ledger table, generating table-level metadata and field-level metadata of the asset ledger table through a second major language model may include: determining the upstream and downstream dependency tables of the asset ledger table based on the table-level lineage path of the asset ledger table; and determining the table-level metadata of the initial asset ledger table through the second major language model, combined with the upstream and downstream dependency tables. The table-level metadata includes at least one of the following: data flow path, key processing link annotations, and business definitions and descriptions.

[0086] In some embodiments, the following examples may be used to determine table-level prompt words so that the second language model can determine table-level metadata based on the table-level prompt words.

[0087] Table-level Prompt=f"""{base_desc}

[0088] Blood relationship analysis: - Upstream dependency table: {lineage['upstream']} / / This is the table for which we are generating metadata; - Downstream consumption table: {lineage['downstream']}; Please supplement with information regarding blood relations: 1. Data Flow Path Description / / Data Lifecycle Management, Business Value: Understanding the origin and development of data, establishing data trust; 2. Key processing steps are marked; 3. The business definitions and descriptions in this table / / Ensure consistency between business and technical understanding of the data; """

[0089] In some embodiments, the initial asset ledger table includes a first field. Based on the field-level lineage path of the first field and the initial asset ledger table, field-level metadata for the first field is generated using a second language model. This includes: determining the field-level lineage path of the first field; and using the second language model, based on the field-level lineage path of the first field, generating conversion rules, potential type conversion risks, and type compatibility handling code for the first field during data flow, so as to determine the field-level metadata of the first field based on the conversion rules, potential type conversion risks, and type compatibility handling code. The field-level metadata includes at least one of field description, value description, and sensitivity level.

[0090] In some embodiments, the following examples may be used to determine field-level prompt words so that the second language model can determine field-level metadata based on the field-level prompt words.

[0091] Field-level Prompt=f"""

[0092] Field name: {column_name}

[0093] Lineage: {lineage}

[0094] Please generate the data processing logic: 1. Infer conversion rules based on source field type / / By analyzing the type characteristics of the source data, automatically infer the steps and logic required to safely, accurately, and compliantly convert it into the target type.

[0095] 2. Identify potential type conversion risks / / This refers to analyzing and judging in advance which situations the conversion may fail, cause data distortion, or generate business logic errors before converting data from one type to another.

[0096] 3. Generate Type Compatibility Handling Code / / During data flow, when a field moves from the source table (Table A) to the target table (Table B), their data types may be inconsistent. "Type compatibility handling code" is an automatically generated code snippet used to safely and correctly perform this data type conversion.

[0097] 4. Generate corresponding descriptions, value explanations, and sensitivity levels.

[0098] """

[0099] In some embodiments, the core metadata information generated by combining data lineage may include table-level Chinese names and business descriptions; it may also include field-level Chinese names, descriptions, value descriptions, and sensitivity levels.

[0100] In some embodiments, a business description in natural language for the table can also be generated using a second oracle model, as illustrated in the following specific embodiment.

[0101] #IDC Card Orders Table (idc_card_orders)

[0102] ##1. Core Business Positioning

[0103] ▸ Business Objectives Record the entire lifecycle of data center resource cards (computing / storage / networking) order information.

[0104] ▸ Business Areas Data Center Resource Management -> Order Transaction Domain

[0105] ▸ Business Scenarios : - Users can purchase IDC resource cards online. -Automatically generate resource allocation work orders -Financial settlement reconciliation basis ##2. Basic Technical Parameters |Attributes|Description| |------|------| |Storage Method|Columnar Storage (Parquet Format)| |Partitioning Strategy|Partition by Monthly Order Date (order_date)| |Data Volume|Currently, there are approximately 12 million records, with an average daily increase of 8,000 records| |TTL Strategy| Orders are automatically archived 3 years after completion| ##3. Key Data Features Data range : - Time range: All valid orders from January 2021 to present - Business Scope: Includes orders with both prepaid and postpaid payment modes. - Exclusion status: Test orders that have been cancelled (status=9) Data quality : - Primary key (order_id) uniqueness: 100% - Non-empty field compliance rate: SQL user_id:100%|product_code:99.98%|order_amount:99.95% Using the methods described above, the large model integrates all the information and outputs detailed metadata information: Business Positioning: This table serves finance and BI, and is used for value analysis and cost accounting; Data Context: Data originates from BSS orders and attributes, and after processing, is provided to IDC analysis reports and business finance systems.

[0106] This technical solution achieves full-process automation and intelligence in IDC data center data governance through the collaborative work of two-stage large language models: In the first stage, the first large language model automatically extracts information from multi-source heterogeneous order, asset card, and log data and populates the initial asset ledger; In the second stage, data lineage analysis (table-level and field-level) is innovatively introduced. Through the second large language model, combined with lineage relationships, table-level metadata (such as data flow path and business definition) and field-level metadata (such as field description, conversion rules, and sensitivity level) rich in business semantics are intelligently generated. Finally, a complete, traceable, and business-understandable target asset ledger table is constructed, which completely changes the traditional data governance model that relies on manual labor, is inefficient, and lacks semantics.

[0107] In some embodiments, the asynchronous engine of vLLM (an open-source, high-performance, low-latency large language model inference and service engine) can be used in conjunction with asyncio (a library for writing concurrent code) to handle multiple requests. Caching can be done using lru_cache (for implementing a least recently used cache). Packing 100 order texts into a single batch improves GPU (Graphics Processing Unit) utilization. For orders with the same supplier, the supplier name is extracted as the key; for example, if the order data contains a supplier field, it can be extracted before processing, and the cache can be checked to see if a result with that key exists. The specific steps are as follows: Model configuration and loading: Select the "deepseek-ai / deepseek-llm-7b-chat" model, maximum batch size = 100, maximum model length = 4096; Use order ID + text hash (to ensure the same order and content hit the cache) to generate a cache key and LRU (Least Recently Used) as the cache component @lru_cache (maxsize == 1000); Iterate through each order, generate a cache key (cache_key). If the cache_key exists in the cached results, add the cached result directly to cached_results; otherwise, generate a prompt and add it to the batch_requests list. If batch_requests is not empty, iterate through each missed request, generate a task and add it to the tasks list. Use await to execute each task concurrently, collect the results, store the results in cached_results, and add the index and result to the outputs list. Finally, the results of cache hits and cache misses are merged to return a list of results arranged in the original order.

[0108] In some embodiments, the system collects 100 similar pending orders. Then, using the vLLM asynchronous engine, these 100 prompts are packaged into a batch and fed into the "deepseek-llm-7b-chat" model for inference. This significantly improves GPU utilization and is far faster than processing orders individually. If 10 orders are found to have the same supplier "Supplier AA" and similar text content, the system calculates its hash value as a cache key. If the same request has been processed before, the cached result is returned directly, avoiding duplicate calculations.

[0109] The above method, through the reasoning capabilities of large AI models, automatically mines and populates diverse core metadata information based on initial data sources and business sample data, achieving automation, real-time performance, and scalability in the generation and supplementation of data governance metadata, reducing manual costs, and improving efficiency and accuracy.

[0110] Figure 3 This is a flowchart illustrating a field filling method according to an exemplary embodiment.

[0111] refer to Figure 3 The above field filling method may include the following steps.

[0112] Step S302: Obtain asset data of the target data center. The asset data includes at least one of order data, asset card data, and log data.

[0113] Step S304: Based on the asset data, construct an asset ledger table to be supplemented. The asset ledger table is used to record multiple equipment assets and contains multiple fields to be filled for each equipment asset.

[0114] Step S306: Search the external knowledge base to determine the relevant knowledge of the asset data. The external knowledge base includes at least one of the following: historical metadata standards, industry terminology, historical order data, and equipment manuals.

[0115] In some embodiments, historical metadata standards, industry terminology, historical order data, equipment manual PDFs, assets and orders, contracts, and maintenance records can be retrieved as external knowledge.

[0116] In step S308, the first language model extracts information from asset data and related knowledge to populate fields to be filled in in order to generate an initial asset ledger table.

[0117] Step S310: Determine the table-level lineage path and field-level lineage path of the initial asset ledger table; where the table-level lineage path describes the dependency relationship and flow path between data tables, and the field-level lineage path describes the data source and transformation relationship at the field level in the table.

[0118] Step S312: Based on the table-level lineage path, field-level lineage path, and initial asset ledger table, generate table-level metadata and field-level metadata of the asset ledger table through the second major language model. The table-level metadata and field-level metadata are used to combine with the initial asset ledger table to generate a target asset ledger table containing metadata information.

[0119] This technical solution achieves high-precision automated filling of data center asset ledger fields by constructing an enhanced retrieval and prompting system that integrates with an external knowledge base. First, it retrieves historical standards and business knowledge related to the current asset data from the external knowledge base, and uses this to construct precise prompt words containing clear instructions, format constraints, and reference fragments. This guides the first language model to intelligently extract and verify information from multi-source heterogeneous data, accurately filling the ledger fields. On this basis, it further utilizes the second language model to integrate data lineage relationships and automatically generate table-level and field-level deep metadata rich in business semantics. Finally, it forms a complete, accurate, and business-understandable target asset ledger, significantly improving the accuracy, automation level, and business relevance of data governance.

[0120] Figure 4 This is a flowchart illustrating a field filling method according to an exemplary embodiment.

[0121] refer to Figure 4 The above field filling method may include the following steps.

[0122] Step S402: Based on the field name of the field to be filled and the related knowledge retrieved from the external knowledge base, construct the first prompt word.

[0123] Step S404: Input the asset data and the first prompt word into the first large language model to extract information from the asset data and perform preliminary filling of the fields to be filled.

[0124] Step S406: If there are still missing fields after the filling is completed, the first language model will refer to the association knowledge to fill in the missing fields.

[0125] In some embodiments, a clear and accurate indicator word can be constructed to help large models better understand external business knowledge of the data, thereby providing more accurate data aggregation and completion.

[0126] In some embodiments, the indicator can consist of a directive, a format, or a RAG. Three to five labeled positive examples (containing correct fields) and negative examples (such as incorrect date formats) can be inserted into the indicator (Prompt) to improve model understanding. Constraint languages ​​such as LMQL can be used to restrict the output format, such as forcing `order_id` to match the regular expression `^IDC-\d{8}`. `.

[0127] Prompt_template (prompt template) = """

[0128] You are an IDC data governance expert. Please process the input data according to the following requirements: (1). Extract fields from order text: {field list} / / Fields defined in Prompt, used to set clear expected boundaries for AI; (2) Output in JSON format, including the required fields: order_id, device_type, purchase_date; (3). If a field is missing, refer to the knowledge base content to complete it (knowledge fragment: {retrieved_context}) / / {retrieved_context} is an information fragment related to the current task retrieved from an external knowledge base; (4). Validation logic: purchase_date must not be earlier than 2020; Input text: {user_input}; """ This step mainly involves building Prompt prompts and large-scale model inference—solving the problem of "how to make AI output accurately," telling the artificial intelligence: your role, what to do, what the format is, what materials you can refer to, and what pitfalls to be aware of, and finally extracting standardized data from the text.

[0129] This technical solution significantly improves the accuracy and completeness of asset ledger field filling through a two-step intelligent filling strategy: First, it constructs precise prompt words based on the fields to be filled and the retrieved related knowledge, guiding the large model to complete the initial extraction and filling from the original asset data; for missing fields that still exist after the initial filling, the model can actively refer to related knowledge in the external knowledge base (such as historical orders and equipment manuals) to make intelligent inferences and complete the data, thereby ensuring that even if the original data is incomplete, an initial asset ledger with complete fields, accurate information, and compliance with business specifications can be generated.

[0130] Figure 5 This is a flowchart illustrating a knowledge retrieval method according to an exemplary embodiment.

[0131] refer to Figure 5 The knowledge retrieval method described above may include the following steps.

[0132] Step S502: Identify the key information to be retrieved from the asset data.

[0133] In the context of asset data, key information refers to the core data units that are crucial for the identification, management, tracking, and valuation of assets, and are essential for subsequent intelligent processing, knowledge association, or business decision-making.

[0134] In some embodiments, key information to be retrieved can be automatically determined from asset data through a combination of strategies: for structured fields, their values ​​are directly extracted as key information; for unstructured text, core entities and phrases such as equipment models, data center locations, and technical attributes are automatically extracted using named entity recognition and key phrase extraction technologies; then, the extracted information is prioritized and filtered according to preset business rules to ensure that the highest value information is retrieved first; at the same time, the system will also intelligently combine related fields to form more semantic query units, thereby providing accurate and efficient search input for subsequent knowledge base retrieval.

[0135] Step S504: Perform word segmentation on the key information to be retrieved to obtain the keywords to be retrieved.

[0136] Step S506: Vectorize the key information to be retrieved to obtain the key vector.

[0137] Step S508: Based on the keywords to be searched, perform keyword retrieval in an external knowledge base to obtain the first related knowledge.

[0138] In some embodiments, the extracted keywords to be searched can be combined into a query statement, and full-text matching can be performed using classic retrieval algorithms such as BM25 in an indexed external knowledge base to quickly find all document fragments or records containing these keywords (such as the section about the equipment in the equipment manual, or the record matching the model in historical orders). The results sorted by relevance are returned as the first associated knowledge, thereby efficiently obtaining explicit information that directly matches the keywords.

[0139] Step S510: Based on the key vector to be retrieved, perform vector similarity retrieval in an external knowledge base to obtain the second related knowledge.

[0140] In some embodiments, documents in an external knowledge base (such as equipment manuals and historical standards) are first converted into high-dimensional vectors using a semantic embedding model and stored in a vector database (such as Milvus). After obtaining the key vector to be retrieved, the system uses an approximate nearest neighbor search algorithm in the vector database to calculate the cosine similarity between the query vector and all vectors in the database. It then returns the original text fragments corresponding to the top K most relevant vectors whose similarity exceeds a set threshold (such as >0.82). These fragments are the second association knowledge that is highly related to the query semantics but may not contain the same keywords.

[0141] Step S512: Merge the first and second related knowledge to determine the related knowledge of the asset data.

[0142] This technical solution employs a hybrid retrieval strategy to accurately and comprehensively acquire knowledge highly relevant to asset data from external knowledge bases: the system executes keyword retrieval and vector similarity retrieval in parallel, respectively acquiring explicit information that directly matches and implicit knowledge related to deep semantics. Then, it merges the two results through an intelligent fusion algorithm (such as inverse sorting fusion), thereby significantly improving recall and precision while ensuring retrieval efficiency, and providing high-quality, multi-dimensional knowledge support for subsequent intelligent data processing.

[0143] Figure 6 This is a flowchart illustrating a method for determining association knowledge according to an exemplary embodiment.

[0144] refer to Figure 6 The first and second related knowledge are integrated to determine the related knowledge of asset data, including the following steps.

[0145] Step S602: Determine the ranking of the first related knowledge in the keyword search results, and use it as the first ranking.

[0146] Step S604: Determine the ranking of the second related knowledge in the vector similarity retrieval results, and use it as the second ranking.

[0147] Step S606: Add a constant k to the first ranking and take the reciprocal to obtain the reciprocal ranking score corresponding to the first related knowledge, where k is an integer greater than 1.

[0148] In some embodiments, a smoothing constant k (usually 60) can be added to the sequential position (i.e., ranking, such as 1st, 2nd, 3rd) of each knowledge item in the keyword search results, and then the reciprocal can be taken to obtain the reciprocal ranking score of that knowledge item. The higher the ranking (the smaller the value), the larger the reciprocal after adding k, and the higher the score. For example, the score of the knowledge item ranked 1st is 1 / (1+60) ≈ 0.0164, and the score of the knowledge item ranked 2nd is 1 / (2+60) ≈ 0.0161, thus transforming the original ranking into a uniform and comparable score value.

[0149] Step S602: Add a constant k to the second ranking and take the reciprocal to obtain the reciprocal ranking score corresponding to the second related knowledge.

[0150] In some embodiments, the order position (i.e., ranking, such as 1st, 2nd, 3rd) of each knowledge in the vector similarity retrieval results can be added to the same smoothing constant k (k is an integer greater than 1, usually 60), and then the reciprocal can be taken to obtain its reciprocal ranking score. This calculation method is completely consistent with that used when processing the first associated knowledge: the higher the ranking (the smaller the value), the higher its reciprocal score, thus mapping the retrieval results of two different scoring systems (keyword relevance and semantic similarity ranking) to the same dimension, laying the foundation for subsequent fair weighted fusion.

[0151] Step S608: Add the scores of the first and second-lowest rankings of each related knowledge to determine the comprehensive score of each related knowledge.

[0152] Step S610: Based on the comprehensive score, determine the association knowledge of the asset data in the first association knowledge and the second association knowledge.

[0153] For each piece of related knowledge, the system directly adds its second-lowest ranking score (calculated in the keyword search results) to its second-lowest ranking score (calculated in the vector similarity search results) to obtain a comprehensive score for that knowledge. This addition process essentially integrates the evaluation of the knowledge's relevance using both search methods with equal weight: if a piece of knowledge ranks highly in both search results, its two lowest ranking scores are both high, resulting in a significantly higher comprehensive score; conversely, if it performs well only in one search, its comprehensive score will be constrained by the lower score in the other. Finally, all related knowledge is sorted in descending order based on this comprehensive score, forming a unified final list that reflects overall relevance.

[0154] Figure 7 This is a flowchart illustrating a data governance method according to one embodiment.

[0155] refer to Figure 7 The intelligent mining method for IDC data center metadata based on a large model provided in this embodiment may include the following steps.

[0156] Step S01: Collect the original data sources of data center orders and asset cards from various collection channels.

[0157] For specific implementation steps, please refer to... Figure 2 Step S202 in the process.

[0158] Step S02: Construct an external knowledge base using RAG technology.

[0159] In some embodiments, historical metadata standards, industry terminology, historical order data, equipment manual PDFs, assets and orders, contracts, and maintenance records can be retrieved as external knowledge.

[0160] Below, this application will provide a detailed explanation and description of how to construct an external knowledge base based on external knowledge, so as to enable knowledge retrieval within it.

[0161] 1. Knowledge Extraction and Structuring. Metadata Standard Processing: Converting data attributes, field definitions, data types, and constraint rules from Excel / database into JSON Schema.

[0162] For example, convert the file "xxxIDC Data Center Rack Ledger.xlsx" into a machine-readable JSON Schema, specifying that the enumerated values ​​of the device_type field are ["switch", "router"].

[0163] 2. Vectorization and Index Optimization. Text is recursively segmented by semantic paragraphs, with a block size of 512 tokens and an overlap rate of 15%. bge-large-zh-v1.5 (an open-source Chinese text embedding model) is chosen as the base model for vectorization. An IVF_SQ8 index (a high-performance, high-compression vector index type) is built in Milvus (an open-source vector database) to accelerate retrieval. Simultaneously, the original text's BM25 (Best Matching 25) keyword index is retained for subsequent Elasticsearch keyword matching.

[0164] This step mainly involves vectorized indexing. For example, documents like "xx Data Center Resource Information Manual.pdf" are semantically segmented into 512 token segments (e.g., "Chapter 1: IDC Data Center Description" is segmented into one segment, and "Chapter 2: Rack List" is segmented into another). The bge-large-zh-v1.5 (text embedding model) model is then used to transform these text segments into strings of numbers (vectors), which are then stored in the Milvus vector database.

[0165] 3. Hybrid retrieval strategy, weighted fusion of results.

[0166] First layer: Use Elasticsearch for keyword matching (BM25 algorithm).

[0167] The body is set in the following format: body = { "query": { "match": { "content": { "query": query, "operator": "and"}}}}.

[0168] This can be achieved by directly calling Elasticsearch.search.

[0169] For example, when querying "What is the power consumption of the Chaiwan xxx IDC data center?", you can use the following method.

[0170] Keyword retrieval: Using Elasticsearch (an open-source distributed search and analysis engine that supports real-time retrieval and analysis of structured and unstructured data), the BM25 algorithm (a classic algorithm in the field of information retrieval) is used to quickly find all document fragments containing "Chaiwan", "xxx computer room", "power supply", and "power".

[0171] Second layer: Query vector library for vector similarity retrieval (cosine similarity threshold > 0.82).

[0172] Reciprocal Fusion Ranking (RRF): This method simplifies hybrid search by eliminating the problem of retrieval scores from two different retrieval methods not being on the same scale.

[0173] The search results are filtered for relevance using BERT-Classifier (a text classifier based on the BERT model), and fragments with a confidence score > 0.7 are retained and injected into the prompt words.

[0174] The specific steps are as follows: Load the pre-trained BERT model (here, 'bert-base-uncased' is selected) and the word segmenter. Then, perform further filtering of segments by calculating probabilities and confidence levels.

[0175] Step S03: Build a high-quality Prompt.

[0176] Step S04: Generate compliant and accurate metadata and extract key information from order text and log files to complete asset and order card fields.

[0177] In some embodiments, information can be extracted from asset data using the first major language model to generate standardized and accurate metadata, and key information can be extracted from order text and log files to complete asset and order card fields, thereby generating an initial asset ledger.

[0178] Step S05: Generate core metadata information by combining data lineage.

[0179] The aforementioned core metadata may include table-level metadata and / or field-level metadata.

[0180] In some embodiments, reference may be made to Figure 2 Steps S206 and S208 generate the aforementioned core metadata information.

[0181] This technical solution achieves deep intelligent mining of IDC data center metadata by constructing an integrated knowledge base and a hybrid retrieval mechanism. First, the system utilizes RAG technology to build a structured external knowledge base from multiple sources, including historical standards and technical manuals. Then, it accurately retrieves knowledge fragments relevant to business scenarios through a hybrid retrieval strategy using keywords and vector similarity. Based on this, a carefully designed prompt word project drives a large language model to extract information from original orders, asset cards, and logs, automatically completing asset ledger fields and generating preliminary metadata. Finally, by integrating data lineage relationships, core metadata containing business semantics and flow logic is further generated. This efficiently transforms messy raw data into high-quality, traceable, and business-understandable data assets, comprehensively improving the automation level and accuracy of data governance.

[0182] It should be particularly noted that the steps in the various embodiments of the above data governance method can be overlapped, substituted, added, or deleted from each other. Therefore, these reasonable permutations and combinations of data governance methods should also fall within the protection scope of this disclosure, and the protection scope of this disclosure should not be limited to the described embodiments.

[0183] It should be noted that the scope of protection of this application should include, but is not limited to, the specific implementation methods described in the embodiments. Any alternative solution that uses a different name but substantially performs the same function and achieves the same technical effect falls within the scope of protection defined by the claims of this application.

[0184] Based on the same inventive concept, this disclosure also provides a data governance device, as described in the following embodiments. Since the principle by which this device solves the problem is similar to that of the method embodiments described above, the implementation of this device embodiment can refer to the implementation of the method embodiments described above, and repeated details will not be repeated.

[0185] Figure 8 This is a block diagram illustrating a data governance apparatus according to an exemplary embodiment. (Refer to...) Figure 8 The data governance device 800 provided in this embodiment may include: an asset data acquisition module 801, a supplementary asset ledger generation module 802, an initial ledger generation module 803, a lineage path determination module 804, and a metadata generation module 805.

[0186] The asset data acquisition module 801 can be used to acquire asset data of the target data center, including at least one of order data, asset card data, and log data; the asset ledger generation module 802 can be used to construct an asset ledger to be supplemented based on the asset data, the asset ledger being used to record multiple equipment assets and containing multiple fields to be filled corresponding to each equipment asset; the initial ledger generation module 803 can be used to extract information from the asset data through a first language model and fill the fields to be filled to generate an initial asset ledger; the lineage path determination module 804... The metadata generation module 805 can be used to determine the table-level lineage path and field-level lineage path of the initial asset ledger table; wherein the table-level lineage path describes the dependency relationship and flow path between data tables, and the field-level lineage path describes the data source and transformation relationship at the field level in the table; the metadata generation module 805 can be used to generate the table-level metadata and field-level metadata of the asset ledger table based on the table-level lineage path, the field-level lineage path and the initial asset ledger table, through a second major language model, wherein the table-level metadata and the field-level metadata are used to combine the initial asset ledger table to generate a target asset ledger table containing metadata information.

[0187] It should be noted that the asset data acquisition module 801, the supplementary asset ledger generation module 802, the initial ledger generation module 803, the lineage path determination module 804, and the metadata generation module 805 correspond to S202 to S210 in the method embodiment. The examples and application scenarios implemented by these modules and their corresponding steps are the same, but they are not limited to the content disclosed in the above method embodiment. It should also be noted that these modules, as part of the apparatus, can be executed in a computer system such as a set of computer-executable instructions.

[0188] In some embodiments, the data governance device 800 may further include an external knowledge retrieval module.

[0189] The external retrieval module can be used to search in an external knowledge base to determine the associated knowledge of the asset data; the external knowledge base includes at least one of historical metadata standards, industry terminology, historical order data, and equipment manuals.

[0190] The initial ledger generation module 803 may include: a related knowledge analysis submodule.

[0191] Among them, the related knowledge analysis submodule can always extract information from the asset data and the related knowledge through the first large language model to fill the fields to be filled.

[0192] In some embodiments, the first large language model extracts information from the asset data and the associated knowledge to populate the field to be populated, including: constructing a first prompt word based on the field name of the field to be populated and the associated knowledge retrieved from the external knowledge base; The asset data and the first prompt word are input into the first large language model to extract information from the asset data and perform preliminary filling of the fields to be filled; if there are still missing fields after filling, the first large language model refers to the association knowledge to complete the missing fields.

[0193] In some embodiments, based on the table-level lineage path, the field-level lineage path, and the initial asset ledger table, table-level metadata and field-level metadata of the asset ledger table are generated using a second language model. This includes: determining the upstream dependency table and downstream dependency table of the asset ledger table based on the table-level lineage path of the asset ledger table; and determining the table-level metadata of the initial asset ledger table using the second language model, combined with the upstream dependency table and the downstream dependency table. The table-level metadata includes at least one of the following: data flow path description, key processing step annotation, and business definition and description.

[0194] In some embodiments, the initial asset ledger table includes a first field; wherein, based on the field-level lineage path of the first field and the initial asset ledger table, the field-level metadata of the first field is generated through a second major language model, including: determining the field-level lineage path of the first field; and generating, through the second major language model and based on the field-level lineage path of the first field, conversion rules, potential type conversion risks, and type compatibility processing code for the first field during data flow, so as to determine the field-level metadata of the first field based on the conversion rules, potential type conversion risks, and type compatibility processing code, wherein the field-level metadata includes at least one of field description, value description, and sensitivity level.

[0195] In some embodiments, retrieving the asset data from an external knowledge base to determine associated knowledge includes: identifying key information to be retrieved in the asset data; performing word segmentation on the key information to be retrieved to obtain keywords to be retrieved; performing vectorization on the key information to be retrieved to obtain a key vector to be retrieved; performing keyword retrieval in the external knowledge base based on the keywords to be retrieved to obtain first associated knowledge; performing vector similarity retrieval in the external knowledge base based on the key vector to be retrieved to obtain second associated knowledge; and fusing the first associated knowledge and the second associated knowledge to determine the associated knowledge of the asset data.

[0196] In some embodiments, fusing the first associated knowledge and the second associated knowledge to determine the associated knowledge of the asset data includes: determining the ranking of the first associated knowledge in the keyword retrieval results as a first ranking; determining the ranking of the second associated knowledge in the vector similarity retrieval results as a second ranking; adding a constant k to the first ranking and taking its reciprocal to obtain the reciprocal ranking score corresponding to the first associated knowledge, where k is an integer greater than 1; adding a constant k to the second ranking and taking its reciprocal to obtain the reciprocal ranking score corresponding to the second associated knowledge; adding the reciprocal ranking scores of the first and second rankings of each associated knowledge to determine the comprehensive score of each associated knowledge; and determining the associated knowledge of the asset data based on the comprehensive score among the first associated knowledge and the second associated knowledge.

[0197] Since the functions of the device 800 have been described in detail in their respective method embodiments, they will not be repeated here.

[0198] The modules and / or sub-modules and / or units described in the embodiments of this disclosure can be implemented in software or hardware. The described modules and / or sub-modules and / or units can also be located in a processor. The names of these modules and / or sub-modules and / or units do not, in some cases, constitute a limitation on the module and / or sub-module and / or unit itself.

[0199] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a portion of a module or program segment containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer program instructions.

[0200] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0201] Figure 9 A schematic diagram of an electronic device suitable for implementing embodiments of the present disclosure is shown. It should be noted that... Figure 9 The electronic device 900 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0202] like Figure 9 As shown, the electronic device 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage section 908 into a random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the electronic device 900. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0203] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 910 as needed so that computer programs read from it can be installed into storage section 908 as needed.

[0204] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing computer program instructions for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs the functions defined above in the system of this disclosure.

[0205] It should be noted that the computer-readable storage medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable computer program instructions. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. Computer program instructions contained on a computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0206] In another aspect, this disclosure also provides a computer-readable storage medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The aforementioned computer-readable storage medium carries one or more programs, which, when executed by a device, enable the device to perform the following functions: acquiring asset data of a target data center, the asset data including at least one of order data, asset card data, and log data; constructing an asset ledger table to be supplemented based on the asset data, the asset ledger table recording multiple device assets and containing multiple fields to be filled corresponding to each device asset; extracting information from the asset data using a first language model to fill the fields to be filled, thereby generating an initial asset ledger table; determining the table-level lineage path and field-level lineage path of the initial asset ledger table; wherein the table-level lineage path describes the dependency relationship and flow path between data tables, and the field-level lineage path describes the data source and transformation relationship at the field level in the table; and generating table-level metadata and field-level metadata of the asset ledger table using a second language model based on the table-level lineage path, field-level lineage path, and the initial asset ledger table, wherein the table-level metadata and field-level metadata are used to combine with the initial asset ledger table to generate a target asset ledger table containing metadata information.

[0207] According to one aspect of this disclosure, a computer program product or computer program is provided, comprising computer program instructions stored in a computer-readable storage medium. The computer program instructions are read from the computer-readable storage medium, and a processor executes the computer program instructions to implement the methods provided in various optional implementations of the above embodiments.

[0208] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions of the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, or portable hard drive) and includes several computer program instructions to cause an electronic device (such as a server or terminal device) to execute the method according to the embodiments of this disclosure.

[0209] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

[0210] It should be understood that this disclosure is not limited to the detailed structures, drawing arrangements or implementations shown herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A data governance method, characterized in that, include: Obtain asset data of the target data center, wherein the asset data includes at least one of order data, asset card data, and log data; Based on the asset data, an asset ledger table to be supplemented is constructed. The asset ledger table is used to record multiple equipment assets and contains multiple fields to be filled for each equipment asset. Information is extracted from the asset data using the first major language model to populate the fields to be filled, thereby generating an initial asset ledger table. Determine the table-level lineage path and field-level lineage path of the initial asset ledger table; The table-level lineage path describes the dependencies and flow paths between data tables, while the field-level lineage path describes the data source and transformation relationships at the field level within a table. Based on the table-level lineage path, the field-level lineage path, and the initial asset ledger table, table-level metadata and field-level metadata of the asset ledger table are generated through a second major language model. The table-level metadata and the field-level metadata are used to combine with the initial asset ledger table to generate a target asset ledger table containing metadata information.

2. The method according to claim 1, characterized in that, The method further includes: The asset data is retrieved from an external knowledge base to determine the associated knowledge; the external knowledge base includes at least one of the following: historical metadata standards, industry terminology, historical order data, and equipment manuals. The process involves extracting information from the asset data using a primary language model and filling in the fields to be filled to generate an initial asset ledger, including: The first large language model extracts information from the asset data and the associated knowledge to populate the fields to be populated.

3. The method according to claim 2, characterized in that, The first large language model extracts information from the asset data and the associated knowledge to populate the fields to be populated, including: Based on the field name of the field to be filled and the related knowledge retrieved from the external knowledge base, a first prompt word is constructed; The asset data and the first prompt word are input into the first large language model to extract information from the asset data and perform preliminary filling of the fields to be filled. If fields are still missing after the filling is completed, the first language model will refer to the aforementioned association knowledge to fill in the missing fields.

4. The method according to claim 1, characterized in that, Based on the table-level lineage path, the field-level lineage path, and the initial asset ledger table, the table-level metadata and field-level metadata of the asset ledger table are generated using the second major language model, including: The upstream and downstream dependency tables of the asset ledger table are determined based on the table-level lineage path of the asset ledger table. Using the second major language model, and in conjunction with the upstream dependency table and the downstream dependency table, the table-level metadata of the initial asset ledger table is determined. The table-level metadata includes at least one of the following: data flow path description, key processing link annotation, and business definition and description.

5. The method according to claim 1, characterized in that, The initial asset ledger table includes a first field; wherein, based on the field-level lineage path of the first field and the initial asset ledger table, field-level metadata of the first field is generated through a second language model, including: Determine the field-level lineage path of the first field; Using the second language model, based on the field-level lineage path of the first field, conversion rules, potential type conversion risks, and type compatibility processing code for the first field during data flow are generated. This allows for the determination of field-level metadata for the first field based on the conversion rules, potential type conversion risks, and type compatibility processing code. The field-level metadata includes at least one of field description, value description, and sensitivity level.

6. The method according to claim 2, characterized in that, Searching external knowledge bases to determine the associated knowledge of the asset data, including: Identify the key information to be retrieved from the asset data; The key information to be retrieved is segmented into words to obtain the keywords to be retrieved; The key information to be retrieved is vectorized to obtain the key vector to be retrieved; Based on the keywords to be searched, a keyword search is performed in the external knowledge base to obtain the first related knowledge; Based on the key vector to be retrieved, a vector similarity search is performed in the external knowledge base to obtain the second related knowledge. The first and second related knowledge are fused together to determine the related knowledge of the asset data.

7. The method according to claim 6, characterized in that, The first and second related knowledge are fused to determine the related knowledge of the asset data, including: Determine the ranking of the first related knowledge in the keyword search results, and use it as the first ranking; Determine the ranking of the second related knowledge in the vector similarity retrieval results, and use it as the second ranking; Add a constant k to the first ranking and take the reciprocal to obtain the reciprocal ranking score corresponding to the first related knowledge, where k is an integer greater than 1; Add a constant k to the second ranking and take the reciprocal to obtain the reciprocal ranking score corresponding to the second related knowledge. The scores of the first and second-lowest rankings of each related knowledge are added together to determine the comprehensive score of each related knowledge. Based on the comprehensive score, the associated knowledge of the asset data is determined from the first associated knowledge and the second associated knowledge.

8. A data governance device, characterized in that, include: The asset data acquisition module is used to acquire asset data of the target data center, wherein the asset data includes at least one of order data, asset card data, and log data; The module for generating an asset ledger to be supplemented is used to construct an asset ledger to be supplemented based on the asset data. The asset ledger is used to record multiple equipment assets and contains multiple fields to be filled for each equipment asset. The initial asset ledger generation module is used to extract information from the asset data through the first language model and fill the fields to be filled in in order to generate the initial asset ledger. The lineage path determination module is used to determine the table-level lineage path and field-level lineage path of the initial asset ledger table; The table-level lineage path describes the dependencies and flow paths between data tables, while the field-level lineage path describes the data source and transformation relationships at the field level within a table. The metadata generation module is used to generate table-level metadata and field-level metadata of the asset ledger table based on the table-level lineage path, the field-level lineage path, and the initial asset ledger table, using a second major language model. The table-level metadata and the field-level metadata are used to combine with the initial asset ledger table to generate a target asset ledger table containing metadata information.

9. An electronic device, characterized in that, include: Memory and processor; The memory is used to store computer program instructions; the processor calls the computer program instructions stored in the memory to implement the data governance method as described in any one of claims 1-7.

10. A computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the data governance method as described in any one of claims 1-7.

11. A computer program product comprising computer program instructions stored in a computer-readable storage medium, characterized in that, When the computer program instructions are executed by the processor, they implement the method of any one of claims 1-7.

Images