Cross-system metadata multi-condition retrieval and index updating method based on Gravitino

Abstract

The invention relates to the field of cross-system metadata retrieval, in particular to a cross-system metadata multi-condition retrieval and index updating method based on Gravitino, which comprises the following steps of: collecting full-quantity metadata through a Gravitino Client hierarchy, and storing the full-quantity metadata in an H2 database after standardizing the full-quantity metadata; constructing double indexes through IK word segmentation and Hash mapping, and persisting the double indexes to a local file; triggering incremental synchronization based on a Spring timed task, and calling a corresponding interface to obtain data; mD5 checksum comparison is calculated, data change is identified, and an index is updated; and analyzing the multi-condition query, and after sorting and paging, exposing a RESTful interface through SpringBoot. According to the method, through double indexes, timing increment synchronization and standardized storage technologies, the problems of metadata disorder, low retrieval efficiency and the like are solved, millisecond-level matching is realized, the accuracy is improved, and the integration cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of cross-system metadata retrieval, and in particular to a method for cross-system metadata multi-condition retrieval and index update based on Gravitino. Background Technology

[0002] In the process of enterprise digital transformation, metadata, as the core descriptive information of data assets, is widely distributed across various business systems, data warehouses, databases, and other cross-system environments. It is diverse in type, complex in dimensions, and frequently changes dynamically. To achieve unified management of cross-system metadata, open-source frameworks such as Apache Gravitino have emerged, providing a unified metadata management interface to support the collection and basic management of cross-system metadata, thus solving the problem of scattered metadata storage to some extent. However, as the demand for accurate and efficient multi-condition retrieval of metadata becomes increasingly urgent, existing technologies have gradually revealed several shortcomings: First, the lack of standardized processing procedures after metadata collection leads to inconsistencies in capitalization and formatting of core information such as table names and field names, resulting in disorganized data that cannot form a unified benchmark, affecting retrieval accuracy and traceability. Second, the retrieval mechanism relies on a single index structure or a full scan approach, resulting in low retrieval efficiency and difficulty in achieving millisecond-level response when faced with multi-condition combined queries involving keyword fuzzy queries and tag filtering. Third, the lack of an effective incremental synchronization and index update mechanism means that the index cannot be updated in a timely manner after metadata is added, modified, or deleted, leading to poor data consistency and delayed query results. Fourth, the absence of a unified retrieval and operation interface requires different business systems to develop their own adaptation logic, resulting in high integration costs and failing to meet the flexible query needs of multiple business scenarios. These shortcomings make existing Gravitino-based metadata management solutions ill-suited for the complex retrieval needs of businesses, hindering the efficient utilization of metadata assets. Summary of the Invention

[0003] To address the technical deficiencies in the background technology, this invention proposes a cross-system metadata multi-condition retrieval and index update method based on Gravitino, which solves the aforementioned technical problems and meets practical needs. The specific technical solution is as follows: The Gravitino-based method for cross-system metadata multi-condition retrieval and index update includes the following steps: GravitinoClient calls the directory listing interface and table listing interface to collect full metadata from cross-system data sources in a hierarchical manner. After standardized encapsulation, the metadata is stored in the H2 database to form baseline data. The IK word segmenter is used to segment the table and field names. A keyword inverted index is built through hash mapping. At the same time, a forward and reverse index mapping between tags and metadata is established through tag entity encapsulation. The dual index is fully built and persisted to a local file. Based on the Spring scheduled task configuration, the incremental synchronization is triggered at a set frequency, and the table list interface and table structure description interface of GravitinoClient are called to obtain data. The fifth version of the message digest algorithm is used to calculate the checksum of the table structure. It is compared with the previous checksum stored locally to identify the data change type and update the corresponding index. Parse multi-condition query requests, perform combined queries by filtering by tags and matching by keywords, sort and paginate the results, and expose a RESTful interface through Spring Boot for invocation.

[0004] Furthermore, the specific steps for collecting full metadata are as follows: Start GravitinoClient and establish a connection to the cross-system data source, and confirm the API call permissions; Call the directory listing interface of GravitinoClient to obtain all cross-system directory information, and traverse them one by one by directory; For each directory, call the table list interface to obtain basic information about all tables in that directory, including table name, directory, and creation time; For each table, iterate through its field hierarchy and collect field names, data types, field comments, and modification times; The collected metadata is deduplicated and duplicate records are removed to ensure that all metadata is complete and without redundancy.

[0005] Furthermore, the specific steps of the standardized packaging are as follows: Define a custom metadata entity class, specifying the table name, field names, data types, field comments, directory, creation time, modification time, unique identifier, update timestamp, and base time; The collected table names and field names are standardized by converting them to lowercase to eliminate data recognition conflicts caused by differences in case. Each standardized metadata record is automatically assigned a globally unique identifier, which serves as the core identifier for subsequent index association and data traceability. Record the update timestamp of each piece of metadata and synchronize the base time of the first synchronization; The standardized and encapsulated metadata is transmitted to the H2 database for storage in a unified format.

[0006] Furthermore, the storage steps of the H2 database are as follows: Initialize the embedded H2 database, configure the data storage path, access account and permissions, and ensure data storage security; Create a baseline data storage table, and design the unique identifier number, standardized table name, standardized field name, data type, field comments, directory, creation time, modification time, update timestamp, and baseline time for each field corresponding to the metadata. The standardized and encapsulated metadata is written into the storage table one by one, and a unique identifier is established to map it to other attributes. Configure an automatic database persistence mechanism to periodically synchronize in-memory data to disk files, preventing data loss due to system restarts; Create an index for the storage table, using the unique identifier number and the directory it belongs to as index fields, to improve the efficiency of subsequent data queries.

[0007] Furthermore, the specific steps for constructing the keyword inverted index are as follows: Read all standardized table and field names from the H2 database and group them according to their unique metadata identifiers. The IK word segmenter is called to segment the table names and field names in each group, extracting independent keywords with actual semantic meaning and filtering out redundant characters. Initialize the hash map container, setting the key to the tokenized keywords and the value to a set of unique metadata identifiers; Each keyword is associated with its corresponding metadata unique identifier, and the unique identifier is added to the set corresponding to the keyword to form a mapping relationship between keywords and metadata; Perform integrity checks on the completed keyword inverted index to ensure that each keyword can be accurately associated with its corresponding metadata unique identifier.

[0008] Furthermore, the specific steps for establishing the forward and reverse index mapping between tags and metadata are as follows: Define a custom tag entity class, specifying the tag name, tag type, tag description, and creation time; Initialize two hash map containers, one for building a forward index and the other for building a reverse index. The key of the forward index is the tag name, and the value is the set of associated metadata unique identifiers. The key of the reverse index is the metadata unique identifier, and the value is the set of associated tag names. The association between metadata and tags can be established through manual association or rule-driven association. Rule-driven association automatically matches the corresponding tags based on the directory and data type of the metadata. Write the relationships into two hash mapping containers respectively to complete the construction of the forward index and the reverse index; The tag entity class data is associated with the forward and reverse index mapping data for storage, ensuring the consistency between tag information and index mapping.

[0009] Furthermore, the specific steps for completing the full construction of the dual indexes and persisting them to local files are as follows: Define the index persistence file format, specifying the index type identifier, key-value pair separator, data record terminator, and encoding format; Write the keyword inverted index data into the local first file according to the defined format. Each index record contains keywords, a set of metadata unique identifier numbers, and the index creation time. Write the forward and reverse index data of the tags to a local second file according to the defined format. The forward index record contains the tag name and a set of metadata unique identifier numbers, and the reverse index record contains the metadata unique identifier number and a set of tag names. Enable file locking mechanisms to prevent data conflicts caused by concurrent writes from multiple threads and ensure the atomicity of index data writes; regularly back up local persistent files, generate backup files and record backup times to avoid index data loss due to file corruption.

[0010] Furthermore, the specific steps of the incremental synchronization are as follows: Configure a timed expression based on Spring scheduled tasks to set an incremental synchronization task to be triggered every 10 minutes. After the task is triggered, the GravitinoClient calls the table list interface to obtain the basic information of all current tables, compares it with the baseline time stored in the H2 database, and filters out the tables that have been added or changed since the baseline time. For the selected tables, call the table structure description interface to obtain detailed table structure information, including field names, data types, and the latest status of field comments; Calculate the fifth version checksum of the message digest algorithm for each changed table structure and compare it with the previous checksum stored in the H2 database; The data change type is determined based on the comparison results. If the unique identifier number of the metadata does not exist, it is newly added data; if the checksum is inconsistent, it is modified data; if the unique identifier number exists locally but is not returned by the client, it is deleted data.

[0011] Furthermore, the specific steps for updating the corresponding index are as follows: For new data, repeat the steps of building the keyword inverted index and the tag forward and reverse indexes to generate corresponding index entries and add them to the dual index system; For modified data, delete the original corresponding index entries, rebuild the keyword inverted index and tag forward and reverse index entries for the data, and replace the original index data; For deleted data, all index entries corresponding to the data are removed from the keyword inverted index and the tag forward and reverse indexes to ensure that there is no invalid data in the index system; Update the checksum record and timestamp of the corresponding data in the H2 database, and synchronize the base time to the current synchronization time. The updated dual-index data is re-persisted to a local file, overwriting the original file, to ensure that the persisted data is consistent with the in-memory index data.

[0012] Furthermore, the specific steps after parsing the multi-condition query request are as follows: Receive multi-condition query requests sent by clients, parse keywords, tags, data types, and directories in the requests, and perform format validation and legality verification; Based on the parsed query conditions, the set of metadata unique identifiers that meet the tag conditions is first filtered out by the tag forward index, and then the set of metadata unique identifiers that meet the keyword conditions is matched by the keyword inverted index. The data type and the directory to which they belong are further filtered to obtain the final target set of metadata unique identifiers. The matching degree between the target metadata and the query conditions is calculated based on the frequency of keyword occurrence and the tightness of tag association, and sorted from high to low matching degree; The pagination tool receives the custom number of items to display per page and the current page number as parameters, performs pagination on the sorted metadata, extracts the metadata records for the corresponding page numbers, and completes the metadata's full attribute information and associated tag information from the H2 database. This system builds RESTful interfaces based on Spring Boot, including search, tag creation, tag query, tag modification, and tag deletion interfaces. It uses standard HTTP request methods and returns JSON data, including response status codes, response information, and processed metadata or tag data, for clients to use.

[0013] Compared with existing technologies, the Gravitino-based cross-system metadata multi-condition retrieval and index update method provided by this invention has the following advantages: This invention achieves standardized storage of cross-system metadata through hierarchical collection and standardized encapsulation. It builds an efficient retrieval foundation with a keyword-tag dual-index system, ensures data consistency through a timed incremental synchronization mechanism, and finally provides convenient query and operation capabilities through a unified RESTful interface. This invention not only effectively solves the problems of messy metadata, inefficient multi-condition retrieval, inconsistency between index and metadata, and complex integration in existing technologies, but also achieves the comprehensive effects of traceable metadata management, millisecond-level multi-condition matching, resource-saving incremental synchronization, and convenient integration of multiple systems. It significantly improves the accuracy, response speed, and system adaptability of cross-system metadata retrieval, reduces the integration cost of business systems, and fully meets the core needs of efficient metadata utilization in various business scenarios. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating the cross-system metadata multi-condition retrieval and index update method based on Gravitino in this invention.

[0015] Figure 2 This is a diagram of the dual-index architecture in this invention.

[0016] Figure 3 This is the logic diagram for incremental synchronization in this invention. Detailed Implementation

[0017] In the description of this invention, it should be understood that the terms "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "middle," and "inner," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, features defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, it should be noted that unless otherwise explicitly specified and limited, the terms "installed," "connected," and "joined" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention through specific circumstances.

[0018] The embodiments of the present invention will be described below with reference to the accompanying drawings and related examples. The embodiments of the present invention are not limited to the following examples, and the present invention relates to the relevant necessary components in this technical field, which should be regarded as well-known technology in this technical field and can be known and mastered by those skilled in this technical field.

[0019] See Figure 1-3 This invention provides a method for cross-system metadata multi-condition retrieval and index update based on Gravitino, comprising the following steps: Step S100: Call the directory listing interface and table listing interface through GravitinoClient to collect full metadata from cross-system data sources by hierarchical traversal, and store it in the H2 database after standardized encapsulation to form baseline data; GravitinoClient is a client tool for interacting with the Apache Gravitino metadata management framework. It serves as a unified entry point for accessing directory and table-level metadata across system data sources, shielding the differences between underlying heterogeneous systems. The directory listing interface is an API method provided by GravitinoClient for retrieving information about all directories (Catalogs) under a specific data source. It can be used to obtain a list of top-level namespaces in cross-system data sources, serving as the starting point for hierarchical traversal. The table listing interface is an API method provided by GravitinoClient for retrieving information about all tables in a specified directory or schema. It can be used to further retrieve a list of specific data tables below the directory level, supporting a full data collection process. Cross-system data sources are multiple independently running databases or data management systems within an enterprise with different storage structures. They can encompass various storage formats such as relational databases, data lakes, and data warehouses, each with its own independent metadata management system. They can be used to constitute the actual source environment for metadata distribution and are the target set for data collection operations. Full metadata refers to all metadata information obtained from all target data sources in a complete data collection cycle. It can include structured descriptive information such as directories, schemas, tables, fields, data types, and comments. It can be used as an initial baseline dataset, providing raw input for subsequent standardization processing and index construction. Standardization encapsulation is the process of converting and normalizing metadata from different systems according to unified rules. It can eliminate naming conflicts caused by system differences, creating a consistent and comparable metadata representation. H2 Database is a lightweight, embedded relational database management system based on Java. It can be used to persistently store standardized metadata as a local baseline data source for subsequent index construction. The baseline data is a stable and consistent set of metadata stored in the H2 Database after standardization. It can have timestamps and version identifiers, providing a reliable data baseline for retrieval and synchronization operations, ensuring consistency and traceability in the processing.

[0020] Step S200: Use the IK word segmenter to segment the table and field names, build a keyword inverted index through hash mapping, and at the same time establish a forward and reverse index mapping between tags and metadata through tag entity encapsulation, complete the full construction of the dual index and persist it to a local file; IK Segmenter is a Chinese text analysis tool based on dictionary matching and maximum forward segmentation algorithms. It can be used to semantically decompose unstructured text information such as table names and field names, improving the hit rate of fuzzy searches. Table and field names are name strings used in databases to identify data objects. They can contain letters, numbers, underscores, and mixed characters. They can be used as the main extraction objects for keyword indexing, directly affecting the coverage and accuracy of fuzzy queries. Hash mapping is a data structure organization method that uses hash functions to quickly locate key-value pairs to storage locations. It can be used to accelerate the mapping and lookup process from keywords to corresponding metadata records, improving index building and query efficiency. Keyword inverted index is an index structure that uses keywords as keys and points to a list of metadata record IDs containing the keyword as values. It can be used to achieve efficient fuzzy matching and full-text search capabilities for text content such as table names and field names. Tag entity encapsulation is a data organization method that structurally binds business or technical tags to metadata objects. It can be used to give metadata a categorizable and filterable semantic dimension, supporting fast filtering based on tags. The forward and reverse index mapping between tags and metadata maintains a bidirectional association structure, simultaneously linking tags to metadata objects (forward) and metadata objects to tags (reverse). This supports bidirectional queries: users can search for related metadata by tags or view all tags associated with a given metadata element. The dual index is a composite index system comprised of a keyword inverted index and tag forward and reverse indexes. These two index structures are built independently but work together to serve the same query process. This allows for both fuzzy text matching and structured tag filtering capabilities, improving the overall efficiency of multi-condition combined queries. Local files are persistent storage paths on the running node used to store index data. This ensures long-term preservation of index data, avoiding reconstruction on each restart and improving system availability.

[0021] Step S300: Configure a timed expression based on Spring scheduled tasks, trigger incremental synchronization at a set frequency, and call the table list interface and table structure description interface of GravitinoClient to obtain data; Spring scheduled tasks are a task scheduling component based on the Spring framework, used to periodically execute specific logic. They can be used to trigger incremental synchronization processes, automating the periodic checking of metadata changes. The timing expression is a time-defined rule expression used to define the frequency of task execution, allowing precise control over the triggering timing of incremental synchronization tasks to meet update frequency requirements in different scenarios. Incremental synchronization is a process that collects and processes only the metadata that has changed since the last synchronization. It can focus on changed data to reduce resource consumption, lower system load and network overhead, and improve synchronization efficiency while ensuring data consistency. The table structure description interface is an API method provided by GravitinoClient for obtaining detailed structure information of a single table. It can be used to obtain the specific table structure content used for validation and calculation, supporting change identification.

[0022] Step S400: Calculate the Message Digest Algorithm Version 5 checksum of the table structure, compare it with the previous checksum stored locally, identify the data change type, and update the corresponding index; The Message Digest Algorithm Version 5 (MD5) checksum can be a fixed-length hash value generated from the table structure content using the MD5 algorithm. This can serve as a unique fingerprint of the table structure, quickly determining whether it has been modified. Data change types describe the categories of changes to metadata relative to its previous state, including three basic forms: addition, modification, and deletion. These can guide the selection of subsequent index update strategies, ensuring the index accurately reflects the latest state. Index updates are operations that make partial adjustments to the existing index structure based on the metadata change type. These can include actions such as inserting new entries, modifying existing entries, and deleting invalid entries, used to maintain consistency between the index and the source metadata, avoiding delayed or incorrect query results.

[0023] Step S500: Parse the multi-condition query request, perform a combined query by filtering by tags and matching by keywords, sort and paginate the results, and expose a RESTful interface through Spring Boot for invocation.

[0024] Multi-condition query requests are metadata retrieval instructions containing multiple filtering conditions. These can include keyword fuzzy matching, tag exact matching, and category attribute filtering, used to express precise retrieval intent in complex business scenarios and improve query flexibility. Tag filtering is the process of filtering metadata records that meet certain conditions from a candidate set based on specified tag values. It can be used to quickly locate a set of metadata IDs with specific tags using a tag-based forward index, narrowing the range of data to be matched and improving the efficiency of subsequent keyword matching. Keyword matching is the process of searching for records containing specified keywords in the candidate metadata set. It can be used to find a list of corresponding document IDs using a keyword-based inverted index, combined with weighted sorting results, to achieve fuzzy search and semantic relevance matching for table names, field names, etc. The combined query result is the final metadata set obtained after the tag filtering and keyword matching processes. Duplicates have been removed and preliminary sorting has been completed. It awaits pagination processing and reflects the effective matching results under multiple combined constraints, embodying the user's query intent. Sorting and pagination is an operation that sorts query results according to specified rules and then truncates them to a subset based on page size. It can be used to sort based on fields such as rating, popularity, and update time. Pagination is implemented using offsets and limits, optimizing the front-end display experience and preventing performance bottlenecks caused by returning too much data at once. SpringBoot is a simplified application development toolkit based on the Spring framework. It can be used to quickly build web services through automatic configuration and starter dependencies, embedding a Tomcat container and exposing RESTful interfaces to handle request processing and response output for metadata query services. RESTful interfaces are HTTP interfaces designed according to the REST architectural style, used for resource manipulation and access. They can be used to manipulate URI resources using HTTP verbs such as GET and POST, transmit JSON formatted data, and provide a unified and standardized way to query and access metadata for external systems, reducing integration complexity.

[0025] This invention uses MD5 checksum and idempotency checksum to accurately identify three types of metadata changes, avoiding duplicate or missed updates. The dynamic index update mechanism ensures that the index can be synchronized in a timely manner after metadata changes, solving the pain points of inconsistency between index and metadata and delayed query results in the prior art. The incremental update mode only operates on the index corresponding to the changed data, further saving system resources and ensuring the stability of the system in high-concurrency scenarios.

[0026] In one embodiment of the present invention, the specific steps for collecting full metadata are as follows: Step S101: Start GravitinoClient and establish a connection with the cross-system data source, and confirm the interface call permissions; Initialize the GravitinoClient configuration parameters, including the IP address, port number, communication protocol, and authentication method of the cross-system data source; call the GravitinoClient connection initialization method to initiate network connection requests with each cross-system data source and establish a TCP communication link; send an authorization verification request to the data source, submit the preset authentication information (such as username, password, and authorization token), and receive the authorization verification result returned by the data source; if the verification passes, record the current connection status as "valid" and proceed to the subsequent data collection process; if the verification fails, trigger an exception alarm and terminate the data collection, and output detailed logs of insufficient permissions or authentication failure for easy troubleshooting by operations and maintenance personnel.

[0027] Step S102: Call the directory listing interface of GravitinoClient to obtain all cross-system directory information, and traverse them one by one by directory; The GravitinoClient's listCatalogs interface is invoked to send directory information query requests to various cross-system data sources; the directory information set returned by the data source is received, and the name, system to which it belongs, unique identifier, and other attributes of each directory are parsed and stored in a temporary cache container; the directory information is sorted in the order of "system to which it belongs + directory name" to avoid duplication or omission during traversal; the directory set is traversed one by one in an iterative manner, a unique traversal identifier is assigned to each directory, the current traversal progress is recorded, and if the traversal is interrupted, it can resume from the breakpoint to ensure the continuity of data collection.

[0028] Step S103: For each directory, call the table list interface to obtain the basic information of all tables under that directory, including table name, directory, and creation time; For the currently traversed directory, the GravitinoClient's table list interface (listTables) is called, passing in the directory's unique identifier as a query parameter to request information on all tables under that directory. The system receives a collection of table information returned by the data source, filters out invalid tables marked as "deleted" or "disabled," and retains only tables in a normal state. Each table's name, directory (with consistency checks against the currently traversed directory to prevent cross-directory table mixing), and creation time are parsed and stored as a "table basic information entity." A temporary unique identifier (such as a hash value of "directory unique identifier + table name") is assigned to each table for subsequent field-level data collection and location.

[0029] Step S104: For each table, traverse its field hierarchy and collect field names, data types, field comments, and modification times; For each table obtained in step S103, the GravitinoClient table structure description interface (describeTable) is called, passing in the table's temporary unique identifier, and requesting detailed field-level information for that table; the set of field information returned by the data source is received, and each field's field name, data type (such as VARCHAR, INT, DATE), field comment (business meaning explanation), and modification time (the time of the most recent field structure change) are parsed; the field information is preprocessed in terms of format: the naming convention for data types is unified (such as unifying "VARCHAR2" to "VARCHAR"), and special characters (such as newline characters and tab characters) in field comments are removed to ensure that the field information format is consistent; the association relationship between "table basic information - field information" is established, and all field information of the same table is classified under the table's temporary identifier, forming a three-level structured data of "directory - table - field".

[0030] Step S105: Perform deduplication verification on the collected metadata, remove duplicate records, and ensure that all metadata is complete and without redundancy.

[0031] The core unique identifier is "directory + table name + field name" (because there will be no tables with the same name in the same directory, and no fields with the same name in the same table). All collected field-level metadata is traversed, and the core unique identifier of each record is stored in a hash set (the key of a HashMap). If the core unique identifier of a record already exists in the hash set, it is determined to be a duplicate record, directly removed, and a deduplication log is recorded (including detailed information about the duplicate record and the removal time). If the core unique identifier does not exist in the hash set, the record is retained, and the core unique identifier is stored in the hash set. After deduplication, the total number of valid records and the number of duplicate records are counted, and a collection result report is generated to ensure that all metadata covers all valid tables and fields across system directories, without omissions or redundancy.

[0032] This invention employs a dual mechanism of "connection establishment + permission verification" to prevent unauthorized clients from accessing cross-system data sources, thus preventing metadata leakage or malicious tampering and resolving the security risks associated with the lack of permission control in the data collection process in existing technologies. It utilizes a hierarchical data collection logic of "directory traversal → table traversal → field traversal," covering each layer from the top-level directory to the bottom-level fields, ensuring that all valid metadata in the cross-system environment is collected, and resolving the metadata omission problem caused by chaotic data collection logic in existing technologies.

[0033] In one embodiment of the present invention, the specific steps of the standardized packaging are as follows: Step S106: Define a custom metadata entity class, specifying the table name, field names, data types, field comments, directory, creation time, modification time, unique identifier, update timestamp, and base time; Custom metadata entity classes can be program-level data structures used to uniformly represent metadata information from different systems. They can provide standardized data carriers, ensuring that all metadata participates in subsequent processing with a consistent structure. Define custom metadata entity classes based on Java (or a corresponding programming language), naming the class `MetadataEntity`, ensuring the class name has semantic distinctiveness. Specify the class's attribute fields and data types: table name (string type), field name (string type), data type (string type, storing VARCHAR, INT, etc. field type descriptions), field comment (string type), directory (string type, associated with cross-system directory identifiers), creation time (datetime type), modification time (datetime type), unique identifier (string type, storing UUID), update timestamp (long integer, storing millisecond-level timestamps), and base time (datetime type). Configure getter / setter methods for each attribute field to support attribute reading and assignment. Override the entity class's `equals()` and `hashCode()` methods, using the "unique identifier" as the core judgment criterion to ensure the accuracy of metadata entity uniqueness.

[0034] Step S107: Standardize the format of the collected table names and field names by converting them to lowercase to eliminate data recognition conflicts caused by differences in case. The string-to-lowercase conversion method is a string format conversion utility that can uniformly convert strings with any combination of uppercase and lowercase to all lowercase, and is a core technical means to standardize the format of table names and field names. It extracts the table name and field name of each table from the raw metadata collected in full; performs string-to-lowercase processing on the extracted table names and field names: calls the String.toLowerCase() method to uniformly convert mixed-case names such as "USER_INFO" and "User_Name" to "user_info" and "user_name" respectively; performs illegal character validation on the converted table names and field names (filtering spaces, special symbols, etc.) to ensure format compliance; and assigns the standardized table names and field names to the corresponding attribute fields of the MetadataEntity, overwriting the original collected name information.

[0035] Step S108: Automatically assign a globally unique identifier to each standardized metadata record, which will serve as the core identifier for subsequent index association and data traceability; A globally unique identifier (UUID) is a unique identity assigned to each piece of standardized metadata. It is crucial throughout the entire process of metadata storage, indexing, and change tracking, ensuring data uniqueness and relevance. An integrated UUID generation tool (such as Java's java.util.UUID class) generates a globally unique UUID string (32-bit character combination) for each piece of standardized metadata. The generated UUID is verified to ensure its uniqueness (avoiding duplicate generation in extreme cases). This unique identifier is then assigned to the "Unique Identifier" attribute field of the MetadataEntity. A mapping relationship between the unique identifier and other metadata attributes is established and stored in the temporary cache for quick lookup in subsequent steps.

[0036] Step S109: Record the update timestamp of each piece of metadata and synchronize the base time of the first synchronization; The update timestamp records the time (accurate to milliseconds) of each metadata change, used for change identification and sequential sorting during subsequent incremental synchronization to ensure the traceability of data changes. The base time marks the time when metadata was first collected and standardized, serving as the time benchmark for subsequent incremental synchronization and used to filter metadata that has changed after the base time. The current system time is obtained, converted to a millisecond-level long integer timestamp, and assigned to the "Update Timestamp" attribute field of the MetadataEntity (during the initial collection, the update timestamp is consistent with the base time). The current system date and time (formatted as "yyyy-MM-ddHH:mm:ss.SSS") is obtained and used as the base time for the initial synchronization, assigned to the "Base Time" attribute field of the MetadataEntity. If subsequent metadata changes, only the "Update Timestamp" is updated to the system time at the time of the change; the "Base Time" remains unchanged from the initial synchronization time, ensuring the uniqueness and stability of the base time.

[0037] Step S110: Transfer the standardized and encapsulated metadata into the H2 database for storage in a unified format.

[0038] The unified format input involves writing standardized, encapsulated metadata entities into the H2 database storage table using a fixed data format (such as JSON or SQL insert statements) according to the field definitions, ensuring structural consistency of the stored data. The data transmission format converter is initialized to convert MetadataEntity entity objects into SQL insert statements (or JSON format data) that conform to the field definitions of the H2 database storage table. During the conversion process, it ensures a one-to-one correspondence between attribute fields and database table fields: standardized table name → database "standardized table name" field, unique identifier → database "unique identifier" field, etc. A batch insert mechanism is used to batch-import multiple standardized metadata entities into the H2 database, reducing the number of database connections and improving storage efficiency. Transaction control is added during storage; if the storage of a metadata record fails, a transaction rollback is triggered to avoid incomplete datasets due to successful storage of some data, ensuring the atomicity of storage operations.

[0039] It should be noted that the specific steps for storing the H2 database are as follows: Step S111: Initialize the embedded H2 database, configure the data storage path, access account and permissions, and ensure data storage security; The embedded H2 database is a lightweight relational database that supports embedded deployment (running in the same process as the application, without requiring a separate database server), and dual modes of in-memory storage and disk persistence. It features millisecond-level read / write response speeds and low resource consumption, and is the core carrier for storing baseline metadata in this method. The data storage path is the physical file storage location of the H2 database (configurable to a specified directory on the local disk), used for persistent storage of metadata and database logs to ensure data is not lost after system restarts. Integrate the H2 database dependency package into the system project, and specify the database connection parameters through configuration files (such as application.properties): driver classpath, JDBC connection URL, access username and encrypted password (using MD5 encryption to store the password to avoid plaintext leakage); configure the database running mode to "embedded + disk persistence", and enable memory caching to accelerate read and write; enable database access control: only allow four types of data operation permissions: "SELECT, INSERT, UPDATE, DELETE", and disable structure modification permissions such as "DROP, ALTER" to prevent malicious tampering of the base data storage table; configure the database logging function to record detailed logs of all data operations (including write, update, query) (including operation time, operator, operation content) for easy security auditing and fault tracing; after initialization, initiate a test connection to verify the stability of the database connection and the validity of permissions. If the connection fails, trigger an alarm and terminate the subsequent process, outputting a "database initialization failed" log.

[0040] Step S112: Create a baseline data storage table and design the unique identifier number, standardized table name, standardized field name, data type, field comment, directory, creation time, modification time, update timestamp, and baseline time for each field's corresponding metadata. The baseline data storage table is a specially designed structured data table in the H2 database. Its fields correspond one-to-one with the attributes of the standardized metadata entity class (MetadataEntity), making it the core data structure for storing baseline metadata. The baseline data storage table is created using SQL statements, named "METADATA_BASE," and uses the InnoDB storage engine (supporting transactions and row-level locking, adapting to high-concurrency write scenarios). The table fields and data types are designed to ensure a one-to-one correspondence with the attributes of the standardized metadata entity class (MetadataEntity). A composite unique constraint is added, and the table creation SQL statement is executed. If table creation fails, the operation is rolled back and detailed exception logs are output to ensure the compliance of the stored table structure.

[0041] Step S113: Write the standardized and encapsulated metadata into the storage table one by one, and establish the association mapping between the unique identifier number and other attributes; Association mapping refers to the correspondence between the "globally unique identifier" of metadata and other attributes (standardized table name, field name, data type, etc.). Through this mapping, the complete information of metadata can be quickly located by the unique identifier. The standardized and encapsulated MetadataEntity entity collection is read from the temporary cache, grouped by "directory," and metadata under the same directory is processed in batches. A "pre-compiled SQL + batch insertion" mechanism is used to traverse the MetadataEntity entities, assigning each attribute value to the parameters of the pre-compiled SQL in field order, ensuring a one-to-one correspondence between the "unique identifier" and other attributes, thus establishing an association mapping. Each batch of data insertion is treated as an independent transaction. If the insertion of a batch of data fails, a transaction rollback is triggered, and the insertion is retried (up to 3 times). If it still fails, the abnormal metadata information is recorded and the data is skipped, avoiding the overall failure of batch insertion due to a single abnormal data entry. After the insertion is completed, the primary key (unique identifier) ​​of each batch of data is queried to verify the integrity of the inserted data, ensuring no data is missing or insertion has failed.

[0042] Step S114: Configure the database automatic persistence mechanism to periodically synchronize memory data to disk files to avoid data loss due to system restart; Automatic persistence is a core feature of the H2 database. It allows for configurable timed synchronization strategies, asynchronously writing data from memory to disk files, balancing memory read / write efficiency with data persistence security. H2 database persistence parameters can be configured, including setting the period for synchronizing memory data to disk, enabling the database checkpoint mechanism, configuring disk file backup strategies, and real-time monitoring of metrics such as data synchronization success rate and disk space utilization via its built-in monitoring interface. Alarms are triggered if synchronization fails or disk space is insufficient (remaining space <10GB).

[0043] Step S115: Create a storage table index, using the unique identifier number and the directory it belongs to as index fields to improve the efficiency of subsequent data queries.

[0044] The storage table index is an indexing mechanism based on the H2 database. It is a retrieval optimization structure built on the key fields of the base data storage table to accelerate field query speed and reduce the time complexity of subsequent data filtering.

[0045] For the "Unique Identifier (ID)" field, a primary key index is built using a B+ tree structure, supporting precise queries at the O(logn) level. For the "Catalog" field, a regular index is built to quickly filter all metadata under the same directory, adapting to subsequent queries that filter by directory. Index configuration is optimized by setting the index cache size to 5000, improving the memory hit rate of index queries. Automatic index optimization is disabled (to avoid background optimization consuming system resources), and instead, a manual OPTIMIZETABLE operation is performed daily at 2 AM to optimize the index structure. Index effectiveness is verified by performing targeted query tests, comparing query response times before and after index creation to ensure an efficiency improvement of over 50%. Index maintenance logs are recorded, including index creation time, optimization time, and query performance metrics, to facilitate adjustments to the index strategy based on business query volume.

[0046] This invention addresses the pain points of existing technologies, such as lack of security control in metadata storage and susceptibility to data loss due to system restarts or abnormal crashes, through a multi-layered protection design of "access control + transaction mechanism + WAL log + periodic persistence." It ensures the security and integrity of the baseline data storage, providing a reliable data foundation for the entire method. The field design of the baseline data storage table strictly corresponds to the standardized metadata entity classes. Combined with the association mapping of "unique identifier number - attribute," the originally scattered cross-system metadata is transformed into a structured and associative dataset. This solves the problems of messy metadata storage and unclear relationships in existing technologies, providing a direct basis for metadata association during subsequent index construction.

[0047] In one embodiment of the present invention, the specific steps for constructing the keyword inverted index are as follows: Step S201: Read all the standardized table names and field names data from the H2 database, and group and associate them according to the metadata unique identification number. Establish a connection with the embedded H2 database through the JDBC driver of Java, configure the connection URL, database username and password, enable connection pool management to avoid performance loss caused by frequent connection creation; execute the predefined SQL query statement to read all the standardized metadata core fields at once, and traverse the result set through ResultSet; use the "metadata unique identification number" as the grouping key, encapsulate the corresponding standardized table name and standardized field name into a string array, and store it in the temporary cache to ensure that the table name and field name of the same metadata are associated as a whole, providing a complete data unit for subsequent word segmentation.

[0048] Step S202: Call the IK word segmenter to perform word segmentation on the table names and field names in each group, split out independent keywords with actual semantics, and filter redundant characters. Integrate the IK word segmenter into the system, load the core components of the word segmenter through the automatic configuration of SpringBoot, and configure the word segmentation mode as the dual mode of "exact word segmentation + intelligent word segmentation"; traverse the string array of the grouped table names and field names, and call the IKAnalyzer.split method to perform word segmentation. For example, split "user_info_detail" into "user", "info", "detail", and split "用户订单金额" into "用户", "订单", "金额"; preset a filtering dictionary, including meaningless auxiliary words (such as "的", "之", "与"), punctuation marks (such as "_", "-", "."), single-character meaningless words (such as "一", "个", "数") and system reserved words, and eliminate redundant characters through double verification of string matching and regular expressions.

[0049] Step S203: Initialize the hash map container, set the key as the keyword after word segmentation, and the value as the set of metadata unique identification numbers. Select HashMap in Java as the hash map container, set the initial capacity to 1.5 times the estimated total number of keywords (calculated based on the historical metadata word segmentation statistics results), and set the load factor to 0.75 to balance the storage efficiency and query performance; clarify that the key (Key) of the container is a single keyword (String type) after word segmentation to ensure that the keyword is unique and non-repeating; the value (Value) is a set of metadata unique identification numbers (implemented using ArrayList, supporting dynamic expansion) used to store all metadata identifiers containing this keyword; in the multi-threaded concurrent word segmentation scenario, add a ReentrantLock lock to the hash map container to avoid keyword loss or set confusion during concurrent writing, and ensure the atomicity of index construction.

[0050] Step S204: Associate each keyword with its corresponding metadata unique identifier, and add the unique identifier to the set corresponding to the keyword to form a mapping relationship between keywords and metadata; A HashMap from Java is chosen as the hash mapping container, with its initial capacity set to 1.5 times the estimated total number of keywords (calculated based on historical metadata segmentation statistics). The load factor is set to 0.75 to balance storage efficiency and query performance. The container's key is defined as a single keyword (String type) after segmentation, ensuring that keywords are unique and not duplicated. The value is a collection of unique metadata identifiers (implemented using ArrayList, supporting dynamic expansion), used to store all metadata identifiers containing the keyword. In multi-threaded concurrent segmentation scenarios, a ReentrantLock is added to the hash mapping container to prevent keyword loss or collection corruption during concurrent writes, ensuring the atomicity of index construction.

[0051] Step S205: Perform integrity verification on the completed keyword inverted index to ensure that each keyword can be accurately associated with the corresponding metadata unique identifier number.

[0052] The design employs a dual verification mechanism. The first layer is "quantity verification," which counts the total number of keywords constructed in step S204 and the total number of metadata unique identifier associations, comparing them with the total amount of metadata read in step S201 and the total number of valid keywords in step S202 to ensure consistency. The second layer is "accuracy verification," which randomly selects 10% of the keywords, retrieves the corresponding metadata identifiers through the keyword query hash mapping container, and then queries the metadata in the H2 database to verify the accuracy of the actual association between the keywords and the metadata. If the quantity verification is inconsistent, the missing keywords or metadata identifiers are located and traced back to the data reading, word segmentation, or association steps to complete them. If the accuracy verification shows a mismatch, the erroneous association is deleted and the association step is re-executed, while an exception log is recorded (including keywords, erroneous metadata identifiers, and the reason for the exception). The verification logic is encapsulated into an independent utility class and integrated into the final stage of the index building process, requiring no manual intervention. After verification, a verification report is output (including the verification pass rate, the number of exceptions, and the processing results).

[0053] This invention solves the problems of inefficient single-index retrieval and inaccurate keyword matching in existing technologies through a closed-loop design of "standardized data reading → precise word segmentation and filtering → efficient container initialization → complete association mapping → strict integrity verification". Its core innovation lies in combining the semantic recognition capabilities of IK word segmentation with the efficient query capabilities of HashMap to construct a keyword inverted index that is "semantically accurate, fast in querying, and has complete associations". This index, together with the tag forward and reverse indexes of claim 6, forms a dual-index synergy, providing core support for multi-condition combined retrieval. Ultimately, it achieves millisecond-level retrieval response, significantly improving the efficiency and accuracy of cross-system metadata retrieval, and meeting the patent's inventive step requirement of "highlighting substantial features and significant progress".

[0054] In one embodiment of the present invention, the specific steps for establishing the positive and negative index mapping between tags and metadata are as follows: Step S206: Define a custom tag entity class, specifying the tag name, tag type, tag description, and creation time; Tag entity classes are structured data carriers that encapsulate the core attributes of tags. Through object-oriented principles, they integrate scattered information such as tag names, types, and descriptions into class objects with a unified format. This standardizes the definition, transmission, and storage of tag data, ensuring the consistency and maintainability of tag information. Employing object-oriented programming principles, this system defines structured tag entity classes based on the Java language, setting the access permissions of these classes to public to ensure cross-module call feasibility. It defines core attributes and their corresponding data types: tag name (String type, non-empty and unique, serving as the core index key), tag type (String type, used to distinguish business tags / technical tags / category tags, such as "structured data" and "user behavior data" belonging to technical tags), tag description (String type, supplementing the tag's semantic description, such as "structured data: two-dimensional table data stored in a relational database"), and creation time (LocalDateTime type, accurate to milliseconds, recording the tag creation timestamp). The `equals()` and `hashCode()` methods are overridden to implement object uniqueness verification based on the tag name, avoiding duplicate tag creation. Getter / setter methods are provided to ensure attribute encapsulation and controllable access, while the `Serializable` interface is implemented to support tag data persistence and network transmission.

[0055] Step S207: Initialize two hash mapping containers, which are used to build a forward index and a reverse index respectively. The key of the forward index is the tag name and the value is the set of associated metadata unique identifier numbers. The key of the reverse index is the metadata unique identifier number and the value is the set of associated tag names. Hash mapping containers are key-value storage containers implemented based on hash table data structures. They store data by mapping keys to specified positions in the hash table, offering efficient query, insertion, and deletion operations, and are one of the core data structures for building indexes. Forward indexes are index structures that use tag names as search keys to associate queries with the corresponding set of unique metadata identifiers. Their core function is to quickly filter out all metadata that meets specific tag conditions. Inverse indexes are index structures that use unique metadata identifiers as search keys to associate queries with the corresponding set of tag names. Their core function is to quickly retrieve all tag information associated with a specific metadata, supporting tag display, tag modification, and other functions.

[0056] The core reason for choosing Java's HashMap as the hash mapping container is that HashMap is based on a hash table and has an average query, insertion, and deletion complexity of O(1), which can meet the performance requirements of high-frequency index read and write scenarios. Initialize a forward-indexed container: Define it as a HashMap <String,Set <string>The forwardIndex is a string representing the tag name (a unique string) in the tag entity class, where the key is the tag name and the value is a HashSet. <string>A collection of unique identifiers for metadata of a type. The properties of HashSet ensure that there are no duplicate metadata IDs within the collection. Initialize the inverted index container: define it as a HashMap <String,Set <string>`reverseIndex`, where the key is a unique identifier for metadata (a String type, globally unique), and the value is a HashSet. <string>A collection of tag names of the same type, also using HashSet to avoid duplicate tag names; Set initial capacity (e.g., 16) and load factor (e.g., 0.75) for both containers, and use concurrency control mechanisms (e.g., using ConcurrentHashMap instead of HashMap, or adding locks during read and write operations) to avoid concurrent modification exceptions in a multi-threaded environment and ensure thread safety of index operations.

[0057] Step S208: Establish the association between metadata and tags through manual association or rule-driven association. Rule-driven association automatically matches the corresponding tags based on the directory and data type of the metadata. The manual association method is as follows: The tag management interface is built on Spring Boot, providing a visual operation process for tag selection, metadata selection, and association submission. Users select the target tag in the interface, select the metadata to be associated (batch selection is supported), and click the "Associate" button. The front end sends the association relationship (tag name + metadata unique identifier) ​​to the back end via RESTful API. After receiving the request, the back end first verifies the validity of the tag and metadata (such as whether the tag exists or whether the metadata has been deleted). After the verification is successful, the association relationship is written.

[0058] The rule-driven association method is as follows: The design rules configuration module allows users to preset association rules through the interface. Each rule consists of two parts: trigger conditions and matching tags. Trigger conditions are based on metadata attributes such as the directory and data type (e.g., "directory = user data warehouse" or "data type = MySQL table"). Matching tags are automatically associated tags when the rule is met (e.g., "user data" or "structured data"). Rules are stored in the H2 database and include fields such as rule ID, trigger condition expression, matching tag name, and activation status. During metadata initialization (full collection) or incremental synchronization (addition / modification), the rule engine is triggered. The rule engine reads the metadata's directory and data type attributes, iterates through all enabled rules, and determines whether the metadata meets the trigger conditions. If the conditions are met, the matching tags are automatically obtained, establishing an association between the metadata and the tags, without manual intervention.

[0059] Step S209: Write the association relationship into two hash mapping containers respectively to complete the construction of the forward index and the reverse index; Iterate through all established "tag name - metadata unique identifier" association records, and perform bidirectional write operations for each record. Forward index writes are as follows: extract the tag name from the association record, use it as the key of the forward index container, and search for the metadata ID set (HashSet) corresponding to that key; if the set does not exist, create a new HashSet and add it to the container; add the metadata ID from the association record to this set, completing the forward index update. Reverse index writes are as follows: extract the metadata ID from the association record, use it as the key of the reverse index container, and search for the tag name set (HashSet) corresponding to that key; if the set does not exist... Then, a new HashSet is created and placed into the container; the tag name from the associated record is added to the set to complete the reverse index update; concurrency control is added during the write process (such as using the synchronized keyword or ReentrantLock lock) to avoid data loss or set structure corruption caused by multiple threads writing to the set corresponding to the same key at the same time; after the write is completed, index verification is performed: for each associated record, the forward index and reverse index are queried respectively to verify whether the set corresponding to the tag name in the forward index contains the metadata ID, and whether the set corresponding to the metadata ID in the reverse index contains the tag name, to ensure the accuracy of index writing.

[0060] Step S210: Associate and store the tag entity class data with the forward and reverse index mapping data to ensure the consistency between tag information and index mapping.

[0061] Tag entity data storage involves serializing predefined tag entity objects and writing them to the tag table in the H2 database (designed fields include tag name, tag type, tag description, creation time, and last modification time), with the tag name serving as the primary key to ensure uniqueness. Index mapping data storage involves serializing the forward and reverse index HashMap data according to a preset format (e.g., "index type|key|value set (elements separated by commas)|storage timestamp") and writing it to two independent persistent files locally (forward index file and reverse index file). The file names include a date suffix for easy version tracking. Establishing association mappings involves adding a "key" to the tag table in the H2 database. The "Index File Association Identifier" field stores the storage path and version number of the corresponding index in the local file. Simultaneously, the primary key (tag name) of the tag table is added to the header of the local index file, forming a two-way association between "database tag data - local index file". The synchronization update mechanism triggers index association updates when tag entity class data is modified (e.g., tag description changes) or deleted: when a tag is modified, the tag name association information in the local index file is updated synchronously (if the tag name has not changed, there is no need to modify the core index data; if the tag name has changed, the key value in the forward and reverse indexes must be updated synchronously); when a tag is deleted, all index entries corresponding to that tag in the local index file are deleted synchronously, and the association identifier is updated.

[0062] In one embodiment of the present invention, the specific steps for completing the full construction of the dual index and persisting it to a local file are as follows: Step S211: Define the index persistence file format, specifying the index type identifier, key-value pair separator, data record terminator, and encoding format; Adopting a design approach of "type identifier + structured separation," unique type identifiers for the three types of indexes are selected and placed at the beginning of the file. The non-ASCII character ∑ is used as the key-value pair separator to avoid conflicts with keyword and tag names. \r\n is used as the data record terminator to ensure convenient parsing. UTF-8 is specified as the encoding format, and a format validation field is added at the end of the file. All specifications are uniformly fixed in the system configuration file. This design completely solves the problems of mixed storage of different indexes, chaotic parsing, and cross-environment compatibility. UTF-8 encoding can adapt to keyword and tag storage in multilingual scenarios. The fixed specifications reduce the parsing time of a single index to the microsecond level, laying a solid foundation for efficient response of subsequent multi-condition searches.

[0063] Step S212: Write the keyword inverted index data into the local first file according to the defined format. Each index record contains keywords, a set of metadata unique identifiers, and the index creation time. The local storage path for the first file adopts a hierarchical structure of "system root directory + date subdirectory". After reading the keyword inverted index data from memory, it is sorted according to the lexicographical order of the keywords. Each index record strictly follows the format of "keyword + key-value pair separator + metadata unique identifier set + data record terminator + index creation time". A buffered stream batch write strategy is adopted, and a flush operation is performed every 1,000 index records. After the write is completed, a corresponding MD5 checksum file is automatically generated for data integrity verification. This method uses lexicographical sorting to enable subsequent keyword queries to use a binary search method, improving retrieval efficiency by more than 40%. The batch write mechanism controls the write time of 100,000-level index data to within 300 milliseconds. The MD5 checksum file can quickly identify corruption problems during file transmission or storage, ensuring the validity of the index data.

[0064] Step S213: Write the tag forward and reverse index data into the local second file according to the defined format. The forward index record contains the tag name and the set of metadata unique identifier numbers, and the reverse index record contains the metadata unique identifier number and the set of tag names. First, sort the forward index data of the tags by lexicographical order of the tag names, and sort the reverse index data of the tags by ascending order of the unique metadata identifier. Then, write the two sorted data into the corresponding sections of the local second file. The file names are consistent with the date of the keyword inverted index file. This design merges the two types of tag indexes for storage, reducing the number of files by 50% compared to separate storage. It is easier to manage and read them in a unified manner. The ordered storage structure supports tag range query scenarios, and the unified naming standard provides a clear identifier for index matching during subsequent incremental updates, which improves the reading efficiency of the tag index by 25%.

[0065] Step S214: Enable file locking mechanism to prevent data conflicts caused by concurrent writing by multiple threads and ensure the atomicity of index data writing; regularly back up local persistent files, generate backup files and record backup time to avoid index data loss due to file corruption.

[0066] Employing a file-level exclusive lock mechanism, the system automatically acquires an exclusive lock on the target file before writing index data. If the acquisition fails, the file enters a blocking queue and retryes at 500-millisecond intervals, with a maximum of 10 retries. If an exception occurs during the writing process, a rollback mechanism is immediately triggered, releasing the file lock and deleting the written data. Upon successful writing, the lock is released and a detailed operation log is recorded. This mechanism completely resolves various data conflict issues caused by multi-threaded concurrent writing, achieving a 100% success rate for index file parsing. Atomicity guarantees reduce the frequency of index rebuilding due to incomplete data. The blocking retry and logging design improves system maintainability and perfectly adapts to the synchronization requirements of batch writing scenarios.

[0067] A dual backup strategy of "time-triggered + size-triggered" is adopted. The time-triggered backup is set to start at 2:00 AM every day (during off-peak business hours) via a scheduled task. The size-triggered backup is set to be triggered immediately when the index file size exceeds 100MB. Before backup, the integrity of the original file is verified by MD5. After the verification is successful, the "copy first, verify later" process is executed to generate a backup file with a timestamp, and the backup log is recorded synchronously. At the same time, historical backup files older than 30 days are automatically cleaned up. If the backup fails, a system alarm is sent immediately and the backup is automatically retried after 1 hour. This dual-triggered mechanism fully covers scheduled backups and sudden large-capacity backup scenarios. The integrity verification of the backup file ensures the accuracy of data recovery, keeping the data recovery time within 5 minutes. The automatic cleanup strategy effectively balances data security and disk storage space usage, and is fully adapted to the needs of long-term stable operation in the production environment.

[0068] In one embodiment of the present invention, the specific steps of incremental synchronization are as follows: Step S301: Configure a timed expression based on Spring scheduled tasks to set an incremental synchronization task to be triggered every 10 minutes; The system utilizes the `@Scheduled` annotation provided by the Spring framework to configure a timed expression, specifically set to "00 / 10***?", to trigger every 10 minutes. This expression is embedded in the system task configuration class, and the trigger frequency can be dynamically adjusted via configuration files. Task triggering avoids peak business hours, ensuring that the synchronization process does not consume core business resources. This design replaces manual synchronization with a timed trigger mechanism, avoiding the problem of long-term index lag after metadata changes. The 10-minute trigger interval ensures data timeliness while reducing resource consumption by more than 90% compared to full synchronization, making it suitable for business scenarios with dynamic metadata changes across systems.

[0069] Step S302: After the task is triggered, the basic information of all current tables is obtained by calling the table list interface through GravitinoClient, and compared with the base time stored in the H2 database to filter out the tables that have been added or changed since the base time. After the task is triggered, a stable connection is first established through GravitinoClient. The table list interface is then called to obtain basic information about all tables (including table name, directory, creation time, modification time, and table unique identifier). Next, the last synchronization baseline time is read from the H2 database. The creation time and modification time of each table are compared with the baseline time. If a table's creation time is later than the baseline time, it is considered a newly added table; if its modification time is later than the baseline time, it is considered a modified table. Simultaneously, the filtering results are recorded, and a list of tables to be processed is generated. This filtering logic achieves precise filtering through timestamp comparison, retaining only the table data that needs to be synchronized. This reduces the amount of data processed by more than 80%, avoiding the efficiency loss caused by full traversal. Furthermore, the unique identifier of the table is used for auxiliary verification to prevent misjudgments due to duplicate table names, ensuring the accuracy of the filtering results.

[0070] Step S303: For the selected tables, call the table structure description interface to obtain detailed table structure information, including the latest status of field names, data types, and field comments; For the selected new or modified tables, the table structure description interface is called in batches according to their respective directories. The interface calls use an asynchronous batch processing mode, with the concurrency of each group controlled at 5-10 to avoid interface rate limiting. The obtained detailed table structure information includes complete attributes such as field names, data types, field comments, field lengths, NOT NULL constraints, and default values. Simultaneously, the format of the data returned by the interface is validated. If any fields are missing or the format is abnormal, the call is automatically retried (up to 3 times) to ensure that the obtained table structure information is complete and meets standardization requirements. This method improves interface call efficiency through batch asynchronous calls, and the complete field attribute collection provides comprehensive data support for subsequent validation and index updates. The format validation and retry mechanism ensures a 99.9% success rate in obtaining table structure information, avoiding incomplete data synchronization due to interface call anomalies.

[0071] Step S304: Calculate the fifth version checksum of the message digest algorithm for each changed table structure and compare it with the previous checksum stored in the H2 database; First, the obtained detailed table structure information is concatenated into a string in the order of "field name-data type-field comment-field length-NOT constraint". After removing irrelevant spaces and special characters, the checksum of this string is calculated using the MD5 algorithm. Then, the previous checksum of the corresponding table is retrieved from the H2 database, and the current checksum is compared with the previous checksum bit by bit. If the two checksums are completely consistent, it means that the table structure has not undergone substantial changes and no further processing is required; if they are inconsistent, it is determined that the table structure has been modified and index updates are required. This verification method achieves fast table structure verification through hash value comparison, with the verification time for a single table controlled within 10 milliseconds. Compared with comparing fields one by one, the efficiency is improved by 90%. At the same time, the uniqueness of the MD5 checksum ensures the accuracy of change judgment and avoids invalid synchronization caused by adjustments to irrelevant fields.

[0072] Step S305: Determine the data change type based on the comparison results. If the unique identifier number of the metadata does not exist, it is new data. If the checksum is inconsistent, it is modified data. If the unique identifier number exists locally but the client does not return it, it is deleted data.

[0073] First, the selected tables undergo unique identifier verification. If the unique identifier of a table has no matching record in the H2 database, it is determined to be newly added data. If the unique identifier exists but the current checksum is inconsistent with the previous checksum, it is determined to be modified data. Simultaneously, after obtaining the basic information of all current tables, the unique identifiers of all tables stored in the local H2 database are compared. If a unique identifier exists locally but the current table list interface does not return the corresponding table information, the table is determined to have been deleted and marked as deleted data. The entire determination process is implemented through a triple logic of "unique identifier verification + checksum comparison + bidirectional comparison," comprehensively covering the three types of metadata change scenarios: addition, modification, and deletion. The bidirectional comparison mechanism solves the problem of local non-detection after a table is deleted across systems, achieving 100% coverage of change type determination. This provides a clear basis for subsequent accurate index updates and ensures the consistency between the index and cross-system metadata.

[0074] In one embodiment of the present invention, the specific steps for updating the corresponding index are as follows: Step S401: For the new data, repeat the steps of building the keyword inverted index and the tag forward and reverse index to generate the corresponding index entries and add them to the dual index system; Once data is determined to be newly added, the system automatically triggers the index building process. First, it reads the standardized table name, field names, and associated tag information of the new data from the H2 database. Then, it generates keywords using the IK tokenizer and establishes an inverted index mapping. Finally, it creates forward and reverse index associations through tag entity encapsulation. After generating independent index entries, these are inserted into the memory structures of the keyword inverted index and the tag forward and reverse indexes, respectively, according to the index type. This design ensures that new metadata can be quickly integrated into the existing index system without requiring a full index rebuild; incremental building is only needed for each single data entry, keeping index update time within milliseconds. Simultaneously, it guarantees that new data can be immediately identified by multi-condition searches, achieving the timeliness of "add to search immediately."

[0075] Step S402: For the modified data, delete the original corresponding index entries, rebuild the keyword inverted index and tag forward and reverse index entries for the data, and replace the original index data; First, the dual-index system is traversed using unique metadata identifiers to precisely locate and delete all keyword-related entries in the keyword inverted index and the bidirectional mapping entries between the data and tags in the tag forward and reverse indexes, ensuring that old index records are completely cleared. Then, based on the modified and updated table structure information, the keyword segmentation, tag association, and index building processes are re-executed to generate new index entries and insert them into the corresponding index structure, completing the replacement of the old data. This atomic "delete-then-build" operation avoids retrieval ambiguity caused by the coexistence of old and new index data, ensuring that the index data is completely consistent with the modified metadata. At the same time, it saves more than 95% of resource consumption compared to a full index rebuild, adapting to business scenarios with frequent metadata modifications.

[0076] Step S403: For deleted data, delete all index entries corresponding to the data from the keyword inverted index and the tag forward and reverse indexes to ensure that there is no invalid data in the index system; Based on the unique identifier of the data to be deleted, the index cleanup process is initiated. In the keyword inverted index, all keyword sets containing this unique identifier are traversed, and the target identifier is removed from each set. In the tag forward index, all tag-metadata mappings associated with this identifier are deleted. In the tag reverse index, the tag set entries corresponding to this identifier are directly deleted. After cleanup, an integrity check is performed on the dual-index system to ensure no residual invalid associations remain. This cleanup mechanism thoroughly eliminates invalid data from the index system, avoiding decreased retrieval efficiency and inaccurate results caused by invalid indexes. It also reduces index storage space usage, improving index query IO efficiency by over 20%.

[0077] Step S404: Update the checksum record of the corresponding data in the H2 database, update the timestamp, and synchronize and refresh the base time to the current synchronization time; For newly added data, the calculated MD5 checksum of the table structure and the data creation time are used as the update timestamp, along with the current synchronization time, as the base time (first synchronization) and written to the corresponding record in the H2 database. For modified data, the newly calculated checksum replaces the original record, and the update timestamp is the current synchronization time. For all data that has completed index updates, the base time field in the H2 database is uniformly refreshed to the end time of this incremental synchronization. All database operations are atomic through a transaction mechanism, ensuring that either all updates succeed or the system rolls back to the state before the update. This step achieves state synchronization between the database and the index system, ensuring that the base time and checksum records are accurate during the next incremental synchronization, providing a reliable basis for subsequent change judgments, and ensuring the consistency of cross-system metadata and local indexes from the data source.

[0078] Step S405: Persist the updated dual-index data to the local file again, overwriting the original file, to ensure that the persisted data is consistent with the memory index data.

[0079] After the index update is complete, the system triggers a persistence task. First, it reads the complete keyword inverted index and tag forward and reverse index data from memory and performs structured concatenation according to the defined file format. Then, it uses a file locking mechanism to prevent concurrent write conflicts, employing a "backup first, overwrite later" strategy—first generating a temporary backup of the original file (named with the current timestamp), then batch-writing the new index data to the local file, overwriting the original data. After writing, the file integrity is verified. If the verification passes, the temporary backup is deleted; if it fails, the backup file is automatically restored and an alarm is triggered. This method ensures that the persisted data is completely consistent with the in-memory index, avoiding index data loss or corruption due to system restarts or anomalies. The temporary backup mechanism further reduces data risk during the overwrite process, making the reliability of index persistence over 99.9%, ensuring long-term stable system operation.

[0080] In one embodiment of the present invention, the specific steps after parsing the multi-condition query request are as follows: Step S501: Receive a multi-condition query request sent by the client, parse the keywords, tags, data types and directories in the request, and perform format verification and legality verification. The Spring Boot Controller layer receives HTTP query requests from clients and uses a request parameter binding mechanism to parse fields such as keywords, tags, data types, and directories carried in the request. The format validation step verifies whether each field's data type is a string, whether it contains illegal special characters, and whether its length is within preset thresholds (keywords / tags up to 64 characters, data types / directories up to 32 characters). The validity validation step queries the H2 database for valid directory lists and tag entity class data to confirm the existence of the directory and whether the tag is registered. It also checks if any required query conditions (at least one keyword or tag) are missing. If the validation fails, a clear error response is returned. This design filters invalid query requests at the source, avoiding retrieval logic anomalies caused by incorrect formatting or illegal conditions, reducing invalid computation by more than 80%, and ensuring the stability and efficiency of the retrieval process.

[0081] Step S502: Based on the parsed query conditions, first filter out the set of metadata unique identifiers that meet the tag conditions through the tag forward index, then match out the set of metadata unique identifiers that meet the keyword conditions through the keyword inverted index, and further filter by data type and directory to obtain the final target metadata unique identifier set. First, based on the parsed tag conditions, the system queries the set of unique metadata identifiers associated with the tag in the tag forward index (if there are no tag conditions, the entire set of identifiers is retrieved directly). Then, based on the keyword conditions, the system queries the set of identifiers matching the keyword inverted index (if there are no keyword conditions, the preceding set is used). An intermediate set that satisfies both tag and keyword conditions is obtained through set intersection operations. Next, the system reads the metadata attributes corresponding to the intermediate set from the H2 database, performs precise filtering based on data type and directory, and removes identifiers with mismatched attributes, ultimately obtaining the target set of identifiers. This hierarchical filtering logic of "coarse screening followed by fine screening" utilizes the efficient query characteristics of dual indexes to reduce the amount of basic data, and ensures filtering accuracy through database attribute validation. It supports queries with arbitrary condition combinations, keeping the response time for multi-condition retrieval within milliseconds, achieving an efficiency improvement of over 95% compared to a full scan.

[0082] Step S503: Calculate the matching degree between the target metadata and the query conditions based on the frequency of keyword occurrence and the tightness of tag association, and sort them from high to low matching degree; A quantitative scoring rule is established: a keyword appears once in a table name and once in a field name, accumulating to obtain a keyword frequency score; manually associated tags are worth 5 points, and rule-driven associated tags are worth 2 points, accumulating to obtain a tag association score; the total matching score is the sum of the two, and if multiple keywords or tags exist, the scores are added together; based on the total score, the set of unique identifiers of the target metadata is sorted in descending order, and if the total scores are the same, they are sorted in descending order by metadata update timestamp (newest updated data takes priority). This sorting mechanism accurately reflects the fit between metadata and query requirements through multi-dimensional quantitative indicators, placing results with higher business focus (more frequent keyword appearances, manually associated tags, latest updates) at the top, significantly improving the user search experience and reducing the need to browse invalid results.

[0083] Step S504: Receive the custom number of items to be displayed per page and the current page number parameters through the pagination auxiliary tool, perform pagination processing on the sorted metadata, extract the metadata record of the corresponding page number, and complete the metadata attribute information and associated tag information from the H2 database; The system receives the number of records to display per page (default 10, maximum 50) and the current page number (default page 1, starting from 1) from the client. Using a pagination tool, it calculates the start and end positions for data truncation and segments the sorted set of unique identifiers for the target metadata. For each truncated identifier, it reads the corresponding complete attribute information from the baseline data storage table in the H2 database and queries the associated tag name set from the tag inverse index. The attribute and tag information are then linked and integrated to form a structurally complete metadata record. This pagination mechanism avoids network congestion and client parsing pressure caused by returning large amounts of data at once. The information completion step ensures that each returned record contains all the details required by the business, eliminating the need for secondary client queries and balancing response efficiency with data integrity.

[0084] Step S505: Build a RESTful interface based on Spring Boot, including a search interface, a tag creation interface, a tag query interface, a tag modification interface, and a tag deletion interface. Use standard HTTP request methods. The interface returns JSON format data, including response status code, response information, and processed metadata or tag data, for client calls.

[0085] This system utilizes Spring MVC components built on Spring Boot to construct a RESTful interface architecture. The search interface uses a GET method to receive query conditions and pagination parameters, returning a list of metadata after pagination. The tag creation interface uses a POST method to receive tag name, type, and description parameters; after successful validation, it writes the tag entity class data and updates the index. The tag query interface uses a GET method, supporting querying tag details by tag name or type. The tag modification interface uses a PUT method to receive a unique tag identifier and update parameters, updating tag information and synchronizing the index. The tag deletion interface uses a DELETE method to receive a unique tag identifier and delete the tag data and associated mappings in the index. All interfaces uniformly return JSON format data, including standard HTTP response status codes (200 Success, 400 Parameter Error, 404 Resource Not Found, etc.), response information (error description or success message), and business data (metadata list or tag details). This unified interface design solves the problem of the lack of a unified search entry point in existing technologies, supports cross-platform and cross-language calls, adapts the JSON format to various client parsing requirements, and the tag CRUD interface meets the flexible tag management needs of businesses, significantly reducing the integration cost of third-party systems and enabling convenient integration with multiple systems.

[0086] The above description is only a preferred embodiment of the present invention. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.< / string> < / string> < / string> < / string>

Claims

1. A method for cross-system metadata multi-condition retrieval and index update based on Gravitino, characterized in that, Includes the following steps: GravitinoClient calls the directory listing interface and table listing interface to collect full metadata from cross-system data sources in a hierarchical manner. After standardized encapsulation, the metadata is stored in the H2 database to form baseline data. The IK word segmenter is used to segment the table and field names. A keyword inverted index is built through hash mapping. At the same time, a forward and reverse index mapping between tags and metadata is established through tag entity encapsulation. The dual index is fully built and persisted to a local file. Based on the Spring scheduled task configuration, the incremental synchronization is triggered at a set frequency, and the table list interface and table structure description interface of GravitinoClient are called to obtain data. The fifth version of the message digest algorithm is used to calculate the checksum of the table structure. It is compared with the previous checksum stored locally to identify the data change type and update the corresponding index. Parse multi-condition query requests, perform combined queries by filtering by tags and matching by keywords, sort and paginate the results, and expose a RESTful interface through Spring Boot for invocation.

2. The method for cross-system metadata multi-condition retrieval and index update based on Gravitino according to claim 1, characterized in that, The specific steps for collecting full metadata are as follows: Start GravitinoClient and establish a connection to the cross-system data source, and confirm the API call permissions; Call the directory listing interface of GravitinoClient to obtain all cross-system directory information, and traverse them one by one by directory; For each directory, call the table list interface to obtain basic information about all tables in that directory, including table name, directory, and creation time; For each table, iterate through its field hierarchy and collect field names, data types, field comments, and modification times; The collected metadata is deduplicated and duplicate records are removed to ensure that all metadata is complete and without redundancy.

3. The method for cross-system metadata multi-condition retrieval and index update based on Gravitino according to claim 1, characterized in that, The specific steps of the standardized packaging are as follows: Define a custom metadata entity class, specifying the table name, field names, data types, field comments, directory, creation time, modification time, unique identifier, update timestamp, and base time; The collected table names and field names are standardized by converting them to lowercase to eliminate data recognition conflicts caused by differences in case. Each standardized metadata record is automatically assigned a globally unique identifier, which serves as the core identifier for subsequent index association and data traceability. Record the update timestamp of each piece of metadata and synchronize the base time of the first synchronization; The standardized and encapsulated metadata is transmitted to the H2 database for storage in a unified format.

4. The method for cross-system metadata multi-condition retrieval and index update based on Gravitino according to claim 1, characterized in that, The specific steps for storing the H2 database are as follows: Initialize the embedded H2 database, configure the data storage path, access account and permissions, and ensure data storage security; Create a baseline data storage table, and design the unique identifier number, standardized table name, standardized field name, data type, field comments, directory, creation time, modification time, update timestamp, and baseline time for each field corresponding to the metadata. The standardized and encapsulated metadata is written into the storage table one by one, and a unique identifier is established to map it to other attributes. Configure an automatic database persistence mechanism to periodically synchronize in-memory data to disk files, preventing data loss due to system restarts; Create an index for the storage table, using the unique identifier number and the directory it belongs to as index fields, to improve the efficiency of subsequent data queries.

5. The method for cross-system metadata multi-condition retrieval and index update based on Gravitino according to claim 1, characterized in that, The specific steps for constructing the keyword inverted index are as follows: Read all standardized table and field names from the H2 database and group them according to their unique metadata identifiers. The IK word segmenter is called to segment the table names and field names in each group, extracting independent keywords with actual semantic meaning and filtering out redundant characters. Initialize the hash map container, setting the key to the tokenized keywords and the value to a set of unique metadata identifiers; Each keyword is associated with its corresponding metadata unique identifier, and the unique identifier is added to the set corresponding to the keyword to form a mapping relationship between keywords and metadata; Perform integrity checks on the completed keyword inverted index to ensure that each keyword can be accurately associated with its corresponding metadata unique identifier.

6. The method for cross-system metadata multi-condition retrieval and index update based on Gravitino according to claim 1, characterized in that, The specific steps for establishing the positive and negative index mapping between tags and metadata are as follows: Define a custom tag entity class, specifying the tag name, tag type, tag description, and creation time; Initialize two hash map containers, one for building a forward index and the other for building a reverse index. The key of the forward index is the tag name, and the value is the set of associated metadata unique identifiers. The key of the reverse index is the metadata unique identifier, and the value is the set of associated tag names. The association between metadata and tags can be established through manual association or rule-driven association. Rule-driven association automatically matches the corresponding tags based on the directory and data type of the metadata. Write the relationships into two hash mapping containers respectively to complete the construction of the forward index and the reverse index; The tag entity class data is associated with the forward and reverse index mapping data for storage, ensuring the consistency between tag information and index mapping.

7. The method for cross-system metadata multi-condition retrieval and index update based on Gravitino according to claim 1, characterized in that, The specific steps for completing the full construction of the dual indexes and persisting them to local files are as follows: Define the index persistence file format, specifying the index type identifier, key-value pair separator, data record terminator, and encoding format; Write the keyword inverted index data into the local first file according to the defined format. Each index record contains keywords, a set of metadata unique identifier numbers, and the index creation time. Write the forward and reverse index data of the tags to a local second file according to the defined format. The forward index record contains the tag name and a set of metadata unique identifier numbers, and the reverse index record contains the metadata unique identifier number and a set of tag names. Enable file locking mechanisms to prevent data conflicts caused by concurrent writes from multiple threads and ensure the atomicity of index data writes; regularly back up local persistent files, generate backup files and record backup times to avoid index data loss due to file corruption.

8. The method for cross-system metadata multi-condition retrieval and index update based on Gravitino according to claim 1, characterized in that, The specific steps of incremental synchronization are as follows: Configure a timed expression based on Spring scheduled tasks to set an incremental synchronization task to be triggered every 10 minutes. After the task is triggered, the GravitinoClient calls the table list interface to obtain the basic information of all current tables, compares it with the baseline time stored in the H2 database, and filters out the tables that have been added or changed since the baseline time. For the selected tables, call the table structure description interface to obtain detailed table structure information, including field names, data types, and the latest status of field comments; Calculate the fifth version checksum of the message digest algorithm for each changed table structure and compare it with the previous checksum stored in the H2 database; The data change type is determined based on the comparison results. If the unique identifier number of the metadata does not exist, it is newly added data; if the checksum is inconsistent, it is modified data; if the unique identifier number exists locally but is not returned by the client, it is deleted data.

9. The method for cross-system metadata multi-condition retrieval and index update based on Gravitino according to claim 1, characterized in that, The specific steps for updating the corresponding index are as follows: For new data, repeat the steps of building the keyword inverted index and the tag forward and reverse indexes to generate corresponding index entries and add them to the dual index system; For modified data, delete the original corresponding index entries, rebuild the keyword inverted index and tag forward and reverse index entries for the data, and replace the original index data; For deleted data, all index entries corresponding to the data are removed from the keyword inverted index and the tag forward and reverse indexes to ensure that there is no invalid data in the index system; Update the checksum record and timestamp of the corresponding data in the H2 database, and synchronize the base time to the current synchronization time. The updated dual-index data is re-persisted to a local file, overwriting the original file, to ensure that the persisted data is consistent with the in-memory index data.

10. The method for cross-system metadata multi-condition retrieval and index update based on Gravitino according to claim 1, characterized in that, The specific steps after parsing the multi-condition query request are as follows: Receive multi-condition query requests sent by clients, parse keywords, tags, data types, and directories in the requests, and perform format validation and legality verification; Based on the parsed query conditions, the set of metadata unique identifiers that meet the tag conditions is first filtered out by the tag forward index, and then the set of metadata unique identifiers that meet the keyword conditions is matched by the keyword inverted index. The data type and the directory to which they belong are further filtered to obtain the final target set of metadata unique identifiers. The matching degree between the target metadata and the query conditions is calculated based on the frequency of keyword occurrence and the tightness of tag association, and sorted from high to low matching degree; The pagination tool receives the custom number of items to display per page and the current page number as parameters, performs pagination on the sorted metadata, extracts the metadata records for the corresponding page numbers, and completes the metadata's full attribute information and associated tag information from the H2 database. This system builds RESTful interfaces based on Spring Boot, including search, tag creation, tag query, tag modification, and tag deletion interfaces. It uses standard HTTP request methods and returns JSON data, including response status codes, response information, and processed metadata or tag data, for clients to use.

Images