Method for dynamic data catalog management and construction in data weaving environments

Abstract

The application provides a kind of dynamic data directory management and the method of being built in data weaving environment, comprising: the classification of multiple source heterogeneous raw data in data weaving environment is neat and metadata extraction is carried out, to generate structured raw data set and associated metadata table;According to structured raw data set and associated metadata table, in combination with the access frequency and associated relationship of data in data weaving environment, the initial skeleton of dynamic data directory is constructed;Based on the initial skeleton of dynamic data directory, real-time mapping logic is configured respectively for different data types in structured raw data set, to generate data-directory mapping rule base;Based on data-directory mapping rule base, real-time change monitoring is carried out to structured raw data set, and the corresponding rule in data-directory mapping rule base is triggered to execute directory node update, to generate available dynamic data directory adapted to data weaving environment.The application reduces the invalid consumption of computing resources and time, avoids unnecessary resource occupation.

Description

Technical Field

[0001] This application relates to the field of data management technology, and more specifically, to a method for dynamic data catalog management and its construction in a data weaving environment. Background Technology

[0002] As digital transformation deepens, enterprises and organizations in a data weaving environment need to integrate heterogeneous data from multiple sources, such as database tables, unstructured log files, and semi-structured API interface data, to support core business operations such as cross-data source querying, synchronization, and analysis. To avoid inefficient access caused by data fragmentation, it is necessary to systematically organize various data resources through a data catalog to clearly present data sources, relationships, and usage permissions. This has become a key requirement for ensuring data utilization efficiency and accuracy in a data weaving environment.

[0003] Currently, the data catalog construction schemes in data weaving environments mostly adopt the following approach: First, collect all heterogeneous data from multiple sources without discrimination; then, use a unified format conversion tool to force all data into a preset single structured format; subsequently, according to a manually defined fixed directory hierarchy (such as dividing the directory according to the data storage location), allocate the structured data to the corresponding directory nodes one by one; finally, use periodic tasks (such as executing once every morning) to batch update the directory content to cover historical changes in the data.

[0004] In the existing solutions mentioned above, the uniform format conversion of all data will invalidate non-core data, resulting in a waste of computing resources and time; the fixed directory hierarchy cannot adapt to the differentiated management needs of different types of data (such as structured field data and unstructured text data), which can easily lead to a mismatch between directory nodes and data characteristics; and the periodic update mechanism cannot capture real-time addition, modification or deletion operations of data in a timely manner, often resulting in a disconnect between directory content and actual data, which in turn affects the reliability of cross-source data access in the data weaving environment and makes it difficult to meet the business requirements for data timeliness and accuracy. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a method for dynamic data catalog management and its construction in a data weaving environment, thereby at least alleviating the aforementioned technical problems.

[0006] The technical solutions provided in this application are as follows:

[0007] A method for dynamic data catalog management and its construction in a data weaving environment, comprising:

[0008] Step 1: Classify and organize the multi-source heterogeneous raw data in the data weaving environment and extract metadata to generate a structured raw dataset and associated metadata table;

[0009] Step 2: Based on the structured original dataset and associated metadata tables, and combined with the access frequency and relationships of data in the data weaving environment, construct the initial skeleton of the dynamic data catalog;

[0010] Step 3: Based on the initial skeleton of the dynamic data catalog, configure real-time mapping logic for different data types in the structured raw dataset to generate a data-catalog mapping rule base;

[0011] Step 4: Based on the data-directory mapping rule base, monitor the changes of the structured raw dataset in real time, and trigger the corresponding rule execution in the data-directory mapping rule base to update the directory nodes, so as to generate a usable dynamic data directory that is adapted to the data weaving environment.

[0012] The technical solution in this application has the following technical advantages:

[0013] This application classifies and organizes multi-source heterogeneous raw data in the data weaving environment and extracts metadata, instead of indiscriminately collecting all data and forcibly converting it into a single structured format. This processing can prioritize the structuring of core data and the extraction of metadata, reducing invalid format conversion operations on non-core data. Compared with the traditional full-processing method, it can reduce the ineffective consumption of computing resources and time and avoid unnecessary resource occupation.

[0014] This application constructs an initial skeleton of a dynamic data directory based on the structured raw dataset and associated metadata tables, combined with the access frequency and relationships of the data, rather than manually defining fixed levels according to storage location, so that the directory hierarchy can reflect the actual usage frequency and the tightness of the data relationships. Subsequently, based on this initial skeleton, real-time mapping logic is configured for different data types in the structured raw dataset to further adapt to the feature differences of various types of data. Compared with the traditional fixed directory hierarchy, it can form a directory structure that is more in line with the data characteristics and actual usage scenarios, alleviating the problem of mismatch between directory nodes and data characteristics.

[0015] In this application, the structured raw dataset is monitored for changes in real time based on a data-directory mapping rule base, and the corresponding rule is executed to update the directory nodes, instead of relying on periodic batch updates such as once a day. This can capture data addition, modification or deletion operations in a timely manner, so that the content of the dynamic data directory is kept in a more synchronized state with the actual state of the structured raw dataset. Compared with the traditional periodic update method, it can improve the consistency between the directory content and the actual data, reduce the situation of the directory and the data being out of sync, and thus improve the reliability of cross-source data access in the data weaving environment, which is more in line with the business's need for data timeliness. Attached Figure Description

[0016] Figure 1This is a flowchart illustrating a method for dynamic data catalog management and its construction in a data weaving environment, as described in an embodiment of this application.

[0017] Figure 2 This is a schematic diagram of the structure of a device for dynamic data catalog management and construction in a data weaving environment, according to an embodiment of this application. Detailed Implementation

[0018] Figure 1 This is a flowchart illustrating a method for dynamic data catalog management and its construction in a data weaving environment, as described in an embodiment of this application. Figure 1 As shown, a method for dynamic data catalog management and its construction in a data weaving environment includes:

[0019] Step 1: Classify and organize the multi-source heterogeneous raw data in the data weaving environment and extract metadata to generate a structured raw dataset and associated metadata table;

[0020] Step 2: Based on the structured original dataset and associated metadata tables, and combined with the access frequency and relationships of data in the data weaving environment, construct the initial skeleton of the dynamic data catalog;

[0021] Step 3: Based on the initial skeleton of the dynamic data catalog, configure real-time mapping logic for different data types in the structured raw dataset to generate a data-catalog mapping rule base;

[0022] Step 4: Based on the data-directory mapping rule base, monitor the changes of the structured raw dataset in real time, and trigger the corresponding rule execution in the data-directory mapping rule base to update the directory nodes, so as to generate a usable dynamic data directory that is adapted to the data weaving environment.

[0023] Optionally, step 1, classifying and organizing the multi-source heterogeneous raw data in the data weaving environment and extracting metadata to generate a structured raw dataset and associated metadata table, specifically includes the following steps:

[0024] Step 11: Perform format parsing and redundant data removal on the multi-source heterogeneous raw data in the data weaving environment to generate preliminarily cleaned multi-source heterogeneous data;

[0025] Step 12: After the initial cleaning, the multi-source heterogeneous data is grouped according to data type, and each group is labeled with the corresponding data source identifier to generate classified multi-source heterogeneous data groups.

[0026] Step 13: Perform structured processing on different types of data in the multi-source heterogeneous data set to generate a single-type structured data subset;

[0027] Step 14: Extract the metadata information of each single-type structured data subset to generate structured data and corresponding metadata records;

[0028] Step 15: Group the structured data according to business logic to generate a structured raw dataset; at the same time, organize the corresponding metadata records according to metadata fields.

[0029] Preferably, the specific implementation process of step 11 is as follows: For the mixed multi-source heterogeneous raw data in the general data weaving environment (such as structured database tables of enterprise transaction systems, semi-structured JSON logs of user behavior analysis, and unstructured scanned documents of product document management), a multimodal format parsing engine is first loaded. This engine has three built-in general-purpose parsers, and the three parsers directly connect to the raw data to perform targeted processing, ensuring that the parsing results can directly support subsequent redundancy removal: First, the structured data parser establishes a link with the transaction system database through a general database connection pool (supporting JDBC / ODBC protocol) and reads the structure definition of the target table (such as " The order table contains a primary key field "Order ID", a numeric field "Transaction Amount", and field constraints (such as "Order ID" being unique and not null, and "Transaction Amount" being greater than 0). The read field data and table structure information are encapsulated into a structured parsing result. Secondly, a semi-structured data parser uses a general-purpose markup parser (compatible with JSON / XML formats) to process user behavior logs. Taking JSON logs as an example, it first identifies the nesting level of key-value pairs (such as "user→behavior→click→page"), and then decomposes the nested structure according to the "parent key-child key-field value" format (such as "user_behavior_click_page"). The first part uses a product details page (ge = product details page) to generate a semi-structured parsing result; the second part uses an unstructured data parser to first perform general OCR text recognition on the scanned product documents (ensuring high recognition confidence), and then call a general document template library (including common templates such as contracts and instructions). It locates key fields such as "document number," "effective date," and "product model" according to the template, and encapsulates the field values ​​and recognition confidence into an unstructured parsing result. Next, the three types of parsing results are integrated to form a multi-source parsing dataset; then, a general redundancy judgment matrix is ​​constructed (rows: structured / semi-structured / unstructured data types; columns: duplicate data / null value percentage / invalid format; elements: general scenarios). Thresholds are set, such as the unique identifier field consistency threshold for structured data ("unique identifier field is consistent and data update time difference is less than a set short period"), and the file hash value consistency threshold for unstructured data ("file hash value is consistent and identification confidence is higher than a set value"). Based on the redundancy judgment matrix, redundancy filtering is performed on the multi-source parsing dataset: duplicate order records, behavior logs with a null value ratio exceeding a set proportion, and scanned documents with corrupted format are deleted to generate pre-cleaned multi-source heterogeneous data. This application differs from traditional designs that separate parsing and cleaning. This technology avoids idle parsed data through a parser-parsing result-redundancy matrix, and the universal adaptability design of the three types of parsers can cover most data weaving scenarios.

[0030] Preferably, in enterprise multi-system data integration scenarios (such as data linkage between transaction systems, CRM systems, and supply chain systems), step 12 is specifically implemented as follows: First, the preliminarily cleaned multi-source heterogeneous data is loaded. This multi-source heterogeneous data includes parsed structured transaction data, semi-structured user behavior data, and unstructured document data. The general business classification engine is invoked, and first-level grouping is performed according to the "core business domain": the business identifiers implicit in the multi-source heterogeneous data (such as transaction data containing the "order" tag and user data containing the "customer" tag) are read, and the multi-source heterogeneous data is classified into first-level business domains such as "transaction management", "customer service", and "supply chain collaboration" to form first-level business group data; then, second-level grouping is performed on the first-level group data according to the "data storage format": the structured database tables in the "transaction management" domain are classified into the "structured data group", the JSON format transaction logs are classified into the "semi-structured data group", and the PDF format order contracts are classified into the "unstructured data group", generating classified multi-source heterogeneous data groups (each group of data is accompanied by dual tags of business domain and storage format). Next, the general source identifier generator is launched to assign a four-layer exclusive identifier to each group of multi-source heterogeneous data (system code: transaction system = TS001 / CRM system = CS002 / supply chain system = SS003; module code: order module = OM01 / customer module = CM02; collection node: database server = DB01 / file server = FS01; timestamp: accurate to the second). The identifier format is "system code-module code-collection node-timestamp". The identifier is embedded in the metadata header of each group of multi-source heterogeneous data (add "source_id" field to structured data, add "source" tag to semi-structured data, and write file attributes to unstructured data) to generate the classified multi-source heterogeneous data group.

[0031] Preferably, in the specific technical implementation of step 13: First, load the categorized data groups with source identifiers, and assign dedicated general processors according to the "storage format" tags within the group to form a one-to-one correspondence between "groups" and "processors," ensuring that each group of categorized data can be processed in a targeted manner: First, for "structured data groups" (such as the order table of a transaction system), call the general field attribute completion processor, read the general business semantic dictionary (such as the default unit of the "transaction amount" field being the set currency unit, and the enumeration value of "order status" being "pending payment / paid / completed"), complete the missing field attributes in the data (such as adding unit constraints to "transaction amount" and adding enumeration value constraints to "order status"), and generate complete structured data with attributes; Second, for "semi-structured data groups" (such as user behavior JSON logs of a CRM system), The process involves three main steps: First, a general nested structure flattening processor is invoked to expand the nested path according to the "business module-field" rule (e.g., "customer-contact-phone number"). Simultaneously, a path mapping table is generated (recording the correspondence between "customer / contact / phone → customer_contact_phone number") to avoid loss of field meaning and generate flattened semi-structured data. Second, for "unstructured data groups," a general processor is invoked based on data type: text data (e.g., product manuals) uses a general NLP structured processor to extract key entities such as "document number," "product model," and "release date" using a pre-trained general text model, encapsulating them into structured records. Media data (e.g., product images) uses a general media feature structured processor to extract features such as resolution, shooting time, and storage format, generating media feature structured data. Finally, the results of these three processing methods are integrated and categorized according to "business domain-storage format" (e.g., "transaction management-structured") to generate single-type structured data subsets (each subset contains a data ontology and processing logs, with the logs recording the processor type, processing time, and semantic dictionary version used).

[0032] Preferably, the specific implementation process of step 14 is as follows: First, load a single-type structured data subset (such as "Transaction Management - Structured Subset" or "Customer Service - Structured Subset"). This single-type structured data subset contains data ontology, business domain tags, and processing logs, providing complete input for metadata extraction and avoiding incomplete metadata due to missing input. The system invokes a general metadata layered extraction engine to extract data from single-type structured data subsets along three dimensions: "basic," "business," and "technology." Each dimension's extraction relies on specific information from the single-type structured data subset, ensuring logical consistency: First, it extracts general basic metadata by reading the data ontology of the single-type structured data subset (e.g., a "transaction subset" containing a set number of records and fields) to obtain basic information such as "data volume," "field name," "data type," and "field length." Second, it extracts general business metadata by reading the "business domain tags" (e.g., "transaction management") of the single-type structured data subset and combining them with general business rules (e.g., "order ID" associated with the "order creation" business process, "customer ID" associated with the "customer filing" business process) to obtain information such as "field business meaning," "associated business process," and "business priority." Third, it extracts general technical metadata by reading the processing logs of the single-type structured data subset (e.g., processor version, data storage path, compression format) to obtain information such as "processing component version," "physical storage location," "data compression method," and "access protocol." Next, a general metadata association matrix is ​​constructed (rows: single-type structured data subsets, such as "transaction management subset"; columns: metadata dimensions + sub-items, such as "basic metadata - data volume" and "business metadata - associated business processes"; elements: specific extraction results, such as "set quantity of records" and "order creation process"). This general metadata association matrix not only integrates the three-dimensional extraction results but also provides a direct basis for the business grouping and metadata standardization in subsequent step 15. Step 15 will group business logic based on "business metadata - associated business processes" in the matrix and standardize metadata format based on "basic metadata - field name / data type". Then, a structured data ontology (complete data after attribute completion and format standardization) and a corresponding metadata record (containing three-dimensional metadata information and a unique subset identifier) ​​are generated for each single-type structured data subset. The two are bound through the unique subset identifier to ensure a one-to-one correspondence between data and metadata, thereby generating structured data and corresponding metadata records.

[0033] Preferably, in an enterprise-level general data weaving scenario, the specific implementation process of step 15 is as follows: First, load the structured data ontology set and the associated metadata record set. The two are associated through a subset unique identifier, serving as the dual input for business logic grouping and metadata regularization, ensuring that there is no data loss in subsequent processing. The general business logic grouping engine is invoked, and based on the "Business Metadata - Related Business Process" field in the associated metadata record set, targeted grouping is performed on the structured data ontology set: data with "Related Business Process = Order Creation" is grouped into the "Transaction Management - Order Processing" group, and data with "Related Business Process = Customer Profile Creation" is grouped into the "Customer Service - Customer Management" group; for data without a clearly defined associated business process (such as temporarily collected market research data), semantic matching is performed between the "Business Metadata - Field Business Meaning" in the associated metadata record set and the general business grouping rules (such as associating "Research Object ID" with the "Market Analysis - Research Management" group; if the similarity is higher than a set threshold, it is automatically grouped into the corresponding group; if it is lower than the threshold, it is marked as "Data for Business Groups to be Confirmed"), generating business grouped structured data (stored according to the path "First-level Business Domain - Second-level Business Process - Data Subset", such as " / Transaction Management / Order Processing / Order Master Data Subset"), which is the structured raw dataset. Simultaneously, the general metadata straightener is invoked, and based on the unified template preset by the general data metadata specification (including "unique data identifier, business group path, data subset name, data volume, data update time, field list and constraints, associated business process, physical storage path, and data access protocol"), the associated metadata record set is straightened: missing fields are filled in (such as filling in the system default storage directory for the "physical storage path" field), and the format is unified (such as converting "data update time" to a general time format and standardizing "field constraints" according to the "field name: constraint type" format), generating a straightened associated metadata table (each row of records contains all fields of the unified template, with no missing fields). Next, a consistency check is performed on the structured original dataset and the normalized associated metadata table: by matching data unique identifiers, it is checked whether the field list of each subset in the structured original dataset is consistent with the "field list and constraints" of the corresponding record in the normalized associated metadata table (e.g., if the "order master data subset" contains the "order ID" and "transaction amount" fields, the corresponding record in the metadata table must also contain these two fields and constraints), ensuring that the field dimensions of the data and metadata are consistent, and avoiding data-metadata mismatch during subsequent directory construction. Finally, the structured original dataset and the normalized associated metadata table are output.

[0034] Optionally, step 2, based on the structured original dataset and associated metadata tables, and combined with the access frequency and relationships of data in the data weaving environment, constructs the initial skeleton of the dynamic data catalog, specifically including the following steps:

[0035] Step 21: Extract data attributes from the structured raw dataset and associated metadata table, and simultaneously perform access frequency statistics on the data weaving environment access logs to generate an integrated table of data attributes and access frequencies.

[0036] Step 22: Perform field association analysis on the data attribute and access frequency integration table, and label the association strength values ​​of the associated datasets in the structured original dataset according to the analysis results to generate a data association table;

[0037] Step 23: Based on the data attribute and access frequency integration table and the data association table, divide the structured original dataset into directory levels to generate a directory level division scheme;

[0038] Step 24: Based on the directory hierarchy partitioning scheme, create index nodes for each level of the structured original dataset, each containing a unique data identifier, data attribute reference, and relationship identifier, to generate an initial skeleton of a dynamic data directory containing directory hierarchy, data index nodes, and relationship identifier.

[0039] Preferably, the specific implementation process of step 21 is as follows: For the structured original dataset (such as "customer basic dataset", "order transaction dataset", and "product information dataset" in e-commerce scenarios) and the normalized associated metadata table (including field attributes, business domains, and storage paths of each dataset), firstly, start the data attribute joint extraction engine. Based on the structured original dataset and supplemented by the associated metadata table, perform bidirectional attribute extraction: read physical attributes such as "data subset name and field list (including field name, data type, and field constraints)" from the structured original dataset; read business attributes such as "belonging business domain, associated business process, and business priority" from the associated metadata table, and bind the two types of attributes according to "data unique identifier" to form a data attribute list. Next, the access frequency statistics engine is invoked to load the historical access logs of the data weaving environment (including access dataset ID, access time, and access user role). Statistics are then grouped and analyzed by "unique data identifier": the daily average number of accesses, peak access periods (e.g., the "order transaction dataset" averages 500 accesses per day, peaking between 10:00 and 12:00), and the percentage of access roles (e.g., operations roles account for 60%, and R&D roles account for 30%) are calculated, generating an access frequency statistics list. Finally, based on the data attribute list and the access frequency statistics list, an integrated table of data attributes and access frequencies is constructed—the rows of the table represent each subset of data in the structured original dataset (e.g., row 1 is the "customer basic dataset"), and the columns are divided into three categories: "physical attributes," "business attributes," and "access frequency" (including sub-columns such as "physical attributes - field list" and "access frequency - daily average number of accesses"), with the corresponding extracted results filled at the intersections (e.g., row 1 - "access frequency - daily average number of accesses" is "300 times").

[0040] Preferably, in the e-commerce data weaving scenario (integrating customer, order, product, and logistics data), step 22 is specifically implemented as follows: First, load the data attribute and access frequency integration table, which already contains the attributes and access characteristics of each dataset, as the core input for association analysis. The multi-dimensional field association analysis engine is launched to perform field-level association probing on all data subsets in the data attribute and access frequency integration table. The results are as follows: First, identify "fields shared across subsets" (e.g., "Customer ID" exists in "Customer Basic Dataset", "Order Transaction Dataset", and "After-sales Work Order Dataset"), and record the consistency of the field name and data type of the shared fields (e.g., "Customer ID" is a string type in all three datasets, and consistency is high if the length is the same); Second, analyze "business process dependencies" (e.g., the "Payment Completed" status in the "Order Transaction Dataset" depends on the "Payment Successful" record in the "Payment Dataset", and the "Logistics Dataset" depends on the "Order Generation" record in the "Order Dataset"), and record the dependency direction according to "Upstream Dataset → Downstream Dataset"; Third, count the "data co-occurrence frequency" (e.g., the number of times "Customer Basic Dataset" and "Order Transaction Dataset" appear together in the same query request, as a percentage of the total number of queries). Based on the above detection results, a correlation strength calculation matrix is ​​constructed (rows and columns represent data subsets, such as row 1 being "customer basic dataset" and column 2 being "order transaction dataset", with matrix elements representing correlation strength values ​​ranging from 0 to 1) to characterize the correlation relationships of all data subsets. The matrix is ​​weighted as follows: business process dependency accounts for 40% (strong dependency such as "order → logistics" assigned a value of 0.8), data co-occurrence frequency accounts for 35% (co-occurrence rate of 80% assigned a value of 0.8), and field consistency accounts for 25% (complete consistency assigned a value of 1.0). The correlation strength value is obtained by weighted calculation (e.g., "customer-order" correlation strength = 0.4 × 0.9 + 0.35 × 0.85 + 0.25 × 1.0 = 0.89). Finally, the correlation relationships of all data subsets (including associated subset IDs, common fields, correlation strength values, and dependency directions) are organized in the format of "data subset - associated subset - correlation details" to generate a data correlation table. This technology differs from traditional association analysis that only matches field names. It calculates association strength through multiple dimensions such as "business + co-occurrence + consistency", making the association relationship more in line with the business flow logic of e-commerce and avoiding misjudging field associations with low business value (such as "creation time" field co-occurrence but weak business association, resulting in a low strength value).

[0041] Preferably, in the specific technical implementation of step 23: First, the data association table is loaded and associated with the data attribute and access frequency integration table through a "data unique identifier," serving as the dual inputs for directory hierarchy division. The dynamic hierarchy division engine is then started, setting the division logic of "access frequency priority + association strength coordination": First, based on the "daily average number of accesses" in the data attribute and access frequency integration table, data subsets are classified—the top 30% of accesses are defined as "high-frequency access data" (such as "order transaction dataset" and "real-time inventory dataset" in e-commerce scenarios), 30%-70% are defined as "medium-frequency access data" (such as "customer base dataset" and "after-sales work order dataset"), and the bottom 30% are defined as "low-frequency access data" (such as "historical order archive dataset" and "annual product report dataset"), thus initially determining the vertical hierarchy basis of "high frequency → medium frequency → low frequency." The second step involves horizontal clustering of subsets of data with similar frequencies based on the "association strength value" in the association table. Subsets with an association strength value ≥ 0.7 are grouped into "strongly associated clusters" (e.g., the "order transaction dataset" and "customer base dataset" have an association strength of 0.89 and are grouped into the same cluster; the "real-time inventory dataset" and "product information dataset" have an association strength of 0.82 and are grouped into another cluster), subsets with an association strength value between 0.3 and 0.7 are grouped into "weakly associated clusters," and subsets with an association strength value < 0.3 are grouped into separate clusters. This ensures high business relevance of subsets within the same cluster and reduces cross-cluster access costs. The third step involves adjustments based on business domains. If a "strongly associated cluster" crosses business domains (e.g., the "logistics tracking dataset" belongs to the "supply chain domain" but has an association strength of 0.85 with the "order transaction dataset" (transaction domain), the "logistics tracking dataset" is adjusted to the "association sub-layer" under the "transaction domain" level to avoid the fragmentation of strongly associated data due to business domain barriers. Finally, the output directory hierarchy scheme clearly defines the vertical structure as "top layer (high-frequency, strongly correlated clusters) - middle layer (medium-frequency clusters + high-frequency, weakly correlated clusters) - bottom layer (low-frequency data)," as well as the horizontal cluster division within each layer (e.g., the top layer includes "transaction core cluster" and "product core cluster"). It also labels the data subset IDs and relationships within each cluster. This technology differs from the traditional static hierarchy division based solely on "business domain - subdomain." Through a dynamic logic of "access frequency determining the vertical hierarchy, correlation strength determining the horizontal hierarchy, and business domain as a supplement," the directory hierarchy better aligns with actual data access habits. High-frequency, strongly correlated data can be accessed through fewer layers, improving query efficiency.

[0042] Preferably, the specific implementation process of step 24 is as follows: First, load the directory hierarchy partitioning scheme, which clearly defines the hierarchical structure, cluster partitioning, and data subset association relationships, serving as the basis for index node creation and initial skeleton generation. Start the directory index node generation engine to create structured index nodes for each data subset in each level and each cluster of the directory hierarchy partitioning scheme: Each structured index node contains three types of core information—first, a "unique data identifier" (such as "order_001", which corresponds one-to-one with the subset ID in the structured original dataset, ensuring that the node can accurately point to the data source); second, a "data attribute reference" (by associating the core attributes of the data subset through the fields of the associated metadata table, such as the "order transaction dataset" node referencing field names and data types such as "order ID, customer ID, transaction amount, payment status", avoiding the disconnect between the node and the data attributes); and third, a "relationship identifier" (recording the IDs of other index nodes associated with this structured index node and the association strength value, such as the "order transaction node" associating with "customer basic node (strength 0.89)" and "logistics tracking node (strength 0.82)", so that the structured index node carries its own association link information). Next, a directory hierarchy topology is constructed: structured index nodes are arranged in the order of "top-middle-bottom" according to the partitioning scheme, and grouped by cluster within the same level (e.g., the top-level "Transaction Core Cluster" includes "Order Transaction Nodes," "Customer Basic Nodes," and "Payment Nodes"). Visual association lines are established between structured index nodes using "relationship identifiers" (e.g., a solid line connects "Order Transaction Nodes" and "Customer Basic Nodes," with an annotation strength of 0.89). Then, a skeleton integrity check is initiated: the completeness of the three types of core information for each structured index node is checked (e.g., nodes without "relationship identifiers" need to be confirmed as independent datasets), the hierarchical topology is consistent with the partitioning scheme (e.g., whether high-frequency data is all at the top level), and whether the attributes referenced by the structured index nodes can be correctly associated with the associated metadata table (to avoid attribute reference failure). For structured index nodes that fail the check (e.g., attribute reference failure), the associated metadata table is reread to supplement the information. Finally, the validated directory hierarchy topology and all structured index nodes are integrated to generate a dynamic initial skeleton of the data directory, containing directory hierarchy, data index nodes, and relationship identifiers. This skeleton presents both a static structure of "hierarchical storage" and dynamic information about "inter-node relationships." This technology differs from traditional simple skeletons that only contain "hierarchy-node-data source." By using "relationship identifiers + attribute references," the initial skeleton possesses dual capabilities of "location + association," avoiding the need to reread data sources or metadata tables to supplement information later. This forms a logical closed loop of "partitioning scheme - index nodes - skeleton - subsequent steps."

[0043] Optionally, step 3, based on the initial skeleton of the dynamic data catalog, configures real-time mapping logic for different data types in the structured raw dataset to generate a data-catalog mapping rule base, specifically including the following steps:

[0044] Step 31: Perform correlation analysis on the initial skeleton of the dynamic data catalog and the structured original dataset, extract the type identifier of each data in the structured original dataset, and match the corresponding index node in the initial skeleton of the dynamic data catalog to generate a data type-catalog node correspondence table.

[0045] Step 32: Based on the data type-directory node correspondence table, configure real-time mapping logic for different data types to generate a single-type real-time mapping logic list;

[0046] Step 33: Add data change triggering conditions and execution priorities to each single-type real-time mapping logic list to generate a data-directory mapping rule base.

[0047] Preferably, in enterprise multi-type data weaving scenarios (such as integrating order field-type structured data, customer feedback text-type unstructured data, and logistics node-type semi-structured data), step 31 is specifically implemented as follows: First, the generated dynamic data catalog initial skeleton and the structured raw dataset generated in step 15 are combined. The initial skeleton includes "order management index node", "customer feedback index node", and "logistics tracking index node" (each node's "data attribute reference" is marked with format characteristics, such as "order management index node" marked "structured field type, including fixed fields such as order ID and transaction amount", and "customer feedback index node" marked "unstructured text type, including semantic fields such as feedback content and sentiment tendency"). The structured raw dataset includes "order field type subset", "customer feedback text subset", and "logistics node type subset" (each subset is accompanied by metadata records, including information such as "data storage format" and "content structure description").

[0048] Next, the "Data-Node Association Analysis Engine" is initialized. First, the built-in type identifier extraction module of the engine is called, and the metadata records of each subset of the structured original dataset are used as the processing objects: For the "Order Field Type Subset", the "Data Storage Format = MySQL Table" and "Content Structure Description = Fixed Field Structure" in the metadata are read, and its data type identifier is marked as "struct_field" (field type); For the "Customer Feedback Text Subset", the "Data Storage Format = TXT File" and "Content Structure Description = No Fixed Field, Plain Text Content" in the metadata are read, and its data type identifier is marked as "unstruct_text" (text type); For the "Logistics Node Type Subset", the "Data Storage Format = JSON File" and "Content Structure Description = Nested Path Containing 'Logistics Order → Transportation Node → Location'" in the metadata are read, and its data type identifier is marked as "semi_struct_node" (node ​​type); The "Data Subset ID, Data Type Identifier, and Type Determination Basis (Metadata Field)" are integrated to generate a "Data Subset-Type Identifier" lookup table.

[0049] Then, the engine's association analysis matrix construction module is invoked to construct a blank "index node-data subset" association analysis matrix (rows: index nodes, such as "order management index node"; columns: data subsets, such as "order field type subset"; matrix elements reserve three calculation positions for "field overlap, type matching, and comprehensive matching").

[0050] Next, the matrix element calculation process begins: First, the field overlap calculation submodule is called to compare the field list of the index node's "data attribute reference" with the core field list of the data subset metadata, calculating the "field overlap" (e.g., the "order management index node" has 2 fields, which overlap with the "order field type subset" by 2 fields, resulting in a field overlap of 100%), and the result is entered into the "field overlap" field of the matrix. Second, the type matching calculation submodule is called, using the "data subset-type identifier" lookup table as the key input, reading the "data type identifier" of each data subset in the table, and matching it with the format annotation of the corresponding index node's "data attribute reference" (e.g., the "struct_field" of the "order field type subset" matches the "structured field type" of the "order management index node"). If the labels are consistent (e.g., "unstruct_text" in "Customer Feedback Text Subset" and "Structured Field Type" in "Order Management Index Node" have inconsistent labels, type matching degree = 0), the type matching degree result is filled into the "Type Matching Degree" position of the matrix. The third step is to call the comprehensive matching degree calculation submodule, and calculate the comprehensive matching degree based on the weight of "Field Overlap 40% + Type Matching Degree 60%" combined with the two values ​​already filled in the matrix (e.g., the comprehensive matching degree of "Order Management Index Node - Order Field Type Subset" = 100% × 0.4 + 100% × 0.6 = 100%; the comprehensive matching degree of "Order Management Index Node - Customer Feedback Text Subset" = 0 × 0.4 + 0 × 0.6 = 0), and fill the result into the "Comprehensive Matching Degree" position of the matrix.

[0051] Then, the engine's matching result filtering module is called, and a comprehensive matching degree threshold (e.g., 70%) is set. Based on the comprehensive matching degree value of the association analysis matrix, "index node-data subset" pairs with matching degrees exceeding the threshold are filtered out (e.g., "order management index node-order field type subset", "customer feedback index node-customer feedback text subset", "logistics tracking index node-logistics node type subset"). At the same time, the "data subset-type identifier" lookup table is called again to read the "data type identifier" of these filtered subsets and supplement the "data type identifier" information for each pair of matching results (e.g., supplement the "order node-order subset" with the "struct_field" identifier).

[0052] Finally, the matching results are grouped according to "data type identifier" (e.g., the "struct_field" group contains order-related matching pairs, and the "unstruct_text" group contains customer feedback-related matching pairs). Within each group, information such as "data type identifier, directory node ID, directory node format label, associated data subset ID, data subset type identifier, comprehensive matching degree, and list of overlapping fields" are compiled to generate a data type-directory node correspondence table.

[0053] This technology differs from traditional manual matching methods by using a "data subset - type identifier" lookup table throughout the entire process of "type matching degree calculation → result type supplementation → final table grouping." This avoids cross-type mismatches during the calculation stage (e.g., text data is excluded due to a type matching degree of 0), and ensures that data of the same type is categorized with directory nodes during the organization stage. This forms a complete logical chain of "type identifier extraction → matrix calculation application → result grouping basis," ensuring that subsequent step 32 can accurately configure the mapping logic according to data type.

[0054] Optionally, in step 32, when configuring real-time mapping logic for different data types according to the data type-directory node correspondence table to generate a single-type real-time mapping logic list, for "field-type" structured data, configure mapping logic that precisely matches field names with directory node attribute names; for "text-type" unstructured data, configure mapping logic that extracts core keywords and matches them with directory node tags; and for "node-type" semi-structured data, configure mapping logic that matches data node paths with directory node hierarchical paths to generate a single-type real-time mapping logic list.

[0055] Preferably, the specific implementation process of step 32 is as follows: First, load the data type-directory node correspondence table. This table is divided into three groups according to "struct_field (field type)", "unstruct_text (text type)" and "semi_struct_node (node ​​type)". Each group contains core information such as "directory node ID, node attribute / tag / path information, associated data subset ID, and overlapping field / feature list", which serve as the baseline input for the mapping logic configuration.

[0056] For the "struct_field (field type)" data group, a field-level precise mapping engine is activated: It extracts the attribute reference list of all "directory nodes" (e.g., "order ID, transaction amount, payment status" for the "order management node") and the field list of "related data subsets" (e.g., "order number, transaction amount, payment status" for the "order subset") from the data type-directory node correspondence table. Format differences are eliminated through string standardization (consistent case sensitivity, removal of special characters). Next, a bidirectional mapping dictionary of "field name - attribute name" is established. For overlapping fields (e.g., "order ID" and "order number" are consistent after standardization), a "direct mapping" rule is set. For non-overlapping but semantically related fields (e.g., "transaction amount" and "total order amount"), "semantic association annotations" are added (the association is based on the business description in the metadata). Simultaneously, field type validation rules are configured (e.g., integer fields can only be mapped to integer attributes), generating a real-time field mapping logic body containing "source field name, target attribute name, mapping method (direct / semantic association), type validation rules, and effective field range."

[0057] For the "unstruct_text (text type)" data set, the text semantic mapping engine is activated: It extracts the "tag set of directory nodes" (e.g., the tags "product quality, logistics service, customer service attitude" for the "customer feedback node") and the "text feature description of associated data subsets" (e.g., "containing product reviews, logistics complaints, service suggestions, etc." for the "customer feedback subset") from the data type-directory node correspondence table; it then calls a pre-trained domain-specific word segmentation model (e.g., a dedicated word segmenter for e-commerce) to extract keywords from historical text data, generating a "high-frequency semantic keyword library" (e.g., the tag "product quality" corresponds to "damaged"). Keywords such as "fault" and "durable" are used to select keywords. A "keyword-tag" matching rule is configured: when the overlap rate between the extracted keywords and the associated keyword library is ≥60%, a successful match is determined. A "multi-tag weight allocation" mechanism is also set (tag weights are allocated based on keyword frequency; for example, if "slow logistics" appears 3 times and "poor customer service" appears 2 times in the text, then the "logistics service" tag weight is 60% and the "customer service attitude" tag weight is 40%). A text-based real-time mapping logic body is generated, containing "tag set, associated keyword library, matching threshold, weight allocation rule, and text preprocessing method (stop word removal, word segmentation)".

[0058] For the "semi_struct_node (node ​​type)" data group, the path hierarchy mapping engine is activated: The hierarchical path of the directory nodes (e.g., the "logistics order → transportation node → location information" hierarchy for "logistics tracking node") and the node path of the associated data subset (e.g., the "logistics → transport → location" JSON path for "logistics subset") are extracted from the data type-directory node correspondence table. The two paths are then parsed in segments. The directory hierarchy path is split into a three-level structure of "top-level node - middle-level node - bottom-level node" (e.g., "logistics order" is the top level, "transport node" is the middle level, and "location information" is the bottom level). The data node path is split into a three-level structure of "root node - parent node - child node" (e.g., "logistics" is the root node, "transport" is the parent node, "l..." is the child node). (where "ocation" is a child node); establish a "path segmentation mapping matrix"—the matrix rows represent the segments of the directory hierarchy path (top, middle, bottom), the columns represent the segments of the data node path (root node, parent node, child node), and the intersections are filled with "mapping confidence" (e.g., the mapping confidence of "logistics" and "logistics" is 90%, based on the description of "logistics corresponds to logistics order" in the metadata); configure "multi-level matching rules": when the mapping confidence of at least two levels in the three-level segments is ≥70%, the path matching is considered successful, and a "dynamic path completion" mechanism is set (for missing segments in the data node path, the corresponding segments of the directory hierarchy path are automatically referenced to complete them); generate a node-type real-time mapping logic body, which contains "directory hierarchy path segments, data node path segments, mapping confidence matrix, multi-level matching threshold, and path completion rules".

[0059] Finally, the real-time mapping logic bodies of field-type, text-type, and node-type data types are integrated according to data type. Each logic body is associated with a corresponding data type identifier, directory node ID, and data subset ID, forming a single-type real-time mapping logic list. In this list, field-type logic ensures the accurate correspondence between structured data and directory attributes, text-type logic solves the semantic association problem of unstructured data, and node-type logic adapts to the hierarchical characteristics of semi-structured data. This technology differs from the traditional single-mapping logic design by achieving differentiated mapping of multiple data types through a "type-specific engine + structured configuration logic body": the semantic association annotation of field-type data solves the problem of heterogeneous field names, the weight allocation mechanism of text-type data avoids misjudgment of single labels, and the path segmentation matrix of node-type data adapts to changes in nested structures. All three together improve the mapping adaptability between data and directory nodes.

[0060] Optionally, step 33, supplementing each single-type real-time mapping logic list with data change triggering conditions and execution priorities to generate a data-directory mapping rule base, specifically includes the following steps: integrating the single-type real-time mapping logic lists, supplementing each mapping logic with data change triggering conditions including data addition triggering, data modification triggering, and data deletion triggering, and execution priorities sorted by data access frequency, and storing them according to data type to generate a data-directory mapping rule base containing data type, mapping logic, triggering conditions, and execution priorities.

[0061] Preferably, the specific implementation process of step 33 is as follows: First, load a single-type real-time mapping logic list. This list contains three types of real-time mapping logic bodies: field type, text type, and node type. Each logic body is associated with a corresponding data type identifier, directory node ID, data subset ID, and mapping rule details (such as "source field-target attribute mapping relationship" for field type and "keyword-tag matching threshold" for text type). This serves as the basic input for supplementing triggering conditions and priorities.

[0062] The trigger condition configuration engine is activated. For each real-time mapping logic body in the single-type real-time mapping logic list, trigger conditions are configured differently according to data type: For field-type mapping logic bodies, the "Field Operation Log" description in the metadata of the associated data subset (e.g., "Order Subset" includes "Order Status Modification Record" and "Amount Update Timestamp") is read, and the "Data Modification Trigger" is configured as "Triggered when the change in the source field value exceeds 10% of the field length or when a key business field (e.g., payment status) changes," the "Data Addition Trigger" is configured as "Triggered when every 10 new records are added or a single record contains 3 or more non-empty key fields," and the "Data Deletion Trigger" is configured as "...". The following conditions are triggered: For text-based mapping logic entities, the "data modification trigger" is configured based on text feature descriptions as "triggered when the number of newly added keywords in the text content exceeds 20% of the original keyword library," and the "data addition trigger" is configured as "triggered when a single text entry exceeds 50 characters in length and contains at least two directory node tag keywords." For node-based mapping logic entities, the "data modification trigger" is configured based on the path structure as "triggered when any segment name in the node path changes or the level depth changes," and the "data deletion trigger" is "triggered when a parent node containing child nodes is deleted." These conditions are bound to the corresponding logic entities to generate a trigger condition-logic entity association table, which contains "logic entity ID, data type identifier, new trigger condition, modification trigger condition, deletion trigger condition, and condition validation method."

[0063] Next, the priority calculation engine is started, calling the data attribute and access frequency integration table to extract the "daily average number of visits," "peak access periods," and "association strength value" from the data association table for each data subset. A "priority calculation vector" is then constructed—the vector dimension is "access frequency weight (60%) + association strength weight (40%)." Access frequency is quantified as "high frequency (≥500 visits per day) → 3 points, medium frequency (200-500 visits) → 2 points, low frequency (<200 visits) → 1 point," and association strength is quantified as "≥0.8 → 3 points, 0.5-0.8 → 2 points, <0.5 → 1 point" (e.g., the "order subset" has 600 daily visits (3 points) and an association strength of 0.89 (3 points), so the priority vector value = 3 × 0.6 + 3 × 0.4 = 3 points). The priority vector value of the corresponding data subset is matched for each real-time mapping logic body, generating a logic body-priority lookup table containing "logic body ID, priority score, and priority level (high / medium / low)."

[0064] The rule base integration engine is then activated, taking the trigger condition-logic body association table and the logic body-priority lookup table as input. It performs association matching using the "logic body ID" to supplement complete trigger condition and priority information for each mapping logic: field-type logic bodies are supplemented with "field-level trigger condition + priority based on access frequency," text-type logic bodies are supplemented with "keyword change trigger condition + priority based on association strength," and node-type logic bodies are supplemented with "path structure trigger condition + comprehensive priority." Simultaneously, a "priority conflict resolution mechanism" is configured: when different types of logic bodies trigger updates for the same directory node, they are sorted a second time according to type weights in the order of "field-type (business core) → node-type (structural association) → text-type (semantic supplement)," ensuring that core data is updated first.

[0065] Finally, the integrated mapping logic is stored by data type. Each type module contains fields such as "logic body ID, data type identifier, directory node ID, mapping rule details, add / modify / delete trigger conditions, priority score, and conflict resolution weight," generating a data-directory mapping rule library containing data type, mapping logic, trigger conditions, and execution priority. This technology differs from traditional designs with fixed trigger conditions and priorities by achieving fine-grained control through "type-specific trigger conditions + dynamic priority vectors": quantified change thresholds for field-type rules prevent frequent triggers due to minor changes; keyword proportion conditions for text-type rules reduce invalid updates; structure-aware triggers for node-type rules adapt to nested data characteristics; and priority vectors, combined with access popularity and association importance, make rule execution more closely aligned with actual business needs.

[0066] Optionally, step 4, based on the data-directory mapping rule base, performs real-time change monitoring on the structured raw dataset and triggers the corresponding rule execution in the data-directory mapping rule base to update the directory nodes, thereby generating a usable dynamic data directory adapted to the data weaving environment. This specifically includes the following steps:

[0067] Step 41: Based on the data-directory mapping rule base, monitor the changes of the structured raw dataset in real time, and trigger the corresponding rules in the data-directory mapping rule base to update the directory nodes, so as to generate a dynamic data directory that is updated in real time.

[0068] Step 42: Based on the cross-data source access requirements of the data weaving environment, perform availability verification and adaptation adjustments on the real-time updated dynamic data catalog to generate a usable dynamic data catalog adapted to the data weaving environment.

[0069] Optionally, step 41, based on the data-directory mapping rule base, performs real-time change monitoring on the structured raw dataset and triggers the corresponding rule in the data-directory mapping rule base to update the directory nodes, thereby generating a dynamically updated data directory in real time. This specifically includes the following steps:

[0070] Step 411: Configure the "rule-data" binding between the data-directory mapping rule base and the structured raw dataset to generate a data monitoring configuration table containing the association between "data identifier-monitoring threshold-rule ID";

[0071] Step 412: Based on the data monitoring configuration table, perform a “hierarchical monitoring-pre-verification-instruction generation” linkage process on the structured raw dataset to generate an ordered directory node update instruction set;

[0072] Step 413: Based on the ordered directory node update instruction set, perform the "incremental update-association synchronization-log closed loop" operation on the initial skeleton of the dynamic data directory to generate a dynamic data directory that is updated in real time.

[0073] Preferably, the specific implementation process of step 411 is as follows: First, load the data-directory mapping rule base and the structured raw dataset. The data-directory mapping rule base contains three types of rules: "field type / text type / node type" (each rule contains "rule ID, data type identifier, mapping logic, trigger condition, and execution priority", such as "field type rule ID = R001, corresponding to the order data subset, trigger condition is 'order amount changes by more than 10%'"). The structured raw dataset contains "order field type subset, customer text type subset, and logistics node type subset" (each subset contains "unique data identifier, core field / text content / node path, and update timestamp"). Then, start the "rule"... - The "Data Binding Engine" has three built-in modules: "Rule Parser," "Threshold Calculator," and "Configuration Table Generator." First, the rule parser extracts the "Rule ID, corresponding data type identifier, and change description in the trigger condition" for each rule from the data-directory mapping rule base (e.g., the change description for R001 is "order amount changes by more than 10%"). Then, it extracts the "unique data identifier, data type identifier, and historical fluctuation range of core fields" for each subset from the structured raw dataset (e.g., the historical fluctuation range of "order amount" for the order subset is "50-10000 yuan"). Finally, it constructs a rule-data binding matrix—the rows of the rule-data binding matrix representing the data in the data-directory mapping rule base. The rule ID (e.g., R001 in row 1), the column representing a subset of the structured original dataset (e.g., column 1 being a subset of order fields), and the intersection of the rule-data binding matrix marked with "Match" (based on data type identifier consistency; e.g., if R001 matches the order subset, mark "Yes"); then the threshold calculator is called to dynamically generate monitoring thresholds for the matched "rule-data" pairs in the rule-data binding matrix: for field-type rules, the monitoring threshold is set to 10%-20% of the historical fluctuation range of the core field (e.g., if the order amount fluctuates between 50-10000 yuan, the monitoring threshold is set to "±10%", i.e., a change exceeding 5-1000 yuan triggers the check); for text-type rules, the monitoring threshold is set according to the historical text relationship... Set a monitoring threshold of 20% of the number of keywords (e.g., if customer feedback text contains an average of 8 keywords, set the monitoring threshold to "add / delete more than 2 keywords"); for node-type rules, set the monitoring threshold according to the number of node path levels (e.g., if a logistics path contains 3 levels of nodes, set the monitoring threshold to "add or remove more than 1 level or change node name"); finally, call the configuration table generator to organize "unique data identifier, dynamically generated monitoring threshold, matching rule ID, and data type identifier" according to "one binding relationship per row" to generate a data monitoring configuration table (the data monitoring configuration table fields include "data ID, data type, monitoring threshold (including threshold type / specific value), associated rule ID, and threshold update cycle").This technology differs from the traditional fixed threshold configuration method. It dynamically calculates the monitoring threshold by "historical fluctuations + data type", avoiding false triggers for high-frequency small changes (such as no trigger if the order amount changes by 1 yuan out of 100 yuan), while reducing missed triggers for low-frequency large changes (such as triggering when the logistics node level suddenly decreases by 2 layers), thus improving the accuracy of monitoring.

[0074] Preferably, in the scenario of enterprise order-logistics data linkage update (changes in order amount require synchronous updates to the logistics cost directory node, and changes in customer feedback keywords require synchronous updates to the service evaluation directory node), step 412 is specifically implemented as follows: First, load the data monitoring configuration table. The data monitoring configuration table contains binding relationships such as "Order Data ID = D001, Monitoring Threshold = Order Amount ±10%, Association Rule ID = R001", "Customer Feedback Data ID = D002, Monitoring Threshold = Keywords ±2, Association Rule ID = R002", and "Logistics Data ID = D003, Monitoring Threshold = Path Level ±1, Association Rule ID = R003", which serve as the input for linkage processing; then... The "Hierarchical Monitoring-Pre-Verification-Command Generation Linkage Engine" is activated. The first step is to perform hierarchical monitoring: Based on the execution priority corresponding to the "Association Rule ID" in the data monitoring configuration table (set in step 33, e.g., order data priority "High", customer feedback priority "Medium", logistics data priority "Medium-High"), a monitoring frequency is assigned to different priority data—high priority data is monitored every 10 seconds, medium-high priority every 30 seconds, and medium priority every 60 seconds. The current status of each subset of the structured raw dataset is read in real time (e.g., order D001 current amount = 1200 yuan, historical amount = 1000 yuan), and compared with the monitoring threshold in the data monitoring configuration table (1000 yuan × 10). If the change is 100 yuan and the current change is 200 yuan, exceeding the monitoring threshold, the "data-rule" pairs that exceed the monitoring threshold (e.g., D001-R001, D003-R003) are filtered out, and a list of data exceeding the threshold is generated. The second step is to perform pre-validation: call the "mapping logic" of the corresponding rule in the data-directory mapping rule base (e.g., the mapping logic of R001 is "order amount field → order management directory node amount attribute"), check whether the change exceeding the monitoring threshold conforms to the legality of the mapping logic (e.g., whether the order amount change is numeric, not character), exclude invalid changes (e.g., incorrectly entered character amount changes), and verify the current status of the corresponding directory node (e.g., whether the order management node is in an updatable state). The first step is to generate a valid updated data list (in an unlocked state). The second step is to generate an instruction: generate a directory node update instruction for each record in the valid updated data list. The directory node update instruction includes "target node ID (extracted from the data-directory mapping rule base, such as node ID = N001 corresponding to R001), update type (add / modify / delete, such as modification for D001), update content (such as 'update the amount attribute of the order management node from 1000 yuan to 1200 yuan'), execution priority (using the rule priority)", and sort them according to execution priority (such as D001-R001 with higher priority being ranked first, and D003-R003 being ranked second), generating an ordered directory node update instruction set.This technology differs from the traditional "monitoring-command" separation process. Through the linkage logic of "hierarchical monitoring to ensure priority response to core data, pre-verification to exclude invalid updates, and priority sorting to avoid command conflicts", it reduces the interference of invalid commands on catalog updates, while ensuring that catalog node update commands for high-priority data (such as orders) are executed first, which meets the needs of enterprise business priorities.

[0075] Preferably, the specific implementation process of step 413 is as follows: First, load the ordered directory node update instruction set and the initial skeleton of the dynamic data directory. The ordered directory node update instruction set includes "Instruction 1: Update the amount attribute of node N001 (order management) to 1200 yuan, high priority; Instruction 2: Update the path attribute of node N003 (logistics tracking) to 'logistics→transport→city', medium-high priority". The initial skeleton of the dynamic data directory includes nodes N001, N003 and associated node N002 (customer management, associated with N001); then start the "incremental update-association synchronization-log closed-loop execution engine", the first... The first step is to perform incremental updates: For each target node in the ordered directory node update instruction set, the current attribute value of the target node in the initial skeleton of the dynamic data directory is read (e.g., N001 current amount = 1000 yuan), compared with the update content in the instruction (1200 yuan), and only the changed attribute fields are updated (the unchanged "order number" attribute is not updated), generating an incremental update node list (the incremental update node list includes "node ID, attribute value before update, attribute value after update, and update time"), avoiding the resource consumption caused by full updates; The second step is to perform association synchronization: Based on the "association relationship identifier" of the nodes in the initial skeleton of the dynamic data directory (e.g., N001 and N00...), 2. **Association (relationship is "Order → Customer"):** Check if there are associated nodes in the incremental update node list. For example, after N001 (Order) is updated, the "Total Historical Order Amount" attribute of N002 (Customer) needs to be updated synchronously (from 5000 yuan to 5200 yuan). Generate an associated synchronization node list (containing "Main Update Node ID, Associated Node ID, and Synchronized Update Content"), and insert the associated synchronization instruction into the corresponding priority position in the ordered directory node update instruction set (e.g., with the same priority as the main instruction). 3. **Execution Log Closure:** Record the complete execution process of the incremental update node list and the associated synchronization node list. The log content includes "instruction ID, target node ID, update type, attribute values ​​before and after update, synchronization status of associated nodes, execution result (success / failure), and execution timestamp," generating a directory update closed-loop log. Simultaneously, the directory update closed-loop log is associated with the "data ID" in the data monitoring configuration table, facilitating the tracing of the entire "data change → directory update" chain. Finally, the logs of the incrementally updated nodes, the synchronized nodes, and the directory update closed-loop log are integrated to generate a real-time updated dynamic data directory (in the real-time updated dynamic data directory, N001 has an amount attribute of 1200 yuan, N002 has a historical order amount of 5200 yuan, and N003 has a path attribute of a new path).This technology differs from the traditional approach of updating only a single node without log traceability. It solves the problems of inconsistent data and inability to locate faults in associated nodes after the directory is updated by "incremental updates to reduce resource consumption, correlation synchronization to ensure consistency of directory nodes (such as matching the amount of customer nodes and order nodes), and log closed-loop to achieve full-link traceability". It is suitable for the needs of multi-node linkage in enterprise data weaving environments.

[0076] Optionally, step 42 involves verifying and adapting the real-time updated dynamic data catalog based on the cross-data source access requirements of the data weaving environment to generate a usable dynamic data catalog adapted to the data weaving environment. This includes the following steps:

[0077] Step 421: Extract the core parameters from the cross-data source access requirements and classify them to generate a scenario-based directory availability verification index table;

[0078] Step 422: Based on the scenario-based directory availability verification index table, perform hierarchical verification on the dynamically updated data directory in real time to generate a directory availability verification assessment containing non-compliant items and cause analysis.

[0079] Step 423: Based on the catalog availability verification and assessment, perform targeted adaptation adjustments on the real-time updated dynamic data catalog to generate a usable dynamic data catalog that adapts to the data weaving environment.

[0080] Optionally, step 421, extracting and classifying the core parameters from cross-data source access requirements to generate a scenario-based directory availability verification indicator table, specifically includes the following steps:

[0081] Step 4211: Perform a requirement-directory association analysis on the cross-data source access requirement parameters of the data weaving environment and the real-time updated dynamic data directory to extract the core parameters in the cross-data source access requirements.

[0082] Step 4212: Classify the core parameters in the cross-data source access requirements into scenarios such as "batch query scenario", "real-time query scenario" and "data synchronization scenario", and build link connectivity indicators, response time indicators and data consistency indicators for each scenario.

[0083] The link connectivity metric corresponds to the cross-source access link type, the response time metric matches the time threshold in the requirements, and the data consistency metric is associated with the association strength value in the data association matrix to generate a scenario-based directory availability verification metric table.

[0084] Preferably, in e-commerce cross-system data access scenarios (such as cross-source query and synchronization of order system, inventory system, and logistics system), step 4211 is specifically implemented as follows: First, the cross-data source access requirement parameters and the real-time updated dynamic data catalog of the data weaving environment are loaded. The cross-data source access requirement parameters include "requirement description (such as 'daily batch query of paid orders and corresponding product inventory and logistics delivery status in East China within 3 days at 9 am' 'real-time synchronization of order payment status to the inventory system to deduct inventory'), requirement initiator (such as operations department, inventory management module), requirement frequency (such as once a day batch query, real-time trigger synchronization)"; the real-time updated dynamic data catalog includes "order management node (related fields 'order ID, payment status, region, time'), inventory management node (related fields 'product ID, inventory quantity, related order ID'), logistics management node (related fields 'order ID, delivery status, logistics tracking number')", and each node is marked with "business attribute, related node ID, access link type".Next, the requirement-directory association analysis engine is launched. The first step is to perform requirement parameter parsing: the cross-data source access requirement parameters are broken down into "requirement scenario type (e.g., 'batch query', 'real-time synchronization'), access node range (e.g., 'order + inventory + logistics node'), data filtering conditions (e.g., 'East China region, within 3 days, paid'), performance constraints (e.g., 'batch query time ≤ 10 seconds', 'synchronization delay ≤ 1 second'), and data association requirements (e.g., 'order ID associated with inventory and logistics')", generating a requirement parameter breakdown list; the second step is to perform directory node matching: based on the "access node range" and "data filtering conditions" in the requirement parameter breakdown list, the corresponding nodes and fields are matched in the real-time updated dynamic data directory - for example, "order node" matches "order ID, payment status" fields, "inventory node" matches "product ID, inventory quantity" fields, and the matched node ID, field name, and the relationship between nodes (e.g., order node and inventory node are associated through "order ID - associated order ID"), generating a directory node matching list; the third step is to construct Build a requirement-directory association matrix: The rows of the matrix represent the parameter items in the requirement parameter breakdown list (such as "access node range", "performance constraint - query time", "data association requirements"), and the columns represent the nodes and attributes in the directory node matching list (such as "order node - field integrity", "inventory node - access link stability", "order - inventory association strength"). The intersection is filled with "matching degree" (such as "access node range" matching "order + inventory + logistics node" 100%, "performance constraint - query time" matching "batch access time ≤ 8 seconds" marked in the directory node 80%), generating a requirement-directory association matrix; The fourth step is to extract core parameters: Based on the parameter items with a matching degree ≥ 70% in the requirement-directory association matrix, filter out "scenario type (batch query / real-time synchronization), access node combination (order + inventory + logistics), data filtering conditions (East China region, within 3 days, paid), performance threshold (batch query ≤ 10 seconds, synchronization delay ≤ 1 second), and associated field (order ID)", and organize them into a cross-data source access core parameter list. This technology differs from the traditional method of directly extracting keywords from requirement text. Through correlation analysis of "requirement decomposition - directory matching - matrix quantification", it eliminates parameters that are irrelevant to the availability of the directory (such as the request initiator and report display format), ensuring that the extracted core parameters accurately correspond to the directory verification requirements and avoiding redundancy in subsequent indicator design.

[0085] Preferably, the specific implementation process of step 4212 is as follows: First, load the cross-data source access core parameter list generated in step 4211. This list includes "Scenario type: batch query / real-time synchronization / data synchronization (supplementing common 'data synchronization' scenarios in e-commerce, such as synchronizing inventory data to the order system), access node combination: order + inventory + logistics, performance threshold: batch query ≤ 10 seconds / synchronization delay ≤ 1 second, associated fields: order ID, association strength (derived from the data association table in step 22: order-inventory association strength 0.9, order-logistics association strength 0.85)", which serves as the input for scenario classification and indicator construction. The first step is to classify the scenarios: categorize the requirements in the core parameter list into "batch query scenario", "real-time query scenario", and "data synchronization scenario". For example, "batch query of orders and corresponding inventory and logistics status in East China within 3 days at 9 am every day" is classified into the batch query scenario, "real-time query of product inventory when users place an order" is classified into the real-time query scenario, and "synchronize inventory deduction records to the order system after order payment" is classified into the data synchronization scenario. A scenario-parameter mapping table is generated (the table contains "scenario type, corresponding core parameters (access node, performance threshold, related fields)"). The second step is to build specific metrics for each scenario: For batch query scenarios, construct link connectivity metrics (corresponding to cross-source access link types: "direct link" and "transfer link" for order → inventory → logistics; the connectivity status of both links needs to be verified to avoid query failure due to a single link failure), response time metrics (matching the threshold of "batch query ≤ 10 seconds" in the core parameters; the total time spent on a single batch query needs to be calculated), and data consistency metrics (corresponding to the association strength values ​​in the data association matrix of step 22: order-inventory association strength of 0.9 corresponds to a consistency threshold of ≥90%, order-logistics association strength of 0.85 corresponds to a consistency threshold of ≥85%; the consistency ratio of associated field values ​​needs to be verified); For real-time query scenarios, construct link connectivity metrics (for... For the "dedicated link for high-frequency access," it is necessary to verify the link's continuous connectivity rate, response time (matching the threshold of "synchronization latency ≤ 1 second," requiring statistics on the response time compliance rate of high-frequency queries per second), and data consistency (association strength of 0.9 corresponds to a consistency threshold of ≥ 92%, as real-time queries have higher requirements for data consistency). For data synchronization scenarios, construct the following metrics: link connectivity (corresponding to the "bidirectional synchronization link," requiring verification of the bidirectional connectivity status of the data upload and download links), response time (matching the threshold of "synchronization latency ≤ 1.5 seconds," as synchronization requires bidirectional confirmation, and the threshold is slightly higher than that for real-time queries), and data consistency (association strength of 0.9 corresponds to a consistency threshold of ≥ 95%, as synchronized data directly affects inventory accuracy, requiring the highest level of consistency).The third step is to organize the indicator information: The "indicator name, indicator definition, corresponding cross-source link / requirement threshold / association strength, and indicator verification method" for each scenario are categorized and organized by scenario, generating a scenario-based catalog availability verification indicator table (e.g., the link connectivity indicator row for the batch query scenario includes "indicator name: cross-source link connectivity rate, indicator definition: connectivity ratio of direct / transfer links, corresponding cross-source link: order → inventory → logistics direct / transfer link, indicator threshold: connectivity rate ≥ 98%, verification method: monitor link status during simulated query"). This technology differs from the traditional approach of designing uniform indicators for all scenarios. By "scenario-specific indicators + binding indicator parameters with core parameters / association strength," the indicators are more closely aligned with the business needs of different scenarios (e.g., real-time queries focus on response time compliance, data synchronization focuses on bidirectional links and high consistency), avoiding the problem of general indicators failing to cover scenario differences.

[0086] Optionally, step 422, based on the scenario-based directory availability verification index table, performs hierarchical verification on the dynamically updated data directory in real time to generate a directory availability verification assessment containing non-compliant items and cause analysis, specifically including the following steps:

[0087] Step 4221: Based on the scenario-based catalog availability verification index table, when performing hierarchical verification on the dynamically updated data catalog in real time, for the "batch query scenario", simulate several cross-source batch query requests and monitor the link connectivity status and actual response time of each request; for the "real-time query scenario", simulate several high-frequency real-time query requests and calculate the response time compliance rate at fixed intervals; for the "data synchronization scenario", compare the consistency of field values ​​of cross-source associated data according to the verification frequency.

[0088] Step 4222: Record the verification items that fail to meet the standards in various scenarios during the graded verification, and analyze the reasons for failure to meet the standards, so as to generate a catalog usability verification assessment containing the failure items and the reasons analysis. The verification items are marked with scenario type, indicator name, actual indicator value and indicator threshold.

[0089] Preferably, the specific implementation process of step 4221 is as follows: First, load the availability verification index table of the step scenario directory. This table is classified into "batch query scenario", "real-time query scenario" and "data synchronization scenario" and includes information such as "scenario type, index name, index threshold, corresponding cross-source link / related field, verification method" (e.g., the link connectivity index threshold for the batch query scenario is "connectivity rate ≥ 98%", corresponding to "order → inventory → logistics direct connection / transfer dual link"; the data consistency index threshold for the data synchronization scenario is "consistency rate ≥ 95%", corresponding to the related field "order ID"). This serves as the benchmark input for hierarchical verification.

[0090] For batch query scenarios, a multi-link parallel simulation engine is launched: according to the configuration of "order → inventory → logistics direct / transfer dual links" in the indicator table, several (e.g., 50) cross-source batch query requests covering different filtering conditions are generated (e.g., "orders paid within 3 days in East China + corresponding inventory + logistics status" and "orders awaiting shipment within 1 day in North China + corresponding inventory"); at the same time, simulated requests are sent to the two links in parallel, and the link connectivity status of each request is monitored in real time (e.g., whether the direct link returns a 'connection successful' response, and whether the transfer link forwards normally through the intermediate node) and the actual response time (the total time from the request to the complete return of data); the verification result of each request is labeled with "link type (direct / transfer), connectivity status (success / failure), actual response time, and whether it meets the standard (response time ≤ 10 seconds is considered to meet the standard)", and a batch query scenario verification record is generated.

[0091] For real-time query scenarios, a high-frequency time-slice statistics engine is activated: Based on the threshold of "synchronization delay ≤ 1 second" and the configuration of "dedicated high-frequency access link" in the indicator table, several (e.g., 100) simulated high-frequency real-time query requests are generated (e.g., "querying the real-time inventory of product A when a user places an order" and "querying the corresponding logistics status when a user views an order"). A fixed time interval (e.g., 10 seconds) is set as a time slice, and 10 requests are sent continuously in each time slice. The percentage of requests with a response time ≤ 1 second in that time slice is counted (i.e., the response time compliance rate). The "statistical time, total number of requests sent, number of compliant requests, compliance rate, and response time of non-compliant requests" for each time slice are recorded, and a real-time query scenario response time compliance rate statistical report is generated.

[0092] For data synchronization scenarios, the consistency verification engine for associated fields is activated: Based on the threshold of "consistency rate ≥ 95%" in the indicator table and the configuration of the associated field "order ID", three types of data are extracted from the real-time updated dynamic data directory: "payment status of order nodes", "order-related inventory deduction quantity of inventory nodes", and "order-related delivery status of logistics nodes". According to the requirement of "verification frequency every 5 minutes" in the indicator table, the three types of data are matched periodically through "order ID" (e.g., matching "order ID = O20241010001" with payment status "paid", inventory deduction quantity "2 items", and logistics status "collected"). The proportion of records with successful matching (the three types of data are logically consistent, such as "paid" corresponding to "deduction quantity > 0" and "collected") to the total number of matched records (i.e., consistency rate) is counted, and "verification time, total number of records, number of consistent records, consistency rate, and field differences of inconsistent records (e.g., order payment status 'paid' but inventory deduction quantity '0')" is marked, generating consistency verification records for the data synchronization scenario.

[0093] Finally, the batch query scenario verification records, real-time query scenario response time compliance rate statistical reports, and data synchronization scenario consistency verification records are integrated and categorized by "scenario type - indicator name" to form a scenario-based hierarchical verification master record. This master record will directly serve as the core input for analyzing non-compliant items in step 4222. This technology differs from traditional single-link, fixed-frequency verification methods: the dual-link parallel simulation of batch queries avoids misjudgments caused by single-link failures; the time-slice statistics of real-time queries reflect response time fluctuations; and the correlation field verification of data synchronization accurately corresponds to business logic, improving the comprehensiveness and reliability of the verification results.

[0094] Preferably, the specific implementation process of step 4222 is as follows: First, load the scenario-based hierarchical verification record. This record contains the "verification result, actual value, whether it meets the standard, and related link / field / time slice information" for three types of scenarios (e.g., in the batch query scenario, 3 direct connection requests failed to connect, and the actual connection rate was 94% < 98%; in the real-time query scenario, the 10:00-10:01 time slice compliance rate was 85% < 100%), which serves as the input for screening non-compliant items and analyzing the reasons.

[0095] The first step is to perform a screening of non-compliant items: Iterate through the total records by "Scenario Type - Indicator Name" and filter out the verification items whose "actual value does not meet the indicator threshold" - such as the link connectivity indicator for batch query scenarios (actual connectivity rate 94% < 98%), the response time compliance rate for real-time query scenarios (10:00-10:01 time slice 85% < 100%), and the data consistency indicator for data synchronization scenarios (consistency rate at a certain verification time point 92% < 95%). Mark each non-compliant item with "Scenario Type, Indicator Name, Indicator Threshold, Actual Value, and Related Information (such as 'Batch Query - Link Connectivity - 3 Direct Links Failed')" to generate a list of non-compliant items.

[0096] The second step involves multi-dimensional cause analysis: For each verification item in the list of non-compliant items, the cause dimensions are broken down by combining the scenario-based catalog availability verification indicator table with real-time updated dynamic data catalog information.

[0097] For items where link connectivity is not up to standard (such as failure of batch query of direct link), check the node status of the corresponding cross-source link (such as "whether the direct connection port between the order node and the inventory node is occupied, and whether there is delay or packet loss in the intermediate network"), and locate the cause (such as "the load of an intermediate switch in the direct link is too high, causing 3 requests to time out").

[0098] For items where the response time compliance rate does not meet the target (such as real-time query of the 10:00-10:01 time slice), check the index structure of the corresponding directory node (such as "whether there is fragmentation in the 'product ID' index of the inventory node, which leads to increased query time"), and locate the cause (such as "the index fragmentation rate of the inventory node is too high, and some requests take 1.2 seconds").

[0099] For items where data consistency does not meet the standard (e.g., consistency rate of 92%), check the association and synchronization mechanism of the directory nodes (e.g., "After the payment status of the order node is updated, is the inventory node deduction record updated synchronously?"), and locate the cause (e.g., "After the payment status of 5 orders is updated, the inventory node is not updated in time, resulting in inconsistent field values").

[0100] For each non-compliant item, record the "reason dimension (link / index / association synchronization), specific reason description, and scope of impact (e.g., 'only 3 requests on the direct link are affected')" to generate a non-compliant item reason analysis table.

[0101] The third step integrates the evaluation results: The list of non-compliant items and the analysis table of reasons for non-compliant items are linked by "scenario type - indicator name," and an identifier is added indicating "whether urgent adjustment is needed (e.g., low real-time compliance rate requires urgent handling)." This forms a catalog availability verification evaluation containing non-compliant items and their reasons (table fields include "scenario type, indicator name, indicator threshold, actual value, reason for non-compliance, scope of impact, and urgent adjustment identifier"). This technology differs from the traditional method of only recording non-compliant results. It accurately locates the root cause of the problem through "multi-dimensional reason breakdown (link / index / association synchronization)," avoiding blind operations during subsequent adaptation adjustments. Simultaneously, it marks the scope of impact and urgency, providing a clear basis for targeted adjustments and forming a logical closed loop of "verification-analysis-adjustment."

[0102] Optionally, step 423, based on the catalog availability verification and assessment, performs targeted adaptation adjustments on the real-time updated dynamic data catalog to generate a usable dynamic data catalog adapted to the data weaving environment, specifically including the following steps:

[0103] Step 4231: Based on the directory availability verification and evaluation, when performing targeted adaptation adjustments on the dynamically updated data directory in real time, optimize cross-source access routes for items where link connectivity does not meet the standard; reconstruct the directory index structure for items where response time does not meet the standard; and synchronously update the attribute values ​​of associated directory nodes for items where data consistency does not meet the standard.

[0104] Step 4232: After the adaptation and adjustment are completed, perform the scenario-based verification of step 422 again. After confirming that the link connectivity, response time, and data consistency meet the standards, integrate the adjusted directory hierarchy, association identifiers, and verification pass records to generate a usable dynamic data directory adapted to the data weaving environment.

[0105] Preferably, the specific implementation process of step 4231 is as follows: First, load the directory availability verification assessment, which includes information such as "scenario type, non-compliant indicators, reasons for non-compliance, and scope of impact" (e.g., the link connectivity indicator of the batch query scenario is non-compliant because "the load of the intermediate switch in the direct link is too high"; the response time indicator of the real-time query scenario is non-compliant because "the index fragmentation rate of the inventory node is too high"; the data consistency indicator of the data synchronization scenario is non-compliant because "the inventory node is not synchronized in time after the order node is updated"). This serves as the core basis for targeted adaptation and adjustment.

[0106] For links that fail to meet connectivity standards, a cross-source routing dynamic optimization engine is activated: First, the "link type (direct / relay), involved nodes (e.g., order node → inventory node), and fault location (intermediate switch)" of the non-compliant links in the directory availability verification assessment are extracted to construct a "route candidate pool"—containing three types of routes: original direct links, backup direct links (replacing low-load switches), and relay links (adding a "order → cache node → inventory" relay path); then, a "link health monitoring mechanism" is configured to collect the "bandwidth utilization, latency, and packet loss rate" of each candidate route in real time, according to the formula "health = (1 - bandwidth utilization)". The health score is calculated as follows: (1 - latency / threshold) × 0.4 + (1 - packet loss rate) × 0.3 + (1 - packet loss rate) × 0.3. Then, the "dynamic route switching rule" is set: when the current route health score is below 80%, it automatically switches to the candidate route with the highest health score (e.g., when the original direct link health score is 70%, it switches to the backup direct link with a health score of 92%). Finally, a route optimization configuration scheme is generated, which includes "candidate route list, health score calculation rule, switching threshold, and monitoring frequency". The route optimization configuration scheme is applied to the dynamically updated data catalog in real time to complete the link connectivity adaptation and adjustment.

[0107] For items failing to meet response time targets, the directory index reconstruction engine is activated based on specific scenarios. This involves extracting the "node type (e.g., inventory node), access scenario (real-time query / batch query), and index issues (high fragmentation rate / index type mismatch)" of the failing nodes from the directory availability verification assessment. The index is then reconstructed in a differentiated manner based on the scenario: For real-time query scenarios (e.g., users checking inventory after placing an order), the original disk index is deleted, and a "memory hash index" (key: "product ID", value: "real-time inventory quantity") is built to reduce disk I / O time. For batch query scenarios (e.g., operations checking regional inventory), the disk index is retained, and "index defragmentation" (merging fragmented pages and rebuilding the index structure) is performed. Simultaneously, a "partitioned index" is added (partitioned by "product category" to narrow the query scope). "Dynamic index maintenance rules" are configured: real-time query indexes undergo memory consistency checks every hour, and batch query indexes undergo defragmentation daily at midnight. An index reconstruction configuration scheme is generated, containing "scenario-index type correspondence, maintenance frequency, and consistency verification method." This scheme is then applied to the dynamically updated data directory to complete the response time adaptation adjustment.

[0108] For data consistency non-compliance items, a node-triggered synchronization engine is activated: The "master node (order node), associated node (inventory node), and synchronization delay reason (no active triggering mechanism)" for non-compliant relationships are extracted from the directory availability verification assessment. A "master-slave synchronization triggering mechanism" is designed: when key fields such as "payment status" and "order amount" are updated on the master node (order node), a "synchronization notification instruction" is automatically generated and pushed to the associated node (inventory node) via a message queue. After receiving the synchronization notification instruction, the associated node executes synchronization according to the process of "first verifying the integrity of the master node data → then updating the corresponding local fields → finally recording the synchronization log". A "synchronization timeout retry rule" is configured: when synchronization times out (e.g., exceeding 1 second), it automatically retryes 3 times with retry intervals of 100ms, 200ms, and 500ms respectively. If it still fails, it is marked as "synchronization abnormal" and an alarm is triggered. A synchronization mechanism configuration scheme is generated, which includes a "trigger field list, synchronization process, retry rules, and alarm threshold". This synchronization mechanism configuration scheme is applied to the dynamically updated data directory in real time to complete the data consistency adaptation adjustment.

[0109] Finally, the execution results of the route optimization configuration scheme, index reconstruction configuration scheme, and synchronization mechanism configuration scheme are integrated to generate a targeted adaptation adjustment record. This record includes "adjustment indicators, adjustment measures, execution time, and adjusted parameters (such as route health, index response time, and synchronization latency)." This targeted adaptation adjustment record will serve as a reference for the re-verification in step 4232. This technology differs from the traditional "one-size-fits-all" adjustment approach: the dynamic route candidate pool for link optimization avoids reliance on single links; the scenario-specific design of index reconstruction adapts to different access needs; and the trigger-based design of the synchronization mechanism reduces latency. Each of these three aspects is precisely adjusted to address the reasons for non-compliance, improving adaptation efficiency.

[0110] Preferably, the specific implementation process of step 4232 is as follows: First, load the directional adaptation adjustment record and the scenario-based directory availability verification index table. The directional adaptation adjustment record provides "key parameters after adjustment (such as route health 92%, index response time 0.8 seconds, synchronization delay 0.5 seconds)", and the scenario-based directory availability verification index table provides "index thresholds for various scenarios (such as link connectivity ≥ 98%, response time ≤ 1 second, consistency ≥ 95%)", and the two together serve as inputs for re-verification.

[0111] The first step is to perform scenario-based secondary verification: following the verification logic of step 4221, but using the "adjusted real-time updated dynamic data catalog" as the verification object. For batch query scenarios, simulated requests are sent using the dynamic routes adjusted by the routing optimization configuration scheme to monitor the link connectivity rate (e.g., from 94% to 99%). For real-time query scenarios, high-frequency queries are performed using the memory index reconstructed by the index reconstruction configuration scheme to calculate the response time compliance rate (e.g., from 85% to 99%). For data synchronization scenarios, the consistency of associated fields is verified using a trigger-based synchronization mechanism designed based on the synchronization mechanism configuration scheme (e.g., from 92% to 98%). The "scenario type, indicator name, adjusted actual value, and compliance status" of the secondary verification are recorded, and a secondary verification result list is generated.

[0112] The second step is to confirm compliance: Iterate through the list of secondary verification results and determine whether all indicators meet the threshold of the scenario-based catalog availability verification indicator table. If there are any non-compliant items (such as consistency of 94% < 95% in a certain scenario), return to step 4231 and make supplementary adjustments based on the list of secondary verification results (such as optimizing the synchronization retry interval). If all indicators meet the standards (such as link connectivity of 99%, response time compliance rate of 99%, and consistency of 98% all meet the threshold), proceed to the next step of integration.

[0113] The third step is to integrate the execution results: Extract the "directory hierarchy (e.g., 'Transaction Management → Orders → Payments') and association identifier (e.g., Order-Inventory Association Strength 0.9)" of the adjusted real-time updated dynamic data directory, and combine it with the targeted adaptation adjustment records (adjustment measures, execution time) and the secondary verification result list (compliance records), and integrate the information according to the logic of "scenario type - adjustment content - verification result"—such as "batch query scenario - dynamic routing optimization - link connectivity rate 99% compliant", "real-time query scenario - memory index reconstruction - response time 0.8 seconds compliant", "data synchronization scenario - triggered synchronization - consistency 98% compliant"; generate an integrated directory information package, which contains "adjusted directory structure, association table, targeted adaptation adjustment records, and secondary verification result list".

[0114] Finally, based on the integrated directory information package, a usable dynamic data directory adapted to the data weaving environment is generated. This directory retains the hierarchy and relationships of the initial dynamic data directory skeleton, while incorporating the adjustment results of routing optimization configuration schemes, index reconstruction configuration schemes, and synchronization mechanism configuration schemes. It can directly support e-commerce cross-source access needs (such as batch queries, real-time order placement, and inventory synchronization). This technology differs from the traditional approach of simply outputting a directory after verification. Through "secondary verification - compliance confirmation - result integration," it ensures that the adjusted directory fully meets the indicator requirements. At the same time, it integrates adjustment records and verification reports, facilitating subsequent traceability and maintenance, and meeting the manageability requirements of the enterprise's data weaving environment.

[0115] Figure 2 This is a schematic diagram illustrating the structure of a device for dynamic data catalog management and its construction within a data weaving environment, as described in an embodiment of this application. Figure 2 As shown, a dynamic data catalog management system and its structure built within a data weaving environment include:

[0116] The multi-source data processing module is used to classify, organize, and extract metadata from multi-source heterogeneous raw data in the data weaving environment to generate structured raw datasets and associated metadata tables.

[0117] The initial skeleton construction module for the catalog is used to construct the initial skeleton of the dynamic data catalog based on the structured original dataset and associated metadata tables, combined with the access frequency and relationships of data in the data weaving environment.

[0118] The directory real-time mapping rule configuration module is used to configure real-time mapping logic for different data types in the structured raw dataset based on the initial skeleton of the dynamic data directory, so as to generate a data-directory mapping rule library.

[0119] The real-time directory update module is used to monitor changes in the structured raw dataset in real time based on the data-directory mapping rule base, and trigger the corresponding rules in the data-directory mapping rule base to update the directory nodes, so as to generate a usable dynamic data directory that is adapted to the data weaving environment.

[0120] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for dynamic data catalog management and its construction in a data weaving environment, characterized in that, include: Step 1: Classify and organize the multi-source heterogeneous raw data in the data weaving environment and extract metadata to generate a structured raw dataset and associated metadata table; Step 2: Based on the structured original dataset and associated metadata tables, and combined with the access frequency and relationships of data in the data weaving environment, construct the initial skeleton of the dynamic data catalog; Step 3: Based on the initial skeleton of the dynamic data catalog, configure real-time mapping logic for different data types in the structured raw dataset to generate a data-catalog mapping rule base; Step 4: Based on the data-directory mapping rule base, monitor the changes of the structured raw dataset in real time, and trigger the corresponding rule execution in the data-directory mapping rule base to update the directory nodes, so as to generate a usable dynamic data directory that is adapted to the data weaving environment. Step 2 includes: Step 21: Extract data attributes from the structured raw dataset and associated metadata table, and simultaneously perform access frequency statistics on the data weaving environment access logs to generate an integrated table of data attributes and access frequencies. Step 22: Perform field association analysis on the data attribute and access frequency integration table, and label the association strength values ​​of the associated datasets in the structured original dataset according to the analysis results to generate a data association table; Step 23: Based on the data attribute and access frequency integration table and the data relationship table, divide the structured original dataset into directory levels to generate a directory level division scheme; Step 24: Based on the directory hierarchy partitioning scheme, create index nodes for each level of the structured original dataset, each containing a unique data identifier, data attribute reference, and relationship identifier, to generate an initial skeleton of a dynamic data directory containing directory hierarchy, data index nodes, and relationship identifier.

2. The method according to claim 1, characterized in that, Step 1: Classify, organize, and extract metadata from the multi-source heterogeneous raw data in the data weaving environment to generate a structured raw dataset and associated metadata tables. This includes the following steps: Step 11: Perform format parsing and redundant data removal on the multi-source heterogeneous raw data in the data weaving environment to generate preliminarily cleaned multi-source heterogeneous data. Step 12: After the initial cleaning, the multi-source heterogeneous data is grouped according to data type, and each group is labeled with the corresponding data source identifier to generate classified multi-source heterogeneous data groups. Step 13: Perform structured processing on different types of data in the multi-source heterogeneous data set to generate a single-type structured data subset; Step 14: Extract the metadata information of each single-type structured data subset to generate structured data and corresponding metadata records; Step 15: Group the structured data according to business logic to generate a structured raw dataset; at the same time, organize the corresponding metadata records according to metadata fields.

3. The method according to claim 1, characterized in that, Step 3: Based on the initial skeleton of the dynamic data catalog, configure real-time mapping logic for different data types in the structured raw dataset to generate a data-catalog mapping rule base. This includes the following steps: Step 31: Perform correlation analysis on the initial skeleton of the dynamic data catalog and the structured original dataset, extract the type identifier of each data in the structured original dataset, and match the corresponding index node in the initial skeleton of the dynamic data catalog to generate a data type-catalog node correspondence table. Step 32: Based on the data type-directory node correspondence table, configure real-time mapping logic for different data types to generate a single-type real-time mapping logic list; Step 33: Add data change triggering conditions and execution priorities to each single-type real-time mapping logic list to generate a data-directory mapping rule base.

4. The method according to claim 3, characterized in that, According to the method described in claim 1, in step 32, when configuring real-time mapping logic for different data types based on the data type-directory node correspondence table to generate a single-type real-time mapping logic list, for "field-type" structured data, a mapping logic is configured to precisely match field names with directory node attribute names; for "text-type" unstructured data, a mapping logic is configured to extract core keywords and match directory node tags; and for "node-type" semi-structured data, a mapping logic is configured to match data node paths with directory node hierarchical paths to generate a single-type real-time mapping logic list.

5. The method according to claim 3, characterized in that, According to the method described in claim 1, step 33, supplementing each single-type real-time mapping logic list with data change triggering conditions and execution priorities to generate a data-directory mapping rule base, specifically includes the following steps: integrating the single-type real-time mapping logic lists, supplementing each mapping logic with data change triggering conditions including data addition triggering, data modification triggering, and data deletion triggering, and execution priorities sorted by data access frequency, and storing them according to data type to generate a data-directory mapping rule base containing data type, mapping logic, triggering conditions, and execution priorities.

6. The method according to claim 1, characterized in that, Step 4: Based on the data-directory mapping rule base, monitor the changes in the structured raw dataset in real time and trigger the corresponding rule execution in the data-directory mapping rule base to update the directory nodes, so as to generate a usable dynamic data directory adapted to the data weaving environment. This includes the following steps: Step 41: Based on the data-directory mapping rule base, monitor the changes of the structured raw dataset in real time, and trigger the corresponding rules in the data-directory mapping rule base to update the directory nodes, so as to generate a dynamic data directory that is updated in real time. Step 42: Based on the cross-data source access requirements of the data weaving environment, perform availability verification and adaptation adjustments on the real-time updated dynamic data catalog to generate a usable dynamic data catalog adapted to the data weaving environment.

7. The method according to claim 6, characterized in that, Step 41: Based on the data-directory mapping rule base, monitor the changes in the structured raw dataset in real time, and trigger the corresponding rules in the data-directory mapping rule base to update the directory nodes, so as to generate a dynamically updated data directory in real time. This includes the following steps: Step 411: Configure "rule-data" binding between the data-directory mapping rule base and the structured raw dataset to generate a data monitoring configuration table containing the "data identifier-monitoring threshold-rule ID" association; Step 412: Based on the data monitoring configuration table, perform a "hierarchical monitoring-pre-verification-instruction generation" linkage process on the structured raw dataset to generate an ordered directory node update instruction set; Step 413: Based on the ordered directory node update instruction set, perform the "incremental update-association synchronization-log closed loop" operation on the initial skeleton of the dynamic data directory to generate a dynamic data directory that is updated in real time.

8. The method according to claim 4, characterized in that, Step 42: Based on the cross-data source access requirements of the data weaving environment, perform availability verification and adaptation adjustments on the real-time updated dynamic data catalog to generate a usable dynamic data catalog adapted to the data weaving environment. This includes the following steps: Step 421: Extract the core parameters from the cross-data source access requirements and classify them to generate a scenario-based directory availability verification index table; Step 422: Based on the scenario-based directory availability verification index table, perform hierarchical verification on the dynamically updated data directory in real time to generate a directory availability verification assessment containing non-compliant items and cause analysis. Step 423: Based on the catalog availability verification and assessment, perform targeted adaptation adjustments on the real-time updated dynamic data catalog to generate a usable dynamic data catalog that adapts to the data weaving environment.

9. The method according to claim 8, characterized in that, Step 421: Extract the core parameters from cross-data source access requirements and categorize them to generate a scenario-based directory availability verification indicator table, which includes the following steps: Step 4211: Perform a requirement-directory association analysis on the cross-data source access requirement parameters of the data weaving environment and the real-time updated dynamic data directory to extract the core parameters in the cross-data source access requirements. Step 4212: Classify the core parameters in the cross-data source access requirements into scenarios such as "batch query scenario", "real-time query scenario" and "data synchronization scenario", and build link connectivity indicators, response time indicators and data consistency indicators for each scenario. The link connectivity metric corresponds to the cross-source access link type, the response time metric matches the time threshold in the requirements, and the data consistency metric is associated with the association strength value in the data association matrix to generate a scenario-based directory availability verification metric table.

Images