Artificial intelligence based adaptive data weaving performance optimization method

Abstract

This application discloses an AI-based adaptive data weaving performance optimization method, which includes: acquiring multi-source heterogeneous data based on a data source list and parsing it to generate a metadata dataset containing at least technical metadata, business metadata, operational metadata, and management metadata; abstracting and encapsulating the metadata dataset using a virtual table interface to generate a logical data carrier; determining the operational attributes of the metadata dataset to generate a data weaving path for the logical data carrier based on the operational attributes; monitoring the data access task queue and access indicator feedback in real time, extracting resource load correlation features from the data access task queue and access indicator feedback based on a constructed resource scheduling model, and generating a resource scheduling strategy based on the resource load correlation features; optimizing the data weaving path of the logical data carrier based on the resource scheduling strategy to obtain an optimized data weaving path for data weaving of multi-source heterogeneous data.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to an adaptive data weaving performance optimization method based on artificial intelligence. Background Technology

[0002] In the process of enterprise digital transformation, the integration and efficient utilization of multi-source heterogeneous data has become a core requirement for supporting business decisions and improving operational efficiency. Currently, enterprise data sources exhibit significant heterogeneity, encompassing relational databases (such as MySQL and Oracle), NoSQL databases (such as MongoDB and Redis), unstructured files (such as CSV and JSON documents), and real-time data streams from IoT devices. This data needs to be integrated across sources through data weaving technology to meet the access requirements of a unified data view for diverse business scenarios such as real-time transaction analysis, batch data statistics, and static report queries. Especially in fields such as finance and e-commerce, higher requirements are placed on the real-time performance, stability, and resource adaptability of data weaving.

[0003] Existing data weaving solutions mostly adopt a static approach: First, they rely on manual sorting of the basic attributes of the data source (such as storage location, data format, and other technical information) and manually configuring the correspondence between the data source and the data access interface; then, they construct the data weaving path based on preset fixed path rules (such as allocating fixed transmission protocols and processing nodes according to the data source type); at the resource scheduling level, they usually pre-allocate fixed computing and storage resources according to business scenarios, such as allocating 50% of memory resources for batch data processing scenarios, and rarely make dynamic adjustments after the path and resource allocation strategy is determined, lacking the ability to adaptively respond to real-time business changes and system status.

[0004] However, traditional solutions suffer from several drawbacks. First, their metadata processing is limited to a single dimension, focusing solely on technical metadata without integrating key information such as business semantics (e.g., the business domain to which the data belongs and business priority), operation records (e.g., historical access frequency and data processing time), and management specifications (e.g., data sensitivity level and compliance requirements). This results in the constructed weaving path being unable to adapt to complex and ever-changing business needs, for example, failing to dynamically adjust data processing priorities based on business urgency. Second, resource scheduling relies on static rules, making it impossible to monitor real-time load changes in the data access task queue (e.g., sudden increases in task volume, high-priority task queue jumping) and system resource indicators (e.g., CPU utilization and memory usage). When resource bottlenecks or task load fluctuations occur, resource allocation and path planning cannot be adjusted in a timely manner, easily leading to increased data access latency, low resource utilization, and even interruptions in the data weaving process. This makes it difficult to meet the actual needs of enterprises for efficient, stable, and adaptive integration of multi-source heterogeneous data. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides an artificial intelligence-based adaptive data weaving performance optimization method to at least alleviate these problems.

[0006] The technical solutions provided in this application are as follows:

[0007] An AI-based adaptive data weaving performance optimization method, comprising:

[0008] Step 1: Obtain multi-source heterogeneous data based on the data source list and parse it to generate a meta-dataset that contains at least technical metadata, business metadata, operational metadata, and management metadata;

[0009] Step 2: Based on the virtual table interface, perform virtual table abstraction and encapsulation on the metadata dataset to generate a logical data carrier;

[0010] Step 3: Determine the operational attributes of the metadata dataset to generate a data weaving path for the logical data carrier based on the operational attributes;

[0011] Step 4: Monitor the data access task queue and access indicator feedback in real time, extract resource load correlation features from the data access task queue and access indicator feedback based on the constructed resource scheduling model, and generate resource scheduling strategies based on the resource load correlation features.

[0012] Step 5: Optimize the data weaving path of the logical data carrier based on the resource scheduling strategy to obtain the optimized data weaving path for data weaving of multi-source heterogeneous data.

[0013] This application acquires multi-source heterogeneous data based on a data source list and parses it to generate a metadata dataset containing technical metadata, business metadata, operational metadata, and management metadata. Compared to the single technical metadata traditionally compiled manually, this dataset supplements information such as the data's business domain, business priority (business metadata), historical access frequency, data processing time (operational metadata), data sensitivity level, and compliance requirements (management metadata), providing a more comprehensive decision-making basis for the subsequent generation of data weaving paths. For example, based on the "real-time transaction business priority" in the business metadata and the "high-frequency access records" in the operational metadata, better processing resources can be allocated to data in real-time transaction scenarios. This avoids the shortcomings of traditional solutions that cannot dynamically adjust according to business urgency due to a lack of business and operational dimension information, making the subsequently generated paths more closely aligned with actual business needs.

[0014] Furthermore, this application uses a virtual table interface to abstract and encapsulate the metadata dataset to generate a logical data carrier. This eliminates the need for manually establishing the correspondence between data sources and interfaces. Instead, it dynamically maps metadata fields to the virtual table structure, shielding the physical differences (such as database type and file format) of the underlying multi-source heterogeneous data from the logical layer. Compared to traditional manual configuration, this approach not only reduces manual operation costs but also forms a unified logical data access layer. This allows subsequent data weaving path generation to disregard the physical attributes of specific data sources, relying solely on standardized logical carrier planning. This solves the problems of poor interface adaptability and high maintenance costs associated with traditional interfaces, while providing a standardized foundation for path generation.

[0015] In this application, the operational attributes of the metadata dataset are determined (these operational attributes can be derived from the operational metadata and business metadata in step 1, such as determining operational attributes like "real-time read / write" and "batch processing" by combining historical data processing frequency and business SLA requirements). Then, a data weaving path for the logical data carrier is generated based on these operational attributes. Compared to traditional fixed paths, the generated path can be adapted to the differentiated operational attributes. For example, low-latency transmission protocols and high-frequency processing nodes can be allocated to "real-time read / write" operational attributes, while high-throughput nodes can be allocated to "batch processing" operational attributes. This solves the problem that traditional paths cannot adapt to the needs of different business scenarios, making the resource consumption of the path more aligned with business requirements.

[0016] In this application, by monitoring data access to the task queue and access metric feedback in real time, a resource scheduling model is constructed to extract resource load correlation features from the two (such as associating "high-priority task ratio of 30%" with "CPU utilization of 85%)," thereby generating a resource scheduling strategy. Compared with traditional static resource allocation, this method can perceive system status and task load fluctuations in real time. For example, when a sudden increase in the length of the real-time transaction task queue and an increase in CPU utilization are detected, resource bottlenecks can be identified in advance and a strategy of "prioritizing the allocation of CPU resources to real-time tasks" can be generated. This solves the problem that traditional methods cannot respond to load changes in real time and provides timely and accurate scheduling basis for subsequent path optimization.

[0017] In this application, the data weaving path of the logical data carrier is optimized based on the generated resource scheduling strategy. Compared with the traditional fixed path, the path can be adjusted in a targeted manner when resource bottlenecks or load changes occur. For example, when a node is overloaded, the tasks on that node are migrated to a low-load node; when batch tasks consume too much memory, the memory usage of the batch data processing path is optimized. Through dynamic optimization, data access latency can be reduced, resource utilization can be improved, and process interruption problems caused by the lack of adjustment in the traditional path can be avoided, ensuring that the weaving of multi-source heterogeneous data can maintain better performance and stability under different load scenarios. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating the performance optimization method for adaptive data weaving based on artificial intelligence, as described in an embodiment of this application. Detailed Implementation

[0019] like Figure 1 As shown in the embodiment of this application, an adaptive data weaving performance optimization method based on artificial intelligence is provided, the method comprising:

[0020] Step 1: Obtain multi-source heterogeneous data based on the data source list and parse it to generate a meta-dataset that contains at least technical metadata, business metadata, operational metadata, and management metadata;

[0021] Step 2: Based on the virtual table interface, perform virtual table abstraction and encapsulation on the metadata dataset to generate a logical data carrier;

[0022] Step 3: Determine the operational attributes of the metadata dataset to generate a data weaving path for the logical data carrier based on the operational attributes;

[0023] Step 4: Monitor the data access task queue and access indicator feedback in real time, extract resource load correlation features from the data access task queue and access indicator feedback based on the constructed resource scheduling model, and generate resource scheduling strategies based on the resource load correlation features.

[0024] Step 5: Optimize the data weaving path of the logical data carrier based on the resource scheduling strategy to obtain the optimized data weaving path for data weaving of multi-source heterogeneous data.

[0025] Optionally, step 1 specifically includes the following steps:

[0026] Step 11: Perform type parsing on the multi-source heterogeneous data sources in the preset data source list to identify the storage engine type and transmission protocol type, and generate a detailed list of data source types accordingly;

[0027] Step 12: Based on the detailed list of data source types, perform protocol adaptation on multi-source heterogeneous data sources to obtain multi-source heterogeneous data;

[0028] Step 13: Perform data field segmentation on the multi-source heterogeneous data to obtain a structured field set. Perform metadata parsing on the structured field set to generate a metadata dataset that includes at least technical metadata, business metadata, operational metadata, and management metadata.

[0029] Optionally, step 11 specifically includes the following steps:

[0030] Step 111: Based on the set data source matching rule base, construct a type parser. Based on the constructed type parser, perform type parsing on the multi-source heterogeneous data sources in the preset data source list and obtain the data source type description.

[0031] Step 112: Extract features from the data source type description to generate a data source type feature encoding table;

[0032] Step 113: According to the set storage protocol encoding mapping rules, perform type matching and verification processing on the data source type feature encoding table to identify the storage engine type and transmission protocol type, and generate a detailed list of data source types accordingly.

[0033] Preferably, the specific implementation process of step 111 is as follows: First, the set data source matching rule base is subjected to structured initialization processing. This rule base contains core identification rules for multi-source heterogeneous data sources. Among them, the technical rules cover data source identifier prefixes (such as "mysql: / / " for relational databases, "mongodb: / / " for NoSQL databases, and "mqtt: / / " for IoT stream data sources), default port ranges (MySQL default port 3306, MongoDB default port 27017, and MQTT default port 1883), feature field identifiers (such as the "information_schema" system table field for relational databases and the "collection" collection field for NoSQL databases), and business rules cover the business domain to which the data source belongs (such as "order system - relational" and "device monitoring - IoT stream"), so as to generate a structured data source matching rule base. Then, an extensible rule engine-style type parser is built based on this structured rule base. This parser is different from the traditional fixed rule parser and supports dynamically adding new data source rules through JSON configuration (such as adding "clickhouse"). The ": / / " identifier corresponds to the time-series database rule, requiring no modification to the underlying code to adapt to the enterprise's data source type expansion needs. Subsequently, multi-source heterogeneous data sources (such as "mysql: / / 192.168.1.100:3306 / order_db" and "mqtt: / / 192.168.1.200:1883 / device_data") from the preset data source list are input into the parser for this type. The parser performs parsing in the order of "identifier prefix matching → port verification → feature field detection". For example, for "mysql: / / 192.168.1.100:3306 / order_db", it first matches the "mysql: / / " prefix, then verifies whether the 3306 port is within the MySQL default port range, and finally detects whether the "information_schema" field exists. Finally, it obtains a data source type description containing the data source ID, identifier prefix, port, and feature field detection results (such as "DS001, identifier prefix: mysql: / / , port: 3306, feature field: information_schema exists, preliminary judgment: relational database").

[0034] Preferably, in the scenario of batch parsing multiple heterogeneous data sources in an enterprise, step 112 is specifically implemented as follows: First, taking the data source type description generated in step 111 as the processing object, perform multi-dimensional feature extraction processing on each data source type description—extracting technical features (storage type: determined as relational / non-relational / file / stream data type based on the identifier prefix and feature fields; initial judgment of transmission protocol: determined as TCP / UDP / MQTT / HTTP based on the identifier prefix), and attribute features (whether transactions are supported: relational databases support transactions by default, non-relational databases do not support transactions by default; whether it is a real-time stream: MQTT identifier corresponds to streaming data, file identifier corresponds to non-stream data), to obtain a multi-dimensional feature set for each data source (e.g., "DS001, storage type: relational, initial judgment of transmission protocol: TCP, supports transactions: yes, real-time stream: no"); then, use a multi-dimensional feature hierarchical encoding scheme to encode the multi-dimensional feature set. This scheme differs from traditional single encoding by dividing the features into 3 encoding levels, totaling 32 bits of binary encoding—the first 8 bits are the storage type encoding (00000001=relational). The first 8 bits are the initial encoding for the transmission protocol (00000001=TCP, 00000010=UDP, 00000100=MQTT, 00001000=HTTP), and the second 16 bits are the attribute feature encoding (000000000000001=Transactional Support, 000000000000010=Real-time Stream, the remaining bits are reserved for future expansion). Finally, each data source... The encoding results are organized into a column structure of "data source ID, storage type encoding, transmission protocol initial judgment encoding, and attribute feature encoding" to generate a data source type feature encoding table. Each row in this table corresponds to a data source, and the intersection of the row and column is the specific encoding value. For example, the data source ID "DS001" corresponds to the storage type encoding "00000001", the transmission protocol initial judgment encoding "00000001", and the attribute feature encoding "0000000000000001", clearly mapping the characteristics of "relational database, TCP transmission, and support for transactions".

[0035] Preferably, in the specific technical implementation of step 113: First, a preset storage protocol encoding mapping rule is loaded. This rule is a bidirectional mapping table—forward mapping is "feature code → specific type" (storage type code 00000001 corresponds to relational storage engines such as MySQL / Oracle, 00000010 corresponds to non-relational storage engines such as MongoDB / Redis; initial transmission protocol judgment code 00000001 corresponds to JDBC / ODBC protocol, 00000100 corresponds to MQTT protocol), and reverse mapping is "specific type → feature code" (MySQL corresponds to storage type code 00000001, JDBC protocol corresponds to transmission protocol code 00000001), and also includes a verification rule (e.g., when the storage engine is MySQL, the transmission protocol must support JDBC / ODBC; if MQTT is matched, it is judged as abnormal); then, based on this mapping rule, type matching processing is performed on the data source type feature encoding table generated in step 112—the encoding table is... The storage type encoding is matched forward with the mapping rules to obtain storage engine type candidates (e.g., encoding 00000001 matches "MySQL, Oracle"). The transport protocol encoding is then matched forward to obtain transport protocol type candidates (e.g., encoding 00000001 matches "JDBC, ODBC"). Subsequently, bidirectional verification is performed: first, based on reverse mapping, the encoding accuracy is verified (e.g., if the reverse mapping encoding of the storage engine candidate "MySQL" is 00000001, and it matches the encoding table, the verification passes); second, based on the verification rules, type compatibility is verified (e.g., if the storage engine "MySQL" is compatible with the transport protocol "JDBC", the verification passes; if the transport protocol candidate is "MQTT", it is marked as incompatible). Abnormal encodings are supplemented with rules or manually confirmed (e.g., adding a special mapping rule for "MySQL+MQTT"). Finally, the verified storage engine type and transport protocol type are associated with the data source ID, and organized into a format containing "data source ID, storage engine type (e.g., MySQL...)". A detailed list of data source types, including "8.0), transmission protocol type (e.g., JDBC 4.2), verification status (passed), and remarks (no anomalies)".

[0036] Optionally, step 12 specifically includes the following steps:

[0037] Step 121: Perform structured parsing on the detailed list of data source types to generate a basic information table for data source adaptation;

[0038] Step 122: Perform template matching processing between the data source adaptation basic information table and the preset "storage engine type - transmission protocol type" two-dimensional pre-configured protocol adaptation template library to generate a protocol adaptation template;

[0039] Step 123: Based on the protocol adaptation template, perform protocol handshake and data reading processing on the multi-source heterogeneous data sources to obtain multi-source heterogeneous data, and perform log writing processing on the key adaptation information to generate an adaptation log table.

[0040] Preferably, the specific implementation process of step 121 is as follows: First, the data source type details list generated in step 113 is used as the processing object. This list contains fields such as "data source ID, storage engine type (e.g., MySQL 8.0, MongoDB 6.0), transport protocol type (e.g., JDBC 4.2, MQTT 3.1.1), and verification status". Structured parsing is performed on this list to extract the core adaptation elements for each type of data source—including the unique identifier of the data source (data source ID), the detailed model of the storage engine (to distinguish different versions, such as the differences in adaptation parameters between MySQL 8.0 and MySQL 5.7), and the transport protocol version (to specify the protocol version number to match the corresponding connection parameters, such as the SSL configuration supported by JDBC 4.2 and JDBC 5.1.1). The table first extracts the core adaptation elements, then organizes them into a data source adaptation basic information table. This table includes the following: Data source ID (different from version 4.0), data source physical address (IP address + port number + target database / collection / topic name, such as "192.168.1.100:3306 / order_db" "192.168.1.200:1883 / device_topic"), and authentication information identifiers (marking whether username / password authentication is required, key file path, or token validity period). Next, the extracted core adaptation elements are organized into a column structure of "data source ID, storage engine model, transport protocol version, physical address, and authentication information identifier." Each row in this table corresponds to a data source, and the intersection of rows and columns represents the specific adaptation element value (e.g., the storage engine model "MySQL 8.0" and transport protocol version "JDBC" corresponding to data source ID "DS001"). 4.2”, physical address “192.168.1.100:3306 / order_db”, authentication information identifier “username and password required, path: / conf / auth / mysql.xml”); This technology differs from the traditional parsing method that only extracts the basic combination of “storage engine-protocol”, and adds the elements of “protocol version” and “authentication information identifier”. Because the connection parameters of different protocol versions (such as the “serverTimezone” parameter in JDBC 4.2, which is not present in JDBC 4.0) and authentication methods will directly affect the adaptation success rate, parsing these elements in advance can reduce the number of parameter debugging times in the subsequent adaptation process and improve adaptation efficiency.

[0041] Preferably, in the scenario of batch protocol adaptation for enterprise multi-source heterogeneous data sources (including relational databases, NoSQL databases, and IoT streaming data sources), step 122 is specifically implemented as follows: First, load the preset "storage engine type - transport protocol type" two-dimensional pre-configured protocol adaptation template library. The special design of this template library is that the row dimension is the storage engine type (subdivided into subtypes such as MySQL, Oracle, MongoDB, Kafka, MQTT devices, etc.), and the column dimension is the transport protocol type (subdivided into subtypes such as JDBC, ODBC, MongoDB Driver, Kafka Consumer API, MQTT Client, etc.). The intersection of the row and column is the exclusive protocol adaptation template for the corresponding combination. Each template contains three core parameters: connection parameters (such as the MySQL+JDBC template's "url format: jdbc:mysql: / / {IP}:{Port} / {DB}?useSSL={Bool}&serverTimezone={Timezone}", and the default timeout value "30s"), and data reading parameters (such as MongoDB+MongoDB). The Driver template's batch size is set to 1000 records per batch, field projection rules are set to "return all fields by default, but the _id field can be excluded", and exception handling parameters are set (e.g., "retry count: 3 times" and "retry interval: 5s" for the MQTT device + MQTT Client template). Next, using the data source adaptation basic information table generated in step 121 as the matching basis, the "storage engine model" and "transmission protocol version" of each row in the table are extracted. First, the template library's row dimension is matched by "storage engine model" (e.g., MySQL 8.0 matches the "MySQL" row), then the column dimension is matched by "transmission protocol version" (e.g., JDBC 4.2 matches the "JDBC" column). If a completely matching template exists, it is directly called. If only the storage engine type matches but the protocol version does not (e.g., there is no dedicated template for JDBC 4.2), then the general protocol template under that storage engine type (e.g., the MySQL + JDBC general template) is called, and version compatibility parameters are automatically added (e.g., adding "serverTimezone=UTC" to the general template to adapt to JDBC). 4.2); Finally, a protocol adaptation template with exclusive parameters is generated for each data source. This technology differs from the traditional single-dimensional (only by storage engine or only by protocol) template library. The dual-dimensional design can accurately cover the combination scenario of "storage engine-protocol", avoiding the adaptation failure caused by the traditional single-dimensional template due to the failure to distinguish the protocol type (such as the difference in adaptation parameters between JDBC and ODBC for MySQL), and improving the accuracy of template matching.

[0042] Preferably, in the specific technical implementation of step 123: First, based on the protocol adaptation template generated in step 122, a protocol handshake is initiated for each multi-source heterogeneous data source—the connection parameters in the template (such as the URL, username, and password of the MySQL+JDBC template) are called to construct a connection request, and a protocol version negotiation instruction is sent to the data source (such as sending the "handshake_v10" instruction for JDBC connections and sending a "CONNECT" message for MQTT connections). After receiving the response message returned by the data source, it is verified whether the response conforms to the template's preset success identifier (such as "OK_Packet" returned by JDBC connections and "CONNACK" message returned by MQTT connections with a return code of 0). If the verification passes, the protocol handshake is completed; if it fails, the exception handling parameters in the template are triggered (such as retrying 3 times with a 5-second interval). After the protocol handshake is successful, the data reading parameters in the template are called to execute data reading processing—for relational databases (such as MySQL), table data is read in batches according to the "batch size," and for NoSQL databases (such as MongoDB), the data reading parameters are read in batches according to the "batch size." After filtering fields according to "field projection rules", the database (DB) reads the data. For IoT stream data sources (such as MQTT devices), it receives streaming data in real time according to "subscribed topics", and records the reading time and data volume of each batch of data. Then, it writes the adaptation key information (data source ID, protocol handshake start time, handshake end time, handshake result, data reading start time, reading end time, data volume, reading time, and exception information (empty if there is no exception)) into a log file in a preset format to generate an adaptation log table. Each row of this table corresponds to a single adaptation process of a data source, and the columns correspond to the above key information, which facilitates subsequent operation metadata statistics (such as extracting "historical access delay" of operation metadata in step 13). This technology is different from the traditional adaptation method that only completes data reading. It integrates the parameter verification of protocol handshake, the performance statistics of data reading and log recording into a single design. This not only ensures the standardization of the adaptation process, but also provides basic data for subsequent metadata parsing and resource scheduling (such as the reading time can be used as the basis for calculating the resource load correlation characteristics in step 4), avoiding redundant operations that require additional collection of this information later.

[0043] Optionally, step 13 specifically includes the following steps:

[0044] Step 131: Perform field splitting on the structured data in the multi-source heterogeneous data according to the preset field delimiter; extract nested fields for the semi-structured data according to the JSON / XML tag path; and locate the key field positions and perform field separation for the unstructured data to generate structured field units.

[0045] Step 132: Extract and map metadata from the structured field units to generate a metadata-related ledger;

[0046] Step 133: Perform metadata completion and integration verification on the metadata-related ledger to generate a metadata dataset that includes at least technical metadata, business metadata, operational metadata, and management metadata.

[0047] Preferably, in the scenario of standardizing fields in multi-source heterogeneous data (including structured transaction data such as MySQL order tables, semi-structured log data such as JSON format system logs, and unstructured document data such as PDF format customer contracts), step 131 is specifically implemented as follows: First, taking the multi-source heterogeneous data obtained in step 123 as the processing object, firstly filter out the structured data (such as CSV files with fields separated by commas, and MySQL data stored according to a fixed table structure), and call the preset field separator configuration library (including common separators such as commas, tabs, and vertical lines, which can automatically match according to the data source type). For example, CSV uses default commas and TSV uses default tabs. This allows for field splitting of structured data. For instance, CSV data in the format "Order ID, User ID, Order Amount, Order Time" can be split into four independent fields: "Order ID," "User ID," "Order Amount," and "Order Time," separated by commas, resulting in structured split fields. Next, for semi-structured data (such as JSON data with nested structures `{"order":{"id":"O001","user":{"id":"U001","name":"Zhang San"}}}`, and XML data), further processing is applied. <order> <id> O001< / id> <user> <id> U001< / id> < / user> < / order>The system extracts nested fields according to preset JSON / XML tag path rules (JSON paths such as "$.order.id" and "$.order.user.id", XML paths such as " / order / id" and " / order / user / id"). It then converts the nested "order.id" and "order.user.id" fields into flattened "order_id" and "user_id" fields, resulting in semi-structured extracted fields. Subsequently, for unstructured data (such as PDF contracts containing key fields like "contract number," "contractor," and "validity period"), it calls the key field localization module based on a BERT pre-trained model (this module has been fine-tuned using historical enterprise contract text and can recognize target fields after prefixes such as "contract number:" and "contractor:"). This technology performs text parsing on unstructured data (converting PDFs into recognizable text), then uses a positioning module to match key field prefixes and extract the content of the fields after the prefixes (e.g., "Contract Number: HT20240801" is extracted as "Contract Number = HT20240801"), resulting in unstructured extracted fields. Finally, the structured split fields, semi-structured extracted fields, and unstructured extracted fields are integrated according to the format "Data Source ID - Field Name - Field Value - Field Source Type (Structured / Semi-structured / Unstructured)" to generate structured field units. This technology differs from traditional methods that only process structured data. It covers three types of heterogeneous data through targeted field processing logic, especially the pre-trained model positioning scheme for unstructured data, solving the problem of difficulty in extracting fixed fields from traditional unstructured data and improving the structuring rate of multi-source data.

[0048] Preferably, in the specific technical implementation of step 132: First, taking the structured field unit generated in step 131 as the processing object, perform multi-dimensional metadata extraction processing on each structured field unit—extracting technical metadata (field data type such as int / varchar / date, field length such as order ID being 10 digits, whether it is a primary key / foreign key such as "order ID" being a primary key), business metadata (field business meaning such as "order amount" corresponding to "customer's actual payment amount", the business domain to which the data belongs such as "order domain", the business person in charge such as "order system operation and maintenance group"), and operation metadata (field historical access frequency such as "user ID" being accessed an average of 1000 times per day, the last update time such as 2024-08-27). 10:30) Manage metadata (field sensitivity level such as "User ID" is high sensitivity, data retention period such as 3 years), to obtain a single-field metadata set; then, load the enterprise's preset business term association library (including semantic mapping relationships of business fields such as "cust_id-Customer ID" and "order_amt-Order Amount"), and perform semantic matching processing on the field names in the single-field metadata set with the business term association library. For example, establish a mapping relationship between the "cust_id" field in the structured field unit and the "Customer ID" in the association library, and at the same time associate the same source field of this field in different data sources (such as "cust_id" in the MySQL order table and "customer_id" in the MongoDB customer table) to obtain the field association mapping result; Finally, the single-field metadata set and the field association mapping results are organized into a column structure of "field unique identifier (data source ID + field name) - technical metadata - business metadata - operation metadata - management metadata - associated field identifier" to generate a metadata association ledger. Each row in this ledger corresponds to the metadata and association information of a structured field unit, and the intersection of the row and column is the specific metadata value (e.g., the business meaning of the field unique identifier "DS001-order ID" is "unique identifier of the customer's order record number"). This technology is different from the traditional method of only extracting technical metadata. It simultaneously covers four types of metadata and adds business semantic association, providing basic information on business and operation dimensions for subsequent operation attribute determination (step 31), and avoiding semantic deviation caused by the traditional metadata association relying only on field names.

[0049] Preferably, the specific implementation process of step 133 is as follows: First, taking the metadata-related ledger generated in step 132 as the processing object, perform metadata completion processing—load the metadata completion rule base (including rules such as "If the field is 'amount type' (business metadata judgment), then the technical metadata needs to be supplemented with 'precision such as 2 decimal places'", "If the field sensitivity level is high sensitivity (management metadata), then 'encryption algorithm type such as SM4' needs to be supplemented", "If the field historical access frequency is > 500 times / day (operation metadata), then 'cache validity period such as 1 hour' needs to be supplemented", etc.), traverse each record in the ledger, and for missing metadata items (such as a certain " (For example, if the "Amount" field lacks precision or a highly sensitive field lacks an encryption algorithm, the information is automatically supplemented according to the rule base. For instance, the "Precision = 2 decimal places" field in the technical metadata is supplemented for the "Order Amount" field, and the "Encryption Algorithm Type = SM4" field in the management metadata is supplemented for the "User ID" field, resulting in a supplemented metadata-related ledger. Next, an integration and verification process is performed on the supplemented metadata-related ledger. The verification is divided into three categories: first, cross-metadata dimension consistency verification (e.g., whether the "Field Type = date" in the technical metadata matches the "Business Meaning = Order Time" in the business metadata; if the field type is int, it is considered inconsistent); second... The system performs three main checks: 1) Integrity verification of associated fields (e.g., in a ledger, "Order ID" is associated with "User ID," requiring verification of the completeness of the "User ID" metadata; if the "User ID" is missing a sensitive level, it is marked as abnormal); 2) Compliance verification of management metadata (e.g., whether highly sensitive fields are all supplemented with encryption algorithms, and whether the data retention period meets the company's compliance requirements, such as no less than 2 years). For verification anomalies (e.g., field types are inconsistent with business meanings, or highly sensitive fields lack encryption algorithms), the anomaly repair module is invoked (e.g., automatically correcting the int type "Order Time" field to date type, and supplementing highly sensitive fields without encryption algorithms with default SM4). The algorithm obtains a verified and repaired metadata association ledger. Finally, the verified and repaired metadata association ledger is reorganized according to four dimensions: "technical metadata - business metadata - operation metadata - management metadata". Redundant fields (such as duplicate association field identifiers) are deleted, and a metadata dataset containing at least four types of metadata is generated. This technology differs from the traditional manual completion and verification method. Through automatic completion of the rule base and multi-dimensional verification, it improves the integrity and consistency of metadata and avoids the subsequent path planning deviation caused by omissions in traditional metadata due to manual operation (such as missing sensitive level information when generating the path in step 3, making it impossible to add encrypted nodes).

[0050] Optionally, step 2 specifically includes the following steps:

[0051] Step 21: Based on the virtual table interface, perform dynamic mapping configuration processing on the metadata dataset between metadata fields and virtual table structure to obtain the metadata-virtual table field mapping relationship;

[0052] Step 22: Perform virtual table structure abstraction and construction on the metadata-virtual table field mapping relationship to generate a virtual table structure definition file;

[0053] Step 23: Perform interface adaptation and functional encapsulation processing on the virtual table structure definition file to generate a logical data carrier.

[0054] Optionally, step 21 involves configuring dynamic mapping between metadata fields and virtual table structures on the metadata dataset based on the virtual table interface to obtain the metadata-virtual table field mapping relationship. This includes the following steps:

[0055] Step 211: Based on the metadata parsing specification of the virtual table interface, perform field attribute extraction processing on the metadata dataset to generate metadata field attribute features;

[0056] Step 212: Based on the attribute characteristics of the metadata fields, dynamically match them with the virtual table fields to generate a candidate set of virtual table mappings;

[0057] Step 213: Based on the initial mapping candidate set of the virtual table, generate the metadata-virtual table field mapping relationship.

[0058] Preferably, in the scenario of adapting enterprise multi-source heterogeneous metadata (including metadata from multiple business domains such as order domain, customer domain, and equipment domain) to the virtual table interface, step 211 is specifically implemented as follows: First, the metadata dataset generated in step 133 is used as the processing object. This metadata dataset contains technical metadata, business metadata, operational metadata, and management metadata. At the same time, the metadata parsing specification preset by the virtual table interface is loaded. This specification differs from the traditional basic specification that only defines "field name-data type" and adds three parsing dimensions: First, the basic attribute dimension of the field (requiring the extraction of field name, data type, length, precision, and whether null is allowed); second, the business association attribute dimension (requiring the extraction of the business domain to which the field belongs, business term tags, and associated business process IDs); and third, the... Operational characteristic attribute dimensions (requirements to extract average access frequency, read / write operation ratio, and last update time); then, perform attribute extraction processing on each field in the metadata set according to this specification: extract basic field attributes from technical metadata (e.g., for the "Order Amount" field, name = order amount, data type = decimal, length = 10, precision = 2, nullable = no); extract business-related attributes from business metadata (e.g., business domain = order domain, business term tag = transaction amount, associated business process ID = P001 order placement process); extract operational characteristic attributes from operational metadata (e.g., average access frequency = 500 times / day, read / write operation ratio = read:write = 8:2, last update time = 2024-08-27). 09:15); Finally, the three types of extracted attributes are integrated according to the structure of "field unique identifier (data source ID + field name) - basic attribute - business association attribute - operation characteristic attribute" to generate a metadata field attribute feature table. Each row in this table corresponds to a metadata field, and the intersection of the row and column is the specific attribute value (e.g., the business domain corresponding to the field unique identifier "DS001-order amount" is the order domain, and the average access frequency is 500 times / day). This technology provides richer decision-making basis for subsequent virtual table field matching by adding attribute extraction of business and operation dimensions, avoiding the matching deviation of "fields with the same name but different business meanings" caused by the traditional reliance on basic attributes (e.g., "customer ID" has different business meanings in the order domain and logistics domain, which need to be distinguished by business association attributes).

[0059] Preferably, in the specific technical implementation of step 212: First, the virtual table field structure library preset by the virtual table interface is loaded. This structure library contains multiple virtual tables (such as "virtual order table", "virtual customer table", and "virtual device status table"). Each virtual table field has preset adaptation features (including field identifier, adapted data type range (such as int / long compatibility), business domain label, and allowed metadata field source). For example, the adaptation features of the "virtual order table - order amount" field are "field identifier = VT001_OrderAmt, adapted data type range = decimal(10,2) / decimal(12,2), business domain label = order domain, allowed source = MySQL order table / ERP order table". Next, based on the metadata field attribute feature table generated in step 211, multi-dimensional dynamic matching processing is performed—first, a preliminary screening is performed according to the "business domain label" (such as only matching metadata fields with business domain = order domain with fields in the "virtual order table"), and then a verification is performed according to the "adapted data type range" (such as the d of the metadata field "order amount"). If the ecimal(10,2) type is within the adaptation range of VT001_OrderAmt, then it passes. Finally, the matching degree is calculated according to the "operation characteristic attribute" (e.g., if the average access frequency of the metadata field matches the preset "high-frequency access / low-frequency access" label of the virtual table field, the matching degree is increased by 10%), and the matching result of each metadata field and the virtual table field is obtained (including the matched virtual table ID, virtual table field identifier, and multi-dimensional matching degree). Then, the matching results are sorted according to "unique identifier of metadata field - matching virtual table information - matching degree", and the results with a matching degree of less than 60% are filtered out (the threshold can be adjusted through configuration) to generate a virtual table mapping candidate set, such as "DS001-order amount, matching virtual table ID: VT001, virtual table field identifier: VT001_OrderAmt, matching degree: 85%". This technology is different from the traditional single-dimensional matching based only on "field name + data type". Through multi-dimensional matching of "business domain - data type - operation characteristic", it significantly reduces mismatches caused by similar field names but different business meanings, and improves the rationality of the matching results.

[0060] Preferably, the specific implementation process of step 213 is as follows: First, taking the virtual table mapping candidate set generated in step 212 as the processing object, perform candidate set conflict detection and resolution processing—traverse the candidate set. If there are multiple virtual table fields corresponding to the same metadata field unique identifier (conflict scenario, such as "DS002-Customer ID" matching both "VT002_Virtual Customer Table-CustID" and "VT003_Virtual Logistics Table-CustomerID"), then call the conflict resolution rule base: Rule 1 prioritizes the virtual table field with "completely consistent business domain labels" (e.g., the business domain of "DS002-Customer ID" is the customer domain, and only the business domain label of "VT002_Virtual Customer Table" is the customer domain, then the field is matched first); Rule 2 if there are still multiple matching results even if the business domains are consistent, then the "access frequency" of the operation metadata is weighted (e.g., the daily average access of "DS002-Customer ID" is 1000 times, which matches the "high frequency access" label of "VT002_CustID", and the weighted matching degree is higher than "VT003_C If the "Metadata ID" is selected as the "Medium-Frequency Access" tag, then select the former. If rules 1 and 2 still cannot resolve the conflict, then mark it as "Pending Manual Confirmation" and generate a conflict alert log. Next, perform a uniqueness check on the resolved candidate set to ensure that each metadata field corresponds to only one virtual table field and each virtual table field can correspond to multiple metadata fields from the same source (e.g., the "Order Amount" field from different data sources can be mapped to "VT001_OrderAmt"). Finally, organize the verified candidate set according to the column structure of "Metadata Field Unique Identifier - Virtual Table ID - Virtual Table Field Identifier - Matching Degree - Matching Status (Automatic Matching / Manual Confirmation)" to generate a metadata-virtual table field mapping relationship table, which can be directly used for subsequent virtual table structure construction (step 22). This technology solves the "one-to-many" conflict problem in traditional automatic matching by combining conflict resolution rules with manual confirmation, while retaining the manual intervention entry point, taking into account both automation efficiency and adaptability flexibility in special scenarios, and ensuring that the generated mapping relationship has high availability.

[0061] Optionally, step 22 specifically includes the following steps:

[0062] Step 221: Based on the metadata-virtual table field mapping relationship and the technical metadata in the metadata set, perform attribute completion and constraint judgment processing on the virtual table fields to generate virtual table field attribute details;

[0063] Step 222: Based on the virtual table field attribute details, perform multi-source field association logic sorting on the metadata-virtual table field mapping relationship to generate a virtual table field association rule table;

[0064] Step 223: Based on the virtual table field attribute details and the virtual table field association rule table, perform structural framework construction processing on the virtual table to generate the virtual table structural framework;

[0065] Step 224: Based on the pre-defined structure definition specifications of the virtual table interface, perform format verification and standardization on the virtual table structure framework to generate a virtual table structure definition file.

[0066] Preferably, in the scenario of constructing a unified virtual table from multi-source heterogeneous metadata (such as MySQL order table, MongoDB customer table, IoT device log table), step 221 is specifically implemented as follows: First, the metadata-virtual table field mapping relationship generated in step 213 is used as the core processing object. This mapping relationship includes the correspondence of "metadata field unique identifier - virtual table ID - virtual table field identifier". At the same time, the technical metadata (including field primary key / foreign key identifiers, default values, and data validation rules) in the metadata set of step 133 is loaded. Then, grouped by virtual table ID, all virtual table fields under each virtual table are extracted, and attribute completion processing is performed on each virtual table field—extracting and integrating multi-source technical metadata from the metadata fields associated with the mapping relationship. Attributes (such as the "Order ID" field in the virtual table "VT001_Order Table", which relates the primary key identifier of the MySQL order table "order_id" and the index identifier of the MongoDB order log table "order_id", completed as "virtual table field primary key identifier = yes, index identifier = yes") are added, along with virtual table-specific attributes (such as the display order of the field in the virtual table, and whether it is a virtual calculated field (such as "total order amount = unit price × quantity")). Subsequently, constraint judgment processing is performed—based on field validation rules of technical metadata (such as "order amount > 0") and business constraints of business metadata (such as "when the customer level is VIP, the minimum order amount = 100 yuan"), generating combined constraint conditions for the virtual table fields (such as "order amount > 0"). And (customer level ≠ VIP or order amount ≥ 100 yuan)); finally, the completed attributes and the judgment constraints are organized according to "virtual table ID - virtual table field identifier - basic attributes (data type / length, etc.) - associated metadata attributes (primary key / foreign key, etc.) - virtual table exclusive attributes - combined constraints" to generate virtual table field attribute details; this technology is different from the traditional method of metadata completion based on a single source technology. By integrating multi-source metadata attributes and business constraints, it ensures that virtual table field attributes are compatible with the characteristics of multi-source data sources and meet the actual business needs, avoiding the usage limitations caused by the single attribute of traditional virtual table fields.

[0067] Preferably, in the specific technical implementation of step 222: First, taking the virtual table field attribute details generated in step 221 as the processing object, all fields within the same virtual table are filtered out, and the metadata association ledger generated in step 132 (including business semantic association relationships between fields, such as "Order ID-Order Item ID" and "Customer ID-Order ID") is loaded; then, multi-source field association logic sorting processing is performed, which is divided into two types of association sorting: one is technical association sorting—based on the primary key / foreign key identifiers in the virtual table field attribute details, the technical dependency relationships between fields within the virtual table are identified (e.g., "Order ID" is the primary key in "VT001_Order Table", and "Order ID" is the foreign key in "VT002_Order Item Table", sorted into "VT002_Order Item Table.Order ID"). The first step is to establish a foreign key relationship between "VT001_Orders.OrderID" and "Customer ID". The second step is to analyze business relationships – based on the business semantic mapping of the metadata-related ledger, identify fields without technical foreign keys but with business relationships (e.g., "Customer ID" in "VT001_Orders" and "Customer Number" in "VT003_Customer" have no technical foreign key relationship, but their business semantics are both "unique customer identifiers", thus establishing "VT001_Orders.CustomerID" as the foreign key relationship between "Customer ID" and "Customer ID"). The semantic association logic of "VT003_Customer Table.Customer Number" is defined. Then, association rules are defined for each association logic—clarifying the association type (technical foreign key association / business semantic association), the association field pair (source field - target field), the association matching condition (e.g., "exact match" or "fuzzy match (customer number = first 8 digits of customer ID)"), and the association direction (one-way association / two-way association). Finally, the association rules are organized according to "association rule ID - virtual table combination (e.g., VT001 + VT002) - association type - association field pair - association matching condition - association direction" to generate a virtual table field association rule table. This technology, by distinguishing between technical and business association logics, solves the problem of "business-related fields cannot be associated" caused by traditional methods that only analyze technical foreign key associations, providing a more comprehensive basis for subsequent data linkage between virtual tables.

[0068] Preferably, the specific implementation process of step 223 is as follows: First, using the virtual table field attribute details of step 221 and the virtual table field association rule table of step 222 as dual inputs, a basic structural framework is first constructed for a single virtual table—using the virtual table ID as an identifier, the fields in the virtual table field attribute details are arranged in "display order" to form a virtual table field list. At the same time, basic attributes and constraints are marked for each field (such as "Order ID: decimal(10,0), primary key, not null"), and basic information of the virtual table (such as virtual table name, business domain, creation time, and update time) is added; then, the association logic between virtual tables is integrated—based on the virtual table field association rule table, different virtual tables are connected through association field pairs to form an association topology structure between virtual tables (such as "VT001_order table ← (order ID association) → VT002_order item table ← (customer ID association) → ... The system first creates a virtual table named VT003_Customer Table and marks the association rule ID and matching conditions in the topology structure. Then, it performs index and query optimization design. Based on the "average access frequency" attribute of the virtual table field attribute details in step 221 (such as "Order ID" accessed 1000 times per day and "Customer Name" accessed 300 times per day), it automatically adds virtual indexes to high-frequency access fields (such as adding a B-tree index to "Order ID" and a hash index to "Customer Name"). At the same time, it presets the field association order of commonly used query templates (such as "query order items and customer information by order ID"). Finally, it integrates the basic framework of a single virtual table, the association topology structure between virtual tables, virtual indexes and query templates to generate a virtual table structure framework containing "virtual table basic information - field list and attributes - inter-table association topology - virtual indexes - query templates". This technology, by combining the design of virtual indexes and query templates with field access frequency, is different from the traditional method of only building the basic structure of virtual tables. It provides support for subsequent data access optimization in advance and reduces the performance consumption during subsequent queries.

[0069] Preferably, in scenarios where the virtual table structure definition file needs to be compatible with multiple system interfaces (such as BI tools and data integration platforms), step 224 is specifically implemented as follows: First, load the virtual table interface's preset structure definition specification—this specification is an XML format standard, clearly defining the root node of the file ( <virtualtabledefinition>), child node hierarchy ( <virtualtableinfo>Virtual table basic information, <fields>Field list <relations>Inter-table relationships, <indexes>Virtual index, <querytemplates>(Query template) and the required attributes for each node (such as...) <virtualtableinfo>It needs to include id / name / businessDomain attributes, and define the format requirements for the attribute values ​​(e.g., date format is "yyyy-MM-dd HH:mm:ss", field length is a positive integer); then, the virtual table structure framework generated in step 223 is formatted according to the specification—mapping the "virtual table basic information" to... <virtualtableinfo>The node's attribute values ​​(such as id=VT001, name=virtual order table, businessDomain=order domain) map the "field list and attributes" to <fields>under the node <field>Child nodes (each) <field>(Including sub-attributes such as id / name / dataType / length), mapping "inter-table relationship topology" to <relations>under the node <relation>Child nodes (containing attributes such as ruleId, sourceField, and targetField); subsequently, format validation is performed—the XML Schema validator is called to verify whether the converted XML file conforms to the structural definition specifications: firstly, node-level validation (such as... <field>The node must be located in <fields>(Under the node), and secondly, the validation of required attributes (such as...) <virtualtableinfo>The first step is to check whether the name attribute is missing. The second step is to validate the attribute format (e.g., whether the field length is a positive integer, whether the date format is correct). For validation exceptions (e.g., missing dataType attribute, incorrect date format), the exception node is automatically located and filled or corrected according to the specification (e.g., adding the default value "varchar(50)" for fields with missing dataType, and correcting "2024.08.27" to "2024-08-27"). Finally, the XML file that passes the validation is standardized—unifying the field naming format (e.g., unifying "order_amt" and "OrderAmt" to "order_amount"), standardizing the expression of constraint conditions (e.g., unifying "order amount>0" to "constraint: order_amount>0"), and adding the file version number and generation timestamp (e.g., version number=V1.0, generation time=2024-08-27). At 14:30:00, a compliant virtual table structure definition file is generated. This technology, through automated format conversion and verification, differs from the traditional manual writing and checking method, significantly improving the standardization and accuracy of the virtual table structure definition file and reducing system integration problems caused by format incompatibility.

[0070] Optionally, step 23 specifically includes the following steps:

[0071] Step 231: Extract field access attributes from the virtual table structure definition file;

[0072] Step 232: Perform multi-protocol adaptation layer construction based on field access attributes to generate "interface path / parameter name" mapping rules;

[0073] Step 233: Based on the "interface path / parameter name" mapping rules, perform interface adaptation and function encapsulation processing on the virtual table structure definition file.

[0074] Preferably, in scenarios where multiple enterprise systems (such as BI reporting tools, e-commerce order management systems, and logistics data integration platforms) access virtual tables, step 231 is specifically implemented as follows: First, the virtual table structure definition file generated in step 224 is used as the processing object. This file contains core fields such as "virtual table ID - virtual table field identifier - field basic attributes - access frequency - sensitivity level". Next, field access attribute extraction processing is performed, and the extraction dimensions are divided into three categories: First, access operation type - based on the operation metadata in the metadata set of step 133 (field read / write ratio, such as "order ID" read:write = 9:1, "order status" read:write = 4:6), the field is determined to be "read-only", "write-only", or "read / write" (read / write ratio > 8:2 is determined to be read-only, < 2:8 is determined to be write-only, and the rest are read / write); Second, access frequency level - based on the "average number of accesses per day" of the operation metadata (such as "order amount" 1200 times per day, "remarks information" 80 times per day). The data is categorized into "high frequency (>1000 times / day)," "medium frequency (100-1000 times / day)," and "low frequency (<100 times / day)." Thirdly, access permission constraints are implemented—based on the "sensitivity level" of management metadata (e.g., "customer mobile number" is highly sensitive, "order number" is moderately sensitive), determining the permission verification requirements for each field (highly sensitive fields require "user role + token dual verification," moderately sensitive fields require "user role verification," and non-sensitive fields have "no special verification"). Finally, the three types of extraction results are integrated in the format of "unique identifier of virtual table field (virtual table ID + field identifier) ​​- access operation type - access frequency level - access permission constraint" to generate a field access attribute table. This technology extracts multi-dimensional access attributes by combining operation metadata and management metadata, unlike the traditional method of extracting only the "read / write" single attribute. This provides a basis for "on-demand adaptation" for subsequent multi-protocol adaptation, avoiding resource waste caused by over-adaptation of low-frequency fields.

[0075] Preferably, in the specific technical implementation of step 232: First, based on the field access attribute table generated in step 231, the enterprise's preset "access attribute-protocol type" mapping rule base is loaded (this rule base defines "read-only + high frequency" fields to adapt to SQL query protocols, and "read-write + medium frequency" fields to adapt to RESTful protocols). The API protocol and "read-write + low-frequency" fields are adapted to the gRPC protocol, while also supporting expansion according to business needs, such as adding an MQTT protocol mapping for "IoT device data". Next, a multi-protocol adaptation layer is built, divided into SQL, RESTful, and gRPC adaptation sub-layers based on protocol type. For the SQL adaptation sub-layer, for "read-only + high-frequency" fields (such as "order amount"), a mapping rule of "interface path = / sql / virtual table ID / field identifier" and "parameter name = field identifier (lowercase)" is generated (e.g., for the "order amount" field of virtual table VT001, interface path = / sql / vt001 / order_amount, parameter name = order_amount). For the RESTful adaptation sub-layer, for "read-write + medium-frequency" fields (such as "order status"), mappings are differentiated according to HTTP method (GET method path = / api / vt001 / order_status?order_id=xxx, PUT method path...). = / api / vt001 / order_status, parameter name=order_status); For the GRPC adaptation sub-layer, for "read / write + low-frequency" fields (such as "remark information"), generate mapping rules of "service name=virtual table ID+FieldService", "method name=Get / Set+field identifier", "parameter type=field data type" (e.g., service name=VT001FieldService, method name=GetRemarkInfo, parameter type=string); Finally, organize the mapping rules of each sub-layer according to "protocol type-virtual table field unique identifier-interface path-parameter name-HTTP method / GRPC method" to generate an "interface path / parameter name" mapping rule table; The special design of this technology is that it does not adopt the traditional fixed adaptation mode of "one table, one protocol", but rather allocates protocols based on the differentiated field access attributes. This allows high-frequency read-only fields to obtain high query efficiency through the SQL protocol, medium-frequency read / write fields to maintain flexibility through the RESTful API, and low-frequency fields to reduce connection overhead through GRPC, which greatly improves the rationality of protocol adaptation in multiple scenarios.

[0076] Preferably, the specific implementation process of step 233 is as follows: First, using the "interface path / parameter name" mapping rule table generated in step 232 and the virtual table structure definition file in step 224 as dual inputs, interface adaptation processing is performed—binding the "interface path / parameter name" in the mapping rule table with the "virtual table field" in the virtual table structure definition file, generating a unique interface description for each virtual table field (e.g., binding "VT001_order amount" to the SQL interface description "path= / sql / vt001 / order_amount, supports SELECT operation, parameter order_amount must match decimal(10,2) type"); then, functional encapsulation processing is performed, integrating three types of core functional modules: one is the protocol conversion module—based on the "interface path / parameter name" mapping rule, automatically converting access requests from different systems into protocols recognizable by the virtual table (e.g., converting SQL query requests from BI tools into data reading instructions within the virtual table, RESTful...). The PUT request is converted into a virtual table field update instruction; secondly, the permission verification module—associating with the "access permission constraints" in the field access attribute table of step 231, automatically triggering "role + token" verification for requests to highly sensitive fields (e.g., a customer mobile number query request needs to verify that the user role is "customer service administrator" and the token has not expired), and triggering role verification for moderately sensitive fields to ensure access compliance; thirdly, the access caching module—for "high-frequency access" fields (e.g., order amount), configuring a caching strategy based on access frequency level (cache validity period = 5 minutes, cache is evicted according to LRU rule when full) to reduce the overhead of duplicate data reading; subsequently, the interface adaptation results are integrated with the three types of functional modules to generate a package containing A structured encapsulation package consisting of "virtual table basic information - field interface list - protocol conversion logic - permission verification rules - caching strategy" is then executed. Finally, an availability verification is performed on the encapsulation package (simulating scenarios such as SQL queries from BI tools and RESTful updates from order systems to verify whether the interface response is normal, whether the permission verification is effective, and whether the cache is hit). After the verification passes, a logical data carrier that can be directly called by multiple systems is generated. This technology differs from the traditional approach of only encapsulating basic interfaces. By integrating protocol conversion, permission verification, and caching strategies, the logical data carrier can meet the differentiated access needs of multiple systems without relying on external components, solving the architectural complexity problem caused by the need to deploy additional adaptation tools in traditional virtual tables.

[0077] Optionally, step 3 specifically includes the following steps:

[0078] Step 31: Determine the operational attributes of the metadata dataset by extracting operational attribute features from the metadata dataset;

[0079] Step 32: Based on the operation attributes, perform multi-target path mapping on the logical data carrier to establish the correspondence between "operation attributes - logical carrier nodes - data interaction protocols";

[0080] Step 33: Based on the correspondence between "operation attribute - logical carrier node - data interaction protocol", generate the data weaving path of the logical data carrier.

[0081] Optionally, step 31, determining the operational attributes of the metadata dataset by extracting operational attribute features from the metadata dataset, specifically includes the following steps:

[0082] Step 311: Perform structured parsing of the operational metadata and business metadata in the metadata set to determine the basic information related to the operational attributes of the metadata dataset;

[0083] Step 312: Based on the determined basic information, perform feature extraction on the metadata dataset for "average response time", "peak access frequency", "business process urgency" and "data update timeliness requirements" to obtain operational attribute features;

[0084] Step 313: Cluster and label the operational attribute features to determine the operational attributes of the metadata dataset.

[0085] Preferably, in the metadata operation attribute determination scenario of enterprise multi-business scenarios (such as real-time transaction settlement, daily batch data statistics, and monthly static report query), step 311 is specifically implemented as follows: First, using the metadata dataset generated in step 133 as the processing object, the operation metadata and business metadata are filtered out. The operation metadata includes "historical data read and write records (each record includes access timestamp, read / write type, and response duration)", "data processing task execution log (including task ID, processed data volume, and execution time)", and "data access frequency statistics (access count by hour / day)". The business metadata includes "the business process to which the data belongs (such as 'order payment process' and 'inventory month-end closing process')", "business SLA requirements (such as 'real-time transaction response ≤ 500ms' and 'batch statistics completion time limit ≤ 2 hours')", and "business priority tags (such as 'core transaction' and 'non-core report')". Next, the operation metadata is subjected to structured parsing: grouped by "unique data identifier (data source ID + field identifier)" and the "last 7" data of each group are extracted. The system retrieves data including "average daily read / write counts," "maximum / minimum single response time," and "high-frequency access time periods (e.g., 9:00-11:00)." It then performs structured parsing on the business metadata, associating it with the same "unique data identifier" to extract "related business process ID and name," "SLA response time limit," and "business priority level (core / important / general)." Finally, it integrates the parsing results of the operation metadata and business metadata according to "unique data identifier - operation dimension information (average read / write counts, response time range, high-frequency time periods) - business dimension information (business process, SLA time limit, priority)" to generate a metadata dataset operation attribute association basic information table. This technology differs from traditional methods that only parse operation metadata by associating it with business metadata to supplement "business requirement dimension" information. Since operation attributes (e.g., "real-time requirements") depend not only on the frequency of technical operations but also directly on the urgency of the business process—for example, the SLA time limit for the "order payment process" is much shorter than that for the "inventory month-end closing process"—early association ensures that subsequent feature extraction is more aligned with actual business needs.

[0086] Preferably, in the specific technical implementation of step 312: First, using the basic information table of operation attribute association generated in step 311 as input, it is processed one by one according to the "unique data identifier" to perform four feature extractions: First, average response time extraction - based on the historical response time data within the "response time range" in the basic information table (e.g., 1000 response records in the last 7 days), the arithmetic mean is calculated (formula: total response time / number of records). If there are outliers (e.g., extreme values ​​exceeding 3 times the standard deviation of the mean), they are removed and recalculated to obtain the average response time feature of the data (e.g., 200ms); Second, peak access frequency extraction - based on the "high-frequency access time period" and the corresponding "hourly access count", the maximum number of accesses within the time period is selected (e.g., 200 accesses in the 9:00-10:00 period, which is the highest in the whole day) as the peak access frequency feature; Third, business process urgency extraction - loading the enterprise's preset "business process-urgency" mapping rules (e.g., "order payment process = urgent", "inventory month-end closing process = normal", "static report query = low"). The first step is to extract the "urgent" business process urgency feature by matching the "business process name" in the basic information table with the "business SLA requirements" in the basic information table. The second step is to extract the "data update interval requirements" based on the "business SLA requirements" in the basic information table. For example, "SLA requirement 'data delay ≤ 1 minute'" corresponds to "high update timeliness", and "SLA requirement 'update once a day'" corresponds to "low update timeliness". The third step is to extract the "business update timeliness requirements" based on the "business SLA requirements" in the basic information table. The fourth step is to extract the "data update interval requirements" based on the "business SLA requirements" in the basic information table. The fifth step is to extract the "business update timeliness requirements" based on the "business SLA requirements" in the basic information table. The sixth step is to extract the "business update timeliness requirements" based on the "business SLA requirements" in the basic information table. The seventh step is to extract the "business update timeliness requirements" based on the "business SLA requirements" in the basic information table. The eighth step is to extract the "business update timeliness requirements" based on the "business SLA requirements" in the basic information table. The ninth ...

[0087] Preferably, the specific implementation process of step 313 is as follows: First, taking the operation attribute feature set generated in step 312 as the processing object, the features are first standardized—numerical features such as "average response time" (unit: ms) and "peak access frequency" (unit: times / hour) are standardized to the [0,1] range using Min-Max (e.g., average response time of 200ms is standardized to 0.2, and 500ms is standardized to 0.5); categorized features such as "business process urgency" (urgent / general / low urgency) and "data update timeliness requirements" (high / medium / low) are standardized. One-hot encoding is used to convert the data into numerical vectors (e.g., "urgent" is encoded as [1,0,0], "high update timeliness" is encoded as [1,0,0]), resulting in a standardized operational attribute feature matrix. The row dimension of this matrix represents the "unique data identifier" (each row corresponds to one data object), and the column dimension represents the "standardized feature items" (e.g., average response time, peak access frequency, urgency encoding bit 1, urgency encoding bit 2, etc.). The intersection of rows and columns represents the specific standardized values. Next, a pre-set K-means clustering model is loaded (the number of clusters K=3, determined through verification using historical business scenarios). (Different correspond to "real-time", "batch", "static"), input the standardized feature matrix into the model, and the model clusters according to feature similarity (e.g., features such as "average response time ≤ 0.3, peak access frequency ≥ 0.8, urgency code [1,0,0], update timeliness code [1,0,0]" are grouped into one category); after clustering, labeling is performed: establish a "clustering result - operation attribute label" mapping based on enterprise business rules (the first category "short response, high frequency, urgent, high update" corresponds to the "real-time read / write" label, the second category "long response, medium frequency, general, medium update" corresponds to the "batch processing" label). The first category is labeled "Rational Type," and the second category, "No Real-Time Response Requirement, Low Frequency, Low Urgency, Low Update," corresponds to the "Static Query Type" label. Finally, the "Unique Data Identifier - Cluster Category - Operation Attribute Label" are integrated to generate a metadata dataset operation attribute judgment result table. This technology differs from the traditional method of manually labeling operation attributes. By automatically clustering through multi-dimensional standardized features, it not only reduces manual costs but also dynamically adjusts labels according to feature changes (e.g., when the peak access frequency of a certain data increases from 0.5 to 0.9, the clustering result automatically changes from "Batch Type" to "Real-Time Type"), adapting to the dynamic changes in business scenarios.

[0088] Optionally, step 32 specifically includes the following steps:

[0089] Step 321: Perform node classification processing on the logical data carrier, synchronously collect the data interaction protocols supported by each type of node, and generate a logical carrier node-protocol adaptation representation.

[0090] Step 322: Associate and match the operation attributes with the logical carrier node-protocol adaptation representation to establish the correspondence between "operation attributes - logical carrier node - data interaction protocol".

[0091] Preferably, the specific implementation process of step 321 is as follows: First, taking the logical data carrier generated in step 2 (including virtual table structure definition, interface adaptation rules, and field access attributes) as the processing object, perform node decomposition and functional analysis processing on it—extract the core logical nodes (such as data receiving nodes, calculation processing nodes, cache nodes, transmission nodes, and result output nodes) used for data processing, transmission, and storage in the logical data carrier, and obtain the core functional parameters of each node (processing latency limit, data throughput limit, cache capacity, and supported concurrent connection count); then, based on the metadata operation attribute type (real-time read / write type, batch processing type, and static query type) determined in step 313, perform node classification processing: classify nodes with "processing latency limit ≤ 100ms, concurrent connection count ≥ 5" as "processing latency limit ≤ 100ms, concurrent connection count ≥ 5". Nodes marked "00" are categorized as "low-latency processing nodes" (suitable for real-time read / write operations); nodes with a data throughput limit ≥ 100MB / s and support for batch data block processing are categorized as "high-throughput storage nodes" (suitable for batch processing operations); and nodes with a cache capacity ≥ 10GB and a cache hit rate ≥ 90% are categorized as "high cache hit rate query nodes" (suitable for static query operations). Subsequently, data interaction protocol acquisition and processing are performed on each type of node—by calling the node's built-in protocol detection interface, all currently supported data interaction protocols and their configuration parameters are obtained (e.g., low-latency processing nodes support MQTT protocol (QoS level = 1) and GRPC protocol (compression method = gzip), while high-throughput storage nodes support Apache). Arrow protocol (batch size = 1024 records), high cache hit rate query node supports Redis protocol (data structure = Hash)); finally, the node classification results and the collected protocol information are integrated according to the structure of "node type (low latency / high throughput / high cache) - node unique identifier - core function parameters - list of supported protocols and configuration" to generate a logical carrier node-protocol adaptation characterization table; this technology is different from the traditional method of classification by "hardware type", it defines the node classification standard in reverse based on the operational attribute requirements, and simultaneously collects protocol configuration parameters to avoid adaptation failure due to unclear protocol parameters during subsequent matching, and improves the rationality of matching between nodes and operational attributes.

[0092] Preferably, in the scenario of matching multiple enterprise operation attributes (real-time read / write, batch processing, static query) with logical carrier nodes, step 322 is specifically implemented as follows: First, load the metadata dataset operation attribute determination result table generated in step 313 (including "data unique identifier - operation attribute type - operation attribute characteristics (such as real-time requirements, data processing volume)"), and simultaneously load the logical carrier node - protocol adaptation representation table generated in step 321; then, perform feature extraction processing on each operation attribute type—extract the "maximum allowed latency (≤200ms), concurrent access requirements (≥300 times / second)" for real-time read / write operation attributes, and the "single data processing volume (≥100,000 records)" for batch processing operation attributes. The system calculates the following parameters: throughput requirement (≥80MB / s), static query type query response latency tolerance (≤1s), and cache validity requirement (≥85% hit rate). This generates an operational attribute feature vector. Subsequently, an association matching process is performed: the operational attribute feature vector is matched against the node's core functional parameters and supported protocol list in the logical carrier node-protocol adaptation representation table. Real-time read / write feature vectors are matched against the "processing latency limit and concurrent connection count" of "low-latency processing nodes," while simultaneously filtering for the "MQTT / GRPC protocol" supported by the node. Batch processing feature vectors are matched against the "throughput limit and batch processing capability" of "high-throughput storage nodes," filtering for "Apache..." The "Arrow protocol" is used to match the "cache capacity and hit rate" of static query type and "high cache hit rate query node" to filter "Redis protocol". After matching, a second protocol compatibility check is performed (such as confirming that the QoS level of the MQTT protocol associated with the real-time read and write operation attribute is consistent with the protocol configuration of the low latency node), and incompatible matching results are eliminated. Finally, the verified "operation attribute type - logical carrier node type - data interaction protocol" are organized according to "operation attribute identifier - node type - node unique identifier - protocol name and configuration parameters" to generate a "operation attribute - logical carrier node - data interaction protocol" correspondence table. The special design of this technology is that it adopts a three-layer matching logic of "operation attribute feature - node function parameter - protocol configuration", which is different from the traditional two-layer hard binding of "operation attribute - node". When a new operation attribute type (such as near real-time analysis type) or logical node type is added, only the corresponding feature vector or node parameter needs to be added to complete the matching. There is no need to modify the underlying matching logic, which has high scalability.

[0093] Optionally, step 33 specifically includes the following steps:

[0094] Step 331: Perform node flow logic reasoning on the correspondence between "operation attribute - logical carrier node - data interaction protocol" to determine the data flow direction between logical carrier nodes and generate node flow logic accordingly;

[0095] Step 332: Generate the data weaving path of the logical data carrier based on the node flow logic.

[0096] Preferably, in the specific technical implementation of step 331: First, taking the "operation attribute - logical carrier node - data interaction protocol" correspondence generated in step 322 as the processing object, the enterprise's preset "business process - node dependency" mapping rule base is loaded synchronously. This rule base defines the pre-dependency logic between nodes based on multiple business scenarios (such as "real-time order transaction process" requiring data validity verification before storage operation, and "customer data batch synchronization process" requiring data format unification before summary operation). It also includes "node protocol compatibility judgment rules" (such as when the output protocol of the preceding node is MQTT, the subsequent node must support the MQTT protocol or the associated protocol conversion child node to avoid data transmission interruption due to protocol incompatibility). Next, the correspondence is grouped according to the operation attribute type (real-time read / write, batch processing, static query). Node flow logic reasoning is performed for each group of operation attributes: taking the real-time read / write operation attribute as an example, the associated logical carrier node is selected from the correspondence ( The system consists of a data receiving node, a data verification node, and a low-latency storage node, along with their corresponding data interaction protocols (all MQTT). Based on the node dependency rules of the "real-time order transaction process," the node flow direction is determined to be "data receiving node → data verification node → low-latency storage node." Simultaneously, protocol compatibility rules confirm that all three nodes support the MQTT protocol, eliminating the need for additional protocol conversion modules. Subsequently, logical conflict checks are performed on each set of inference results (e.g., if an abnormal dependency of "storage node preceding verification node" is detected, it is automatically adjusted to "verification node preceding" according to the rule base). Data flow triggering conditions between nodes are also added (e.g., "when the data receiving node receives a cumulative total of 10 data entries, a transmission operation to the data verification node is triggered"). Finally, the verified "operation attribute type - node sequence - protocol connection relationship - triggering condition" are integrated to generate a node flow logic table. Each row in this table corresponds to the complete flow logic for one operation attribute, for example, "Real-time read / write type: data receiving node (MQTT) →..." Data Verification Node (MQTT) → Low Latency Storage Node (MQTT), Triggering Condition: Trigger a transmission every 10 data entries; This technology differs from the traditional fixed node order reasoning method. By dynamically combining business process dependencies and protocol compatibility judgments, the node flow logic not only meets the actual business needs, but also ensures the smoothness of data interaction, avoiding path execution failures caused by ignoring business dependencies or protocol compatibility in traditional methods.

[0097] Preferably, in the scenario of generating data weaving paths across multiple business domains (order domain, customer domain, and logistics domain), step 332 is specifically implemented as follows: First, using the node flow logic table generated in step 331 as the processing object, a structured path transformation is performed for each node flow logic—a unique identifier is assigned to each logical carrier node in the logic (e.g., "DRN_001" represents a real-time data receiving node, "DVN_001" represents a real-time data verification node, and "LLSN_001" represents a low-latency storage node). Simultaneously, the "data interaction protocol configuration parameters" of each node are extracted from the correspondence in step 322 (e.g., QoS level of MQTT protocol = 1, transmission port = 1883, message timeout = 5s, compression method of GRPC protocol = gzip, connection timeout = 3s). Next, the connection relationship between nodes is defined: using node identifiers... For indexing, the mapping relationship between preceding and subsequent nodes is clearly defined (e.g., "DRN_001→DVN_001" "DVN_001→LLSN_001"), and data transmission rules are added for each connection relationship (e.g., "DRN_001 to DVN_001 uses batch packaging mode when transmitting, and the data size of a single packet is ≤1KB" "DVN_001 to LLSN_001 uses real-time push mode when transmitting, and the data is transmitted immediately after the data verification is passed"). Subsequently, path fault tolerance configuration is integrated: based on the operation metadata (historical node fault records, fault recovery time) in the metadata set, fault tolerance mechanisms are added to key nodes (e.g., fault retry count = 3 times, retry interval = 2s, and automatic switch to the backup node when the primary node fails, such as switching to "DVN_002" when "DVN_001" fails).Finally, the node identifier sequence, protocol configuration parameters, connection relationships, and fault tolerance configuration are integrated according to the path definition format (such as JSON format) preset by the virtual table interface to generate a data weaving path file for the logical data carrier. An example of the core content of the "real-time read / write" path in this file is: {"Operation Attribute":"Real-time Read / Write","Path ID":"RWP_001","Node Sequence":["DRN_001","DVN_001","LLSN_001"],"Protocol Configuration":{"DRN_001":{"protocol":"MQTT","port":1883,"QoS":1,"timeout":5},"DVN_001":{"protocol":"MQTT","port":1884,"QoS":1,"timeout":5} The unique design of this technology lies in the fact that the generated path file not only contains the node order but also integrates the specific protocol configuration and fault tolerance mechanism. This differs from traditional path generation methods that only record the node sequence. This eliminates the need for additional querying of protocol parameters or fault tolerance rules during subsequent path execution, significantly improving the execution efficiency and stability of data weaving. It also facilitates subsequent resource scheduling strategies (step 4) for path optimization and adjustment.

[0098] Optionally, step 4 specifically includes the following steps:

[0099] Step 41: Collect data access task queue information in real time by deploying a distributed task monitoring agent, and collect access indicator data through the indicator collection module to monitor data access task queue and access indicator feedback.

[0100] Step 42: Based on the constructed resource scheduling model including LSTM time series prediction network and random forest classification network, perform preliminary feature extraction processing on the data access task queue and access index feedback to obtain task resource demand features and index load trend features. Perform correlation and fusion processing on task resource demand features and index load trend features to extract resource load correlation features. Then match the resource load correlation features with the preset resource scheduling rule base to generate resource scheduling strategy.

[0101] Preferably, the specific implementation process of step 41 is as follows: First, for all nodes in the data access task chain (including data access task scheduling nodes, business domain computing nodes such as order / inventory / customer, and data source access nodes such as MySQL / MongoDB / IoT devices), a lightweight distributed task monitoring agent is deployed (each node independently deploys one agent instance to avoid single-point failures affecting overall monitoring). This agent establishes a long TCP connection with the node's local task scheduler to collect data access task queue information in real time. The specific fields collected include the unique task identifier (task ID, in the format "Task_Node IP_Timestamp"), the task business type (real-time transaction query / batch data synchronization / static report generation), and the target data identifier (data source ID). The data collection mechanism, which combines virtual table fields such as "DS001_VT001_OrderID", task priority levels (level 1 being the highest and level 5 the lowest, preset by business metadata), task submission timestamps (accurate to milliseconds), and current task execution status (waiting for scheduling / executing / execution failed / execution completed), is set to "queue change trigger" (i.e., data is collected immediately when a task queue is added, its status changes, or it is deleted), to avoid information lag caused by periodic collection and generate node-level task queue details. Next, an indicator collection module is deployed synchronously on all nodes in the aforementioned end-to-end chain. This module uses native APIs of the node's operating system (such as reading CPU / memory data from the proc file system in Linux and obtaining disk I / O data from the Performance Counter interface in Windows) and built-in resource monitoring interfaces of the nodes (such as the Redis INFO command for cache nodes and the SHOW command for database nodes). The STATUS command collects access metric data—specific metrics include node CPU utilization (unit: %, the average utilization of all CPU cores), memory utilization (unit: %, the ratio of used memory to total memory), disk I / O throughput (unit: MB / s, distinguishing between read and write throughput), network transmission latency (unit: ms, the average of the time taken for 3 TCP connections between the node and the target data source), and cache hit rate (unit: %, only for cache nodes, the ratio of the number of hits to the total number of queries). The collection frequency is fixed at 1 second / time to balance real-time performance and resource consumption, generating node-level metric data units. Subsequently, the distributed task monitoring agent and the metric collection module transmit their respective generated task queue details and metric data units to the central monitoring node via the lightweight MQTT protocol (QoS level set to 1 to ensure no data loss and no duplication). The central monitoring node deploys a data cleaning module to first perform format verification on the received data (such as verifying whether the task ID conforms to the preset format and whether the metric value is within a reasonable range (CPU utilization 0-100%)), and remove data with incorrect format or abnormality.Finally, the central monitoring node uses "node IP - collection timestamp" as the association key to integrate the task queue details and indicator data units within the same 1-second time window of the same node, generating a real-time monitoring dataset containing "node basic information (IP address, node type such as scheduling node / computing node) - collection timestamp (accurate to the second) - task queue aggregation information (total number of tasks, number of tasks of each business type, number of tasks to be executed of each priority) - indicator data aggregation information (average and maximum value of each indicator within 1 second)". This dataset is directly used as input data for the resource scheduling model in step 42, providing task and indicator linkage information for extracting resource load correlation features. This technology differs from traditional monitoring schemes (traditional methods collect task information at scheduling nodes and indicator data at the server level, with no correlation between the two and incomplete monitoring node coverage). It achieves full-link node monitoring through distributed agents, while associating task business attributes (type, priority) with resource indicators, enabling the monitoring data to reflect the correspondence between "business task type - resource consumption", avoiding the problem of traditional monitoring where data fragmentation leads to subsequent resource scheduling failing to accurately match business needs.

[0102] Optionally, step 42 specifically includes the following steps:

[0103] Step 421: Perform standardized preprocessing on the data access task queue and access indicator feedback to generate task data matrix and indicator time series data matrix respectively;

[0104] Step 422: Input the task data matrix into a random forest classification network for feature extraction to obtain the task resource requirement features;

[0105] Step 423: Input the indicator time series data matrix into the LSTM time series prediction network to capture the time dependency of the indicator data, output the indicator load prediction value, and predict the load rise rate and load fluctuation amplitude based on the prediction value to obtain the indicator load trend characteristics.

[0106] Step 424: Construct a feature weight mapping table based on the operation metadata in the metadata set to perform weighted fusion of task resource demand features and indicator load trend features to obtain resource load correlation features;

[0107] Step 425: Traverse and match the resource load association features with the preset resource scheduling rule base, and generate specific resource scheduling strategies based on the matching results.

[0108] Preferably, in the data access monitoring data processing scenario of enterprise multi-business scenarios (including real-time transactions, batch statistics, and static queries), step 421 is specifically implemented as follows: First, taking the real-time monitoring dataset generated in step 41 as the processing object, the dataset is split into two types of core data—data access task queue data (including task ID, task business type, priority, target data identifier, and submission timestamp) and access indicator feedback data (including node IP, collection timestamp, CPU utilization, memory utilization, disk I / O throughput, network transmission latency, and cache hit rate); then, standardized preprocessing is performed on the data access task queue data: for each task... Classification features such as business type (real-time transaction / batch statistics / static query) and priority (levels 1-5) are converted into numerical vectors using one-hot encoding (e.g., "real-time transaction" is encoded as [1,0,0], and priority level 1 is encoded as [1,0,0,0,0]). For task submission timestamps (milliseconds), the time difference (in seconds) between the timestamp and the current time is calculated and standardized to the [0,1] interval using Min-Max. The encoded classification features and standardized time difference features are integrated by "task ID" to generate a task data matrix—the row dimension of this matrix represents the number of tasks (each row corresponds to one data access task), and the column dimension represents the number of features (e.g., business type, batch statistics / static query), etc.). The task is divided into 9 columns: 3 columns for task type, 5 columns for priority, and 1 column for time difference. The intersections of rows and columns contain standardized values ​​(e.g., "1" in row 1, column 1 indicates a real-time transaction). Subsequently, standardized preprocessing is performed on the access indicator feedback data: grouping by "node IP + 1-minute sliding window" (each window contains 60 1-second data collection points), calculating the mean of each indicator within each window (e.g., mean CPU utilization, mean memory usage), and using Z-Score standardization (subtracting the mean and dividing by the standard deviation) to eliminate the influence of units; the standardized window indicators are then integrated by "node IP - window start timestamp" to generate an indicator time-series data matrix. The matrix's row dimension represents the number of time windows (each row corresponds to a 1-minute window), and the column dimension represents the number of metrics (such as CPU / memory / disk I / O / network latency / cache hit rate, a total of 5 columns). The intersection of rows and columns represents the specific standardized metric value (e.g., "0.8" in the 2nd row and 1st column represents the standardized CPU utilization value of a certain node in a certain window). This technology differs from the traditional method of standardizing only numerical features by incorporating task business attributes (type, priority) into the preprocessing scope, enabling subsequent models to extract features based on business semantics and avoiding misjudgments of resource requirements due to ignoring business attributes (e.g., treating high-priority real-time tasks and low-priority batch tasks with the same standard).

[0109] Preferably, in the specific technical implementation of step 422: First, a preset random forest classification network model is loaded. The special structure design of this model is adapted to the resource demand classification scenario of data access tasks—it contains 15 CART decision trees (determined through cross-validation; too many trees can easily lead to overfitting, and too few trees result in low classification accuracy). The maximum depth of each tree is set to 8 (to limit complexity and avoid learning noise). The splitting feature selection uses the Gini coefficient (to calculate feature impurity for classification tasks), and the priority of the splitting features is configured according to "task business type > priority > time difference" (because the business type directly determines the resource demand type, such as real-time transactions requiring high CPU and batch statistics requiring high I / O). The model training phase has been completed based on the historical task-resource consumption dataset (containing 100,000+ records, each record containing task features and labeled "high CPU demand / high I / O demand / balanced demand"). During the training process, 5-fold cross-validation is used, and the classification accuracy is stable at over 85%. Then, the model generated in step 421 is... The task data matrix is ​​input into the random forest classification network, which classifies the data in parallel using multiple decision trees. Each tree traverses from the root node based on task features (e.g., if "business type = real-time transaction", it enters the left subtree; if "priority = level 1", it further filters the data), and outputs the tree's prediction of the task resource requirement type. Subsequently, the network performs a voting mechanism (majority rule) on the predictions of the 15 trees to determine the final task resource requirement type and calculates the confidence score of that type (number of votes / total number of trees). Finally, the "task ID - resource requirement type (high CPU / high I / O / balanced) - confidence score" is integrated to generate a task resource requirement feature table. This technique differs from the traditional random forest's "indiscriminate splitting feature" design. By configuring the splitting feature priority according to business importance, the model prioritizes judging resource requirements based on business type. For example, for "real-time transaction" type tasks, even if there are differences in time difference features, it is still prioritized as "high CPU requirement", which is more in line with the resource consumption patterns of actual business operations.

[0110] Preferably, the specific implementation process of step 423 is as follows: First, load the preset LSTM temporal prediction network model. The structure of this model is optimized for the temporal features of access indicators—it includes one input layer (number of neurons = number of indicators, i.e., 5, corresponding to CPU / memory / disk I / O / network latency / cache hit rate), two hidden layers (each layer has 64 LSTM units, using the ReLU activation function to alleviate the gradient vanishing problem; each layer adds a Dropout layer, dropout...). The model is set to a rate of 0.2 to prevent overfitting, and a fully connected output layer (number of neurons equal to the number of indicators, i.e., 5, using a Linear activation function to output the predicted indicator values). During training, the model was trained using one month's worth of historical indicator time-series data (1440 one-minute windows per node per day). The loss function used was mean squared error (MSE), and the optimizer was Adam (learning rate = 0.001). After training, the indicator prediction error (MAE) was controlled within 5%. Next, the indicator time-series data matrix generated in step 421 was input into the LSTM network in chronological order. The input layer converted the standardized indicator values ​​into feature vectors and then passed them to the first hidden layer. The LSTM unit captured the time dependencies of the indicators (such as the upward trend of CPU utilization with increasing real-time task volume and the fluctuation of memory usage with batch data processing) through a forget gate (controlling the discarding of historical information), an input gate (updating cell state), and an output gate (generating the current output). The second layer... The hidden layer further processes the output of the first layer to enhance the extraction of long-term time-series features. The output layer outputs the predicted values ​​of the indicator load for the next 1-minute window (such as predicted CPU utilization and predicted memory utilization). Subsequently, based on the predicted values ​​and the actual indicator values ​​of the last 3 windows, the indicator load increase rate (such as CPU utilization increase rate = (predicted value - actual value 3 minutes ago) / 3 minutes) and indicator load fluctuation amplitude (such as memory utilization fluctuation amplitude = predicted value - absolute value of the actual value of the most recent window) are calculated. Finally, the "node IP - prediction time window - indicator load predicted value - increase rate - fluctuation amplitude" are integrated to generate an indicator load trend feature table. This technology adds a business-oriented time-series window design (1-minute window matches the decision cycle of enterprise resource scheduling) to the traditional LSTM, and strengthens the capture of time-series dependencies through two hidden layers. Compared with a single hidden layer LSTM, the trend prediction accuracy for sudden loads (such as a sudden increase in real-time task volume) is improved by about 15%.

[0111] Preferably, in a scenario involving the fusion of resource load correlation features across multiple nodes and tasks, step 424 is specifically implemented as follows: First, based on the operation metadata in the metadata set of step 133, a feature weight mapping table is constructed—the row dimension of this table represents the task resource demand type (high CPU / high I / O / balanced), the column dimension represents the load trend features (CPU growth rate / memory fluctuation / disk I / O throughput / network latency / cache hit rate), and the intersection of the row and column represents the weight value (summing up to 1); the weight determination logic is based on the "task type" in the historical operation metadata. The "Type-Indicator Sensitivity" statistic (e.g., CPU metric sensitivity is 0.6 for high CPU demand tasks, and disk I / O metric sensitivity is 0.5 for high I / O demand tasks) is calculated as follows: "High CPU Demand" has a weight of [0.6, 0.1, 0.1, 0.1, 0.1, 0.1] (CPU rate of increase has the highest weight), "High I / O Demand" has a weight of [0.1, 0.1, 0.5, 0.1, 0.1] (disk I / O throughput has the highest weight), and "Balanced Demand" has a weight of [0.2, 0.2, 0.2, 0.2, 0.2, 0.2]. Next, the task resource requirement characteristics from step 422 are associated with the indicator load trend characteristics from step 423 according to "node IP + task association" (e.g., a real-time transaction task (high CPU requirement) of a node is associated with the CPU rise rate characteristic of that node). Subsequently, a weighted fusion calculation is performed: for each association combination, the weight vector corresponding to the task resource requirement type is multiplied element-wise with the indicator load trend characteristic vector (standardized rise rate, fluctuation range, etc.), and the sum is obtained to obtain the resource load association characteristic value (e.g., the association characteristic value of a high CPU requirement task = 0.6 × CPU rise rate + 0.1 × memory fluctuation range + ... + 0.1 × cache hit rate). Finally, the "node IP - associated task type - resource load association characteristic value" are integrated to generate a resource load association characteristic table. This technology differs from the traditional fixed weight fusion method. It dynamically adjusts the weights based on operation metadata, so that the feature fusion result can reflect the association relationship between "task requirement - indicator status". For example, when the proportion of high I / O tasks is high, the weight of disk I / O indicators is automatically increased, avoiding the disconnect between the association characteristics and the actual resource bottleneck caused by the solidification of weights.

[0112] Preferably, in the specific technical implementation of step 425: First, a preset resource scheduling rule base is loaded. This rule base is classified into "resource constraints - task priority constraints - security constraints". Each rule includes triggering conditions (based on resource load correlation feature values), execution actions, and priorities (in case of conflicts between rules, priority is used for execution); for example, the resource constraint rule: "a node's CPU load correlation feature value > 0.8 (corresponding to actual CPU utilization > 85%) → trigger CPU resource allocation + suspend low-priority (level 4-5) batch tasks on this node"; the task priority constraint rule: "real-time transaction task proportion > 30% and node network latency correlation feature value > 0.7 → prioritize allocating low-latency transmission links for real-time tasks"; the security constraint rule: "the cache hit rate associated with highly sensitive data tasks (management metadata tags) < 0.6 → "Prohibit cache node resource reduction"; then, using the resource load association feature table generated in step 424 as the matching basis, perform rule base traversal matching for each feature: extract "node IP-associated task type-feature value" from the feature, and compare it with the trigger conditions of each rule in the rule base (e.g., a node's CPU feature value 0.85 > 0.8, matching the CPU resource increase rule); for multiple matched rules, sort them according to rule priority (security constraint > task priority constraint > resource constraint), and filter out the rule combination with the highest priority; subsequently, generate a specific resource scheduling strategy based on the filtered rules: specify the scheduling object (node ​​IP), scheduling action (e.g., adding 2 CPU cores, ... The process involves pausing five level 4 priority batch tasks, switching transmission links, and determining the execution time (immediate execution / delayed execution for 30 seconds, with emergency rules executed immediately). Finally, the scheduling strategy is organized according to "strategy ID-node IP-scheduling action-execution time-trigger rule" to generate an executable resource scheduling strategy file. In this application, the rule base design incorporates security constraints on management metadata (such as protection of highly sensitive data tasks), and the rule priority matches actual business needs, avoiding the security risks caused by focusing only on resource indicators in the traditional approach (such as reducing the allocation of cache nodes for highly sensitive data to save resources). At the same time, rule traversal matching ensures the comprehensiveness of strategy generation and reduces the probability of missing key scheduling actions.

[0113] Optionally, step 5 specifically includes the following steps:

[0114] Step 51: Generate path optimization constraints based on resource scheduling strategies;

[0115] Step 52: Analyze and evaluate the current resource consumption of the data weaving path of the logical data carrier, and identify the bottleneck logical nodes in the data weaving path that do not match the path optimization constraints.

[0116] Step 53: Optimize the data weaving path of the logical data carrier based on the bottleneck logical node to obtain the optimized data weaving path for data weaving of multi-source heterogeneous data.

[0117] Optionally, step 51 specifically includes the following steps:

[0118] Step 511: Perform structured parsing on the core instructions in the resource scheduling strategy to generate strategy instruction parsing factors;

[0119] Step 512: Based on the operational metadata and management metadata in the metadata set, determine the constraint dimensions and corresponding constraint indicators according to the policy instruction parsing factors;

[0120] Step 513: Based on the constraint dimensions and constraint indicators, generate path optimization constraints that are categorized and integrated according to "resource constraints - business constraints - security constraints".

[0121] Preferably, the specific implementation process of step 511 is as follows: First, the specific resource scheduling strategy generated in step 425 is used as the processing object. This strategy includes "resource adjustment instructions (such as 'add 20% CPU resources to real-time transaction nodes', 'reduce 15% memory resources to batch statistics nodes')", "task priority adjustment instructions (such as 'increase the priority of order payment tasks to level 1', 'decrease the priority of monthly report tasks to level 4')", and "node load allocation instructions (such as 'limit the load of a single cache node to ≤80%', 'reduce the load of IoT stream data')". The core instructions are categorized into three types: "load offloading to backup nodes," "load offloading to standby nodes," and "load offloading to backup nodes." Next, structured parsing is performed, defining the parsing dimensions as "instruction type - resource type - adjustment range - associated node / task type - effective condition": For resource adjustment instructions, "resource type (CPU / memory / disk I / O)," "adjustment range (increase by 20% / decrease by 15%)," "associated node (real-time transaction node / batch statistics node)," and "effective condition (effective when task queue length > 50)" are extracted; for task priority adjustment instructions, "task type (order)" is extracted. For single payment / monthly report, extract "target priority (level 1 / 4)" and "effective conditions (effective immediately without special conditions)"; for node load allocation instructions, extract "resource type (load rate)", "adjustment range (≤80% / distribution)", "associated node (cached node / IoT stream data node)" and "effective conditions (effective when load rate is >90% for 1 minute)"; finally, organize the parsing results according to "parsing factor ID - instruction type - core parsing item (resource type / task type, etc.) - specific value" to generate a strategy instruction parsing factor table. For example, "parse factor ID=F001, instruction type=resource adjustment, resource type=CPU, adjustment range=increase by 20%, associated node=real-time transaction node, effective condition=task queue length>50"; this technology differs from the traditional simple parsing method that only extracts "resource-range". By supplementing "associated object (node / task)" and "effective condition", it enables subsequent constraints to accurately locate the application scenario and avoids the generalization of constraints caused by incomplete information in traditional parsing (such as applying the CPU increase instruction indiscriminately to all nodes).

[0122] Preferably, in the scenario of defining constraints for data weaving paths across multiple business domains (order domain, customer domain, logistics domain) within an enterprise, step 512 is specifically implemented as follows: First, load the operational metadata and management metadata from the metadata set in step 133—the operational metadata includes "historical data weaving path resource consumption logs (each record includes path ID, node ID, average CPU / memory usage, and data transmission latency)" and "path execution frequency statistics (execution counts by hour / day)"; the management metadata includes "field sensitivity levels (high / medium / low sensitivity)" and "data transmission security specifications (e.g., high-sensitivity fields require additional security measures)." The transmission protocol must comply with TLS 1.3. Business process SLA requirements (e.g., real-time path response latency ≤ 500ms, batch path execution time ≤ 2 hours) are also specified. Next, using the strategy instruction parsing factor table generated in step 511 as the matching basis, metadata is associated one by one according to the "parsing factor ID": For the parsing factor "CPU increase by 20% (associated with real-time transaction nodes)," the historical average CPU usage of the node (e.g., 70%) is extracted from the operation metadata. Combined with the increase, the "resource constraint dimension" is determined, with the corresponding constraint indicator being "real-time transaction node CPU utilization ≤ (70% × (1)" +20%) = 84%; For the "order payment task priority upgraded to level 1" parsing factor, the SLA response latency (≤500ms) of the order payment business is extracted from the management metadata to determine the "business constraint dimension", and the corresponding constraint indicator is "order payment task data weaving path response latency ≤500ms"; For the "highly sensitive field associated node" parsing factor (implied in the security requirements of the resource scheduling strategy), the field sensitivity level is extracted from the management metadata to determine the "security constraint dimension", and the corresponding constraint indicator is "the path where the highly sensitive field is located must integrate the national cryptographic SM4 encryption module and transmission protocol". "Requires TLS 1.3"; Finally, integrate "Resolver ID - Constraint Dimension (Resource / Business / Security) - Constraint Indicator Name - Constraint Indicator Threshold / Requirement - Associated Metadata Basis" to generate a table corresponding to constraint dimensions and indicators; The special design of this technology is that it does not rely on a single resource scheduling instruction to formulate constraints, but combines the historical consumption patterns of operational metadata with the security / business requirements of management metadata, so that the constraint indicators not only meet the resource scheduling goals, but also fit the actual operation patterns of data weaving (such as determining the threshold based on historical CPU usage to avoid resource waste due to excessively high thresholds or insufficient performance due to excessively low thresholds).

[0123] Preferably, in the specific technical implementation of step 513: First, using the constraint dimension and indicator correspondence table generated in step 512 as input, the constraints are classified and integrated according to three dimensions: "resource constraints - business constraints - security constraints". The resource constraint dimension integrates indicators related to "CPU / memory / disk I / O utilization", "node load rate", and "resource adjustment range", such as "real-time transaction node CPU utilization ≤ 84%", "cached node load rate ≤ 80%", and "batch statistics node memory utilization after reduction ≤ 65%". The business constraint dimension integrates indicators related to "path response latency", "task priority matching", and "path execution time". Indicators include, for example, "order payment task path response latency ≤ 500ms", "level 1 priority task path execution priority higher than level 4", and "monthly report path execution time ≤ 2 hours"; security constraints integrate indicators related to "field sensitivity level matching", "transmission security specifications", and "encryption module requirements", such as "highly sensitive field paths must integrate SM4 encryption modules", "all path data transmission protocols must be TLS 1.3", and "medium sensitive field paths must retain permission verification nodes"; then, standardization processing is performed on the indicators for each type of constraint: for numerical indicators (such as CPU utilization ≤ 84%, latency ≤ 500ms)... First, clearly define the "metric name - threshold - unit - effective node / task type"; for rule-based metrics (such as integrated encryption modules, protocol requirements), clearly define the "metric name - specific requirements - effective objects (highly sensitive field paths / all paths)"; then, perform constraint conflict verification: check whether different constraints of the same node / task type are contradictory (e.g., "real-time transaction node CPU utilization ≤ 84%" and "CPU configuration must meet peak load" are not contradictory, but if "CPU utilization ≤ 84%" and "CPU utilization ≥ 90%" are present, then mark them as conflicting), in case of conflict, the business / security requirements of managing metadata shall prevail. The criteria are as follows (e.g., security constraints have higher priority than resource constraints); finally, the verified constraints are organized according to "constraint category - constraint ID - indicator name - specific requirements - effective object - effective condition" to generate a path optimization constraint table that is categorized and integrated according to "resource constraints - business constraints - security constraints"; this technology differs from the traditional uncategorized constraint generation method. By categorizing and integrating, the subsequent bottleneck node identification (step 52) can be matched item by item according to the category. At the same time, the conflict verification mechanism ensures the rationality of the constraints and avoids the optimization failure caused by the contradiction of constraints in the traditional method (e.g., requiring both high and low CPU usage at the same time).

[0124] Optionally, step 52 specifically includes the following steps:

[0125] Step 521: Perform node decomposition processing on the data weaving path of the logical data carrier, extract all logical nodes in the path, and obtain the real-time resource consumption data of each node to generate node resource consumption details.

[0126] Step 522: Match and evaluate the node resource consumption details with the path optimization constraints item by item, mark the nodes that do not meet the constraints, and generate the target node sequence.

[0127] Step 523: Perform influence range analysis on the nodes in the target node sequence, and filter out the nodes whose influence weight on the overall performance of the path is greater than the set weight threshold as abnormal nodes and bottleneck logical nodes in the data weaving path that do not match the path optimization constraints.

[0128] Preferably, the specific implementation process of step 521 is as follows: First, the data weaving path of the logical data carrier generated in step 332 is taken as the processing object. This path includes multiple scenario paths such as real-time transaction path (e.g., order payment data weaving path), batch statistics path (e.g., daily inventory data weaving path), and static report path (e.g., monthly revenue data weaving path). Each path is decomposed according to the link logic of "data source access - data processing - data transmission - result output", extracting all logical nodes in the path and recording the dependencies between nodes (e.g., "data source access node A → data verification node B → real-time calculation node C → result output node D"), avoiding the inability to analyze the impact scope later due to traditional decomposition only extracting nodes without retaining dependencies. Next, the distributed resource monitoring interface (this interface is deployed on the server where each node is located and supports real-time data query) is called to collect real-time resource consumption data of each logical node at a frequency of 1 second / time. The collection dimensions include: CPU utilization rate (unit: %, taking the CPU of the server to which the node belongs). The system collects data on the following: average core usage, memory utilization (%, used memory / total allocated memory), data processing throughput (MB / s, processing nodes only), data transmission latency (ms, transmission nodes only), and encryption module running status (boolean value, security nodes only, true for normal operation). Then, the disassembled node information and collected resource consumption data are integrated according to the structure: "Unique node identifier (path ID + node ID) - Node type (access / processing / transmission / output / security) - Real-time resource consumption data (CPU / memory / throughput / latency / encryption status) - Collection timestamp," generating a detailed node resource consumption report. This technology differs from the traditional simple method of only collecting CPU / memory data by supplementing business-related resource dimensions (throughput, latency) and security dimensions (encryption status). Because the performance of the data weaving path depends not only on basic resources but also directly on business processing efficiency (throughput), transmission real-time performance (latency), and security compliance (encryption), comprehensive collection ensures no omissions in subsequent evaluations.

[0129] Preferably, in the scenario of node compliance verification across multiple enterprise constraint dimensions (resource constraints, business constraints, and security constraints), step 522 is specifically implemented as follows: First, load the path optimization constraints generated in step 513, which are categorized and integrated into "resource constraints, business constraints, and security constraints," and clarify the verification dimensions and thresholds / requirements for each type of constraint—resource constraints include "CPU utilization ≤ 84% (real-time node)", "memory utilization ≤ 75% (batch node)", and "throughput ≥ 50MB / s (processing node)"; business constraints include "real-time path transmission delay ≤ 500ms" and "batch path execution time ≤ 2 hours"; security constraints include "highly sensitive field node encryption module status = true" and "transmission node protocol = TLS1.3"; then, using the node resource consumption details generated in step 521 as the verification basis, associate the corresponding path optimization constraints one by one according to the "unique node identifier" (e.g., associating the real-time computing node C with the CPU constraints and delay constraints of the real-time path); subsequently, perform item-by-item matching evaluation: for resource constraints, compare the node's real-time CPU / memory / throughput with the constraint thresholds (e.g., for nodes with different memory / throughput), and compare the node's real-time CPU / memory / throughput with the constraint thresholds (e.g., for nodes with different memory / throughput). Point C's CPU utilization is 88% > 84%, which is deemed unsatisfactory. For business constraints, the transmission node latency is compared with the constraint threshold (e.g., transmission node E's latency is 580ms > 500ms, which is deemed unsatisfactory). For security constraints, the encryption module status and protocol type are verified (e.g., the encryption module status of the highly sensitive field node F is false, which is deemed unsatisfactory). A "Constraint Satisfaction Status" field (satisfied / unsatisfied) is added to each node, and the unsatisfied constraint items and deviation values ​​are recorded (e.g., "CPU utilization: 88% > 84%, deviation 4%" "Latency: 580ms > 500ms, deviation 80ms"). Finally, nodes with "Constraint Satisfaction Status = Unsatisfied" are filtered out and organized according to "Unique Node Identifier - Unsatisfied Constraint Category (Resource / Business / Security) - Specific Constraint Item - Deviation Value" to generate a target node sequence. The special design of this technology is that the matching evaluation is sorted according to the priority of "Security Constraint > Business Constraint > Resource Constraint". If a node does not satisfy multiple types of constraints at the same time, the higher priority constraint item is marked first, avoiding the problem of not being able to grasp the key points in subsequent analysis due to the traditional lack of priority marking.

[0130] Preferably, in the specific technical implementation of step 523: First, taking the target node sequence generated in step 522 as the processing object, the output results of the random forest classification network in step 422 and the LSTM temporal prediction network in step 423 are loaded synchronously—the random forest classification network (containing 15 CART decision trees, with splitting features configured according to "task business type > priority > resource requirement") has output the "resource requirement weight" of each node's associated task (e.g., the resource requirement weight of the node associated with the real-time transaction task is 0.7, and the weight of the node associated with the batch statistics task is 0.3), which reflects the resource importance of the node's associated task; the LSTM temporal prediction network (containing 2 hidden layers, 64 LSTM units per layer, Dropo The "load impact coefficient" for each node in the next 5 minutes has been output (e.g., the load impact coefficient of node C is 0.8, indicating that its abnormal load has a high impact on downstream nodes). This coefficient is generated based on node dependencies and historical load propagation patterns. Next, a node impact range analysis model is constructed: based on the node dependency graph recorded in step 521, the "number of directly affected nodes" (e.g., node C directly affects output node D, direct impact number = 1) and "number of indirectly affected nodes" (e.g., node C indirectly affects report generation node G, indirect impact number = 1) of each target node are calculated. Combining the resource demand weights of the random forest and the load impact coefficient of the LSTM, the total impact weight is calculated according to the formula: "Total impact weight = (number of directly affected nodes × 0.4)". The total impact weight is calculated as follows: (1 × 0.4 + 1 × 0.2) × 0.7 × 0.8 = 0.336. Then, a preset impact weight threshold (e.g., 0.3, which can be adjusted according to the business scenario) is loaded to filter out target nodes with a total impact weight > 0.3. Finally, the filtered nodes are organized according to "unique node identifier - total impact weight - associated task type - list of affected nodes" to generate a list of bottleneck logic nodes in the data weaving path that do not match the path optimization constraints. This technology differs from the traditional method of filtering bottlenecks based solely on resource consumption deviation values. By combining the task importance assessment of random forests with the load impact prediction of LSTM, the filtered bottleneck nodes not only meet resource constraints but also reflect their actual impact on the overall performance of business tasks and paths, avoiding the problem of misjudging low-impact nodes as bottlenecks in the traditional approach.

[0131] Optionally, step 53 specifically includes the following steps:

[0132] Step 531: Analyze the anomaly types of bottleneck logic nodes in order to develop targeted path optimization solutions;

[0133] Step 532: Based on the path optimization scheme, the data weaving path of the logical data carrier is decomposed into segments to locate the path segment where the bottleneck logical node is located.

[0134] Step 533: Based on the specific adjustment rules in the path optimization scheme, optimize the path segment where the bottleneck logic node is located according to the process of "node parameter adjustment - module integration - link adaptation" to obtain the optimized target path segment;

[0135] Step 534: Call the protocol adaptation module to perform protocol compatibility verification and splicing processing on the optimized target path segment and the non-optimized segment in the original path to obtain the spliced ​​path. At the same time, check whether the spliced ​​path meets the path optimization constraints. If it does, the whole path is used as the optimized data weaving path. Otherwise, repeat steps 531-534 until the optimized data weaving path is obtained.

[0136] Step 535: Call the optimized data weaving path to perform field association, format conversion and secure transmission processing on the multi-source heterogeneous data to complete the data weaving of the multi-source heterogeneous data.

[0137] Preferably, the specific implementation process of step 531 is as follows: First, the bottleneck logic node list generated in step 523 is used as the processing object. This list includes "unique node identifier - total influence weight - associated task type - list of affected nodes". The output results of the random forest classification network in step 422 and the LSTM time series prediction network in step 423 are loaded simultaneously. The random forest classification network (containing 15 CART decision trees, with splitting features configured according to "task business type > priority > resource requirement") has output the "resource requirement type (high CPU / high I / O / balanced)" of the tasks associated with the bottleneck nodes. For example, the resource requirement type of the real-time transaction associated node is "high CPU". The LSTM time series prediction network (containing 2 hidden layers, 64 LSTM units per layer, with a dropout rate of 0.2) The load trend (rising / stable / falling) for the bottleneck node over the next 5 minutes has been output, such as a high-CPU node showing a "continuously rising" load trend. Next, anomaly analysis is performed on the bottleneck logic node, classifying it into three categories: 1) resource adaptation anomalies (e.g., high-CPU demand nodes consistently exceed constraint thresholds in CPU utilization, and LSTM predicts an increasing load); 2) functional deficiency anomalies (e.g., nodes associated with highly sensitive fields lack integrated encryption modules, failing to meet security constraints); and 3) link adaptation anomalies (e.g., incompatibility between transmission node protocols and downstream nodes, causing data transmission delays exceeding business constraints). Subsequently, based on the anomaly type and model output, targeted path optimization solutions are developed: for resource adaptation anomalies (high CPU + rising load), the solution is "node CPU parameter adjustment (increasing the number of cores by 20%) +..." The solution includes load balancing (migrating 30% of low-priority tasks to backup nodes); for functional missing anomalies (lack of integrated encryption module), the solution is "module integration (adding the national cryptographic SM4 encryption module) + encryption parameter configuration (key rotation cycle set to 24 hours)"; for link adaptation anomalies (protocol incompatibility), the solution is "protocol conversion submodule deployment (supporting GRPC and MQTT protocol conversion) + link latency monitoring (collecting transmission latency every 1 second)"; finally, the solution integrates "unique identifier of bottleneck node - anomaly type - optimization scheme details (adjusting parameters / modules / submodules) - related model basis (random forest resource requirements / LSTM load trends)" to generate a path optimization scheme table. This technology differs from the traditional "one-size-fits-all" optimization scheme. By combining the resource requirement judgment of random forest with the load trend prediction of LSTM, the solution can not only solve the current anomaly but also cope with future load changes. For example, for nodes with "high CPU + rising load", not only are current resources increased, but tasks are also pre-loaded to avoid the bottleneck from reappearing in a short period of time.

[0138] Preferably, in the scenario of segmenting data weaving paths (real-time transaction path, batch statistics path, static report path) in multiple enterprise scenarios, step 532 is specifically implemented as follows: First, based on the data weaving path of the logical data carrier generated in step 332, according to the business link logic of "data source access - data preprocessing - core processing - data transmission - result output", segment decomposition processing is performed on each path containing a bottleneck node - for example, the real-time transaction path "data source access node A → data verification node B (bottleneck node) → real-time computing node C → transmission node D → output node E" is decomposed into "access segment (A) - preprocessing segment (B) - core processing segment (C) - transmission segment (D) - output segment (E)", and the node dependency relationship of each segment is recorded at the same time (such as the output of the preprocessing segment being the input of the core processing segment); then, the target path segment where the bottleneck logic node is located is located: from the path in step 531 The optimization scheme extracts the unique identifier of the bottleneck node from the table and matches it with the list of decomposed segment nodes to determine the segment to which the bottleneck node belongs (e.g., bottleneck node B belongs to the preprocessing segment). Then, it extracts the key attributes of the target path segment, including the business scenario associated with the segment (real-time / batch / static), the number and type of nodes within the segment, the input and output data format of the segment (e.g., JSON / Parquet), and the average current resource consumption of the segment (CPU / memory / latency). Finally, it organizes "Path ID - Target Path Segment (e.g., preprocessing segment) - Segment Key Attributes - Location of Bottleneck Node in Segment" to generate a target path segment location table. This technology differs from traditional methods that optimize directly without decomposition. By decomposing segments according to business links, subsequent optimization is performed only on the target segment, avoiding interference with segments without bottlenecks (e.g., optimizing only the preprocessing segment does not affect the normal operation of core processing and transmission segments), thus improving optimization efficiency.

[0139] Preferably, in the specific technical implementation of step 533: First, using the path optimization scheme table of step 531 and the target path segment location table of step 532 as dual inputs, the "node parameter adjustment - module integration - link adaptation" process optimization is performed on the target path segment; the first step is to perform node parameter adjustment: taking the preprocessing segment with "high CPU demand + increased load" as an example, the node parameter configuration interface is called to increase the number of CPU cores of bottleneck node B from 4 cores to 5 cores (an increase of 20%), while adjusting the CPU scheduling policy to "real-time task priority scheduling", and configuring a load threshold warning for node B in advance based on the load trend predicted by LSTM (the warning is triggered when the CPU utilization exceeds 80%); the second step is to perform module integration: for the transmission segment with "functional missing anomaly", the module management interface is called to add the national cryptographic SM4 encryption module to the bottleneck transmission node D, automatically loading the preset encryption key in the management metadata (path: / conf / security / sm4.key), and configuring the operating parameters of the encryption module (encryption block size = 128 bits, key rotation cycle = 24 hours), and after integration, the module is verified through the interface. The block runs in progress (if "Running normally" is returned, proceed to the next step); the third step is to perform link adaptation: for the core processing segment with "link adaptation error", deploy the GRPC-MQTT protocol conversion submodule, read the "interface path / parameter name" mapping rule generated in step 232, and configure the parameters of the conversion submodule (input interface path = / grpc / core / process, output interface path = / mqtt / transmit, parameter name mapping = "process_id → transmit_id") to ensure that the GRPC protocol of the optimized core processing node C is compatible with the MQTT protocol of the downstream transmission node D; finally, collect real-time resource consumption data (CPU utilization, latency, encryption module status) of the optimized target path segment and generate an evaluation detail of the optimized target path segment; the special design of this technology is that parameter adjustment, module integration, and link adaptation are executed in the order of "first basic resource adjustment, then function completion, and finally link compatibility", to avoid resource conflicts when adjusting parameters due to module integration first. At the same time, a status check is performed after each optimization step to ensure that the optimization action is effectively implemented.

[0140] Preferably, the specific implementation process of step 534 is as follows: First, the multi-protocol adaptation module (supporting multiple protocol conversions such as SQL, RESTful API, MQTT, and GRPC) built in step 232 is called. Taking the optimized target path fragment generated in step 533 and the non-optimized fragment in the original path as processing objects, protocol compatibility verification is performed: the output protocol of the optimized target path fragment (e.g., the output protocol of the preprocessed fragment is GRPC) and the input protocol of the non-optimized fragment in the original path (e.g., the input protocol of the core processing fragment is GRPC) are extracted. If the protocols are consistent, they are directly concatenated; if the protocols are inconsistent (e.g., the output protocol of the optimized transmission fragment is MQTT, and the input protocol of the original output fragment is HTTP), the protocol conversion submodule is called to perform protocol conversion according to the "interface path / parameter name" mapping rule (converting the MQTT message into an HTTP request, mapping the parameter name "transmit_data" to "http_data"), and generating protocol-compatible fragment connection data; then, the optimized target path fragment and the non-optimized fragment are concatenated according to the node dependency relationship of the original path to form a complete concatenated path; subsequently, the data generated in step 513 is loaded. The process involves optimizing the path constraints and performing constraint checks on the spliced ​​path: checking resource constraints (e.g., whether the CPU / memory utilization of all nodes in the spliced ​​path is ≤ threshold), business constraints (e.g., whether the total path latency is ≤ 500ms), and security constraints (e.g., whether the encryption module of highly sensitive field nodes is running normally). If the checks pass, the spliced ​​path is used as the optimized data weaving path. If the checks fail (e.g., the latency of the core processing node after splicing is 550ms > 500ms), the process returns to step 531 to re-analyze the anomaly type (determined to be "link adaptation anomaly not fully resolved"), adjusts the optimization plan (e.g., adding 1 CPU core to the core processing node), and repeats steps 531-533 until the spliced ​​path meets all constraints. This technology differs from the traditional method of not checking after splicing. Through a closed-loop process of "protocol verification - splicing - constraint verification," it ensures that the optimized path not only solves the original bottleneck problem but also meets the overall constraint requirements, avoiding new performance or security risks after optimization.

[0141] Preferably, in the scenario of integrating multi-source heterogeneous data (MySQL order table, MongoDB customer table, and JSON format IoT device logs), step 535 is specifically implemented as follows: First, the optimized data weaving path generated in step 534 is called, and the metadata dataset generated in step 133 (containing technical / business / operation / management metadata) is loaded to determine the processing requirements of the multi-source heterogeneous data—field association is performed on the MySQL order table (structured data) (associating "order ID" with "order ID" in the customer table), format conversion is performed on the MongoDB customer table (semi-structured data) (converting to Parquet format to improve transmission efficiency), and secure transmission is performed on the IoT device logs (JSON format) (based on the high-sensitivity field annotation of management metadata, SM4 encryption is performed on the "device key" field); then, processing is performed according to the node flow logic of the optimized path: the data source access node reads the multi-source heterogeneous data, the preprocessing node performs field cleaning (removing null value order records in the order table), and the core processing node performs field association (associating order table and customer table data by "order ID") and format conversion. The nodes convert the associated data from JSON / MySQL format to Parquet format. Encryption nodes perform SM4 encryption on data containing highly sensitive fields. Output nodes write the processed data to the target data warehouse (e.g., Hive). During this process, key performance indicators (SPIs) of the path execution are collected in real time (field association success rate, format conversion time, encryption transmission latency), historical execution indicators in the association operation metadata, and the performance improvement percentage of this execution is calculated (e.g., encryption transmission latency decreased from 600ms to 350ms, an improvement of approximately 42%). Finally, a data weaving processing report is generated, including "Optimized Path ID - Multi-source Data Processing Volume - Key Indicators (Association Success Rate 98%+, Conversion Time 20s, Transmission Latency 350ms) - Performance Improvement Percentage," completing the data weaving of multi-source heterogeneous data. This technology differs from traditional manual script writing methods. By calling optimized adaptive paths, it automatically completes the association, transformation, and secure transmission of multi-source data. Furthermore, the processing incorporates the business and security requirements of the metadata, ensuring that the output data conforms to both business semantics and security specifications, while achieving a significant performance improvement compared to before optimization.

[0142] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.< / virtualtableinfo> < / fields> < / field> < / relation> < / relations> < / field> < / field> < / fields> < / virtualtableinfo> < / virtualtableinfo> < / querytemplates> < / indexes> < / relations> < / fields> < / virtualtableinfo> < / virtualtabledefinition>

Claims

1. A performance optimization method for adaptive data weaving based on artificial intelligence, characterized in that, include: Step 1: Obtain multi-source heterogeneous data based on the data source list and parse it to generate a meta-dataset that contains at least technical metadata, business metadata, operational metadata, and management metadata; Step 2: Based on the virtual table interface, perform virtual table abstraction and encapsulation on the metadata dataset to generate a logical data carrier; Step 3: Determine the operational attributes of the metadata dataset to generate a data weaving path for the logical data carrier based on the operational attributes; Step 4: Monitor the data access task queue and access indicator feedback in real time, extract resource load correlation features from the data access task queue and access indicator feedback based on the constructed resource scheduling model, and generate resource scheduling strategies based on the resource load correlation features. Step 5: Optimize the data weaving path of the logical data carrier based on the resource scheduling strategy to obtain the optimized data weaving path for data weaving of multi-source heterogeneous data; Step 2 specifically includes the following steps: Step 21: Based on the virtual table interface, perform dynamic mapping configuration processing on the metadata dataset between metadata fields and virtual table structure to obtain the metadata-virtual table field mapping relationship; Step 22: Perform virtual table structure abstraction and construction on the metadata-virtual table field mapping relationship to generate a virtual table structure definition file; Step 23: Perform interface adaptation and functional encapsulation processing on the virtual table structure definition file to generate a logical data carrier; Step 4 specifically includes the following steps: Step 41: Collect data access task queue information in real time by deploying a distributed task monitoring agent, and collect access indicator data through the indicator collection module to monitor data access task queue and access indicator feedback. Step 42: Based on the constructed resource scheduling model including LSTM time series prediction network and random forest classification network, perform preliminary feature extraction processing on the data access task queue and access index feedback to obtain task resource demand features and index load trend features. Perform correlation and fusion processing on task resource demand features and index load trend features to extract resource load correlation features. Then match the resource load correlation features with the preset resource scheduling rule base to generate resource scheduling strategy. Step 5 specifically includes the following steps: Step 51: Generate path optimization constraints based on resource scheduling strategies; Step 52: Analyze and evaluate the current resource consumption of the data weaving path of the logical data carrier, and identify the bottleneck logical nodes in the data weaving path that do not match the path optimization constraints. Step 53: Optimize the data weaving path of the logical data carrier based on the bottleneck logical node to obtain the optimized data weaving path for data weaving of multi-source heterogeneous data.

2. The method according to claim 1, characterized in that, Step 1 specifically includes the following steps: Step 11: Perform type parsing on the multi-source heterogeneous data sources in the preset data source list to identify the storage engine type and transmission protocol type, and generate a detailed list of data source types accordingly; Step 12: Based on the detailed list of data source types, perform protocol adaptation on multi-source heterogeneous data sources to obtain multi-source heterogeneous data; Step 13: Perform data field segmentation on the multi-source heterogeneous data to obtain a structured field set. Perform metadata parsing on the structured field set to generate a metadata dataset that includes at least technical metadata, business metadata, operational metadata, and management metadata.

3. The method according to claim 1, characterized in that, Step 21 specifically includes the following steps: Step 211: Based on the metadata parsing specification of the virtual table interface, perform field attribute extraction processing on the metadata dataset to generate metadata field attribute features; Step 212: Based on the attribute characteristics of the metadata fields, dynamically match them with the virtual table fields to generate a candidate set of virtual table mappings; Step 213: Based on the initial mapping candidate set of the virtual table, generate the metadata-virtual table field mapping relationship.

4. The method according to claim 1, characterized in that, Step 3 specifically includes the following steps: Step 31: Determine the operational attributes of the metadata dataset by extracting operational attribute features from the metadata dataset; Step 32: Based on the operation attributes, perform multi-target path mapping on the logical data carrier to establish the correspondence between "operation attributes - logical carrier nodes - data interaction protocols"; Step 33: Based on the correspondence between "operation attribute - logical carrier node - data interaction protocol", generate the data weaving path of the logical data carrier.

5. The method according to claim 4, characterized in that, Step 31 specifically includes the following steps: Step 311: Perform structured parsing of the operational metadata and business metadata in the metadata set to determine the basic information related to the operational attributes of the metadata dataset; Step 312: Based on the determined basic information, perform feature extraction on the metadata dataset for "average response time", "peak access frequency", "business process urgency" and "data update timeliness requirements" to obtain operational attribute features; Step 313: Cluster and label the operational attribute features to determine the operational attributes of the metadata dataset.

6. The method according to claim 1, characterized in that, Step 42 specifically includes the following steps: Step 421: Perform standardized preprocessing on the data access task queue and access indicator feedback to generate task data matrix and indicator time series data matrix respectively; Step 422: Input the task data matrix into a random forest classification network for feature extraction to obtain the task resource requirement features; Step 423: Input the indicator time series data matrix into the LSTM time series prediction network to capture the time dependency of the indicator data, output the indicator load prediction value, and predict the load rise rate and load fluctuation amplitude based on the prediction value to obtain the indicator load trend characteristics. Step 424: Construct a feature weight mapping table based on the operation metadata in the metadata set to perform weighted fusion of task resource demand features and indicator load trend features to obtain resource load correlation features; Step 425: Traverse and match the resource load association features with the preset resource scheduling rule base, and generate specific resource scheduling strategies based on the matching results.

7. The method according to claim 1, characterized in that, Step 51 specifically includes the following steps: Step 511: Perform structured parsing on the core instructions in the resource scheduling strategy to generate strategy instruction parsing factors; Step 512: Based on the operational metadata and management metadata in the metadata set, determine the constraint dimensions and corresponding constraint indicators according to the policy instruction parsing factors; Step 513: Based on the constraint dimensions and constraint indicators, generate path optimization constraints that are categorized and integrated according to "resource constraints - business constraints - security constraints".

Images