Intelligent storage system supporting multi-source heterogeneous data fusion management

Abstract

The invention discloses an intelligent storage system supporting multi-source heterogeneous data fusion management, which relates to the technical field of multi-source heterogeneous data fusion, and comprises a data source adaptation module, an intelligent data source adapter is configured in a DataWorks data integration module, multi-source heterogeneous data is accessed through the intelligent data source adapter, and the data source adaptation module is connected with the DataWorks data integration module; extracting a semantic feature vector and a technical feature set; the neural symbol hybrid inference module inputs the semantic feature vector and the technical feature set into a neural symbol hybrid inference engine, calculates a similarity matrix between multi-source heterogeneous data fields through a BERT-based neural network, and imports the technical feature set into a field knowledge graph constructed by Neo4j for symbol logic verification to generate a unified metadata model; according to the method, a semantic similarity matrix calculated by a BERT-based neural network is combined with symbol logic verification of a Neo4j knowledge graph through a neural symbol hybrid inference engine, so that automatic semantic alignment and logic consistency verification of multi-source heterogeneous data are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-source heterogeneous data fusion, and in particular to an intelligent storage system supporting multi-source heterogeneous data fusion management. Background Art

[0002] Against the backdrop of the rapid development of big data and cloud computing technologies, the efficient integration and management of multi-source, heterogeneous data has become a core challenge in building enterprise data centers. Among existing technologies, Alibaba Cloud's DataWorks and Hologres combination is widely used in data integration and storage management. It extracts, transforms, and loads multi-source data through standardized ETL processes, and leverages a distributed storage engine to provide highly concurrent data query services. A financial enterprise uses DataWorks' data integration module to connect to heterogeneous data sources such as MySQL and Kafka, and builds a tiered storage strategy in Hologres to optimize query performance and storage costs. It performs well in batch processing scenarios for structured data and can meet basic data synchronization and storage needs.

[0003] However, existing technologies still have room for improvement in the intelligent fusion and dynamic storage management of multi-source heterogeneous data. Traditional solutions rely on manually configured data mapping rules and storage policies, making it difficult to adaptively identify semantic relationships between different data sources. This limits the flexibility and accuracy of metadata management. When enterprises need to integrate customer behavior data from CRM systems and log services, they must manually write field matching logic, and are unable to dynamically predict access patterns of different data to optimize storage tiering. This limitation makes it difficult to further improve data fusion efficiency and storage resource utilization, especially in scenarios where business tags change dynamically or data access patterns fluctuate significantly. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides an intelligent storage system that supports multi-source heterogeneous data fusion management to solve the problems that traditional methods rely on manual rule configuration and are difficult to adaptively fuse multi-source data and optimize storage resources.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] The present invention provides an intelligent storage system that supports multi-source heterogeneous data fusion management, which includes an intelligent data source adapter module. The intelligent data source adapter is configured in the DataWorks data integration module, accesses multi-source heterogeneous data through the intelligent data source adapter, and extracts semantic feature vectors and technical feature sets.

[0008] The neural-symbolic hybrid reasoning module inputs the semantic feature vector and technical feature set into the neural-symbolic hybrid reasoning engine. Using the BERT-based neural network, it calculates the similarity matrix between multi-source heterogeneous data fields. It then imports the technical feature set into the domain knowledge graph built by Neo4j for symbolic logic verification and generates a unified metadata model.

[0009] The dynamic storage optimization module annotates business tags in a unified metadata model, uses an LSTM-Hawkes hybrid model to calculate the expected access frequency of multi-source heterogeneous data fields, and generates a storage configuration list.

[0010] The storage topology construction module sends the data to the Hologres execution engine through the DataWorks workflow based on the storage configuration, creates logical tablespaces and physical storage partitions, and obtains the storage topology map.

[0011] The performance monitoring module monitors the performance indicators of the storage topology map to generate performance reports, combines the storage topology map and the unified metadata model through the data service view layer of Hologres, and provides a secure watermarked service catalog.

[0012] As a preferred solution of the intelligent storage system supporting multi-source heterogeneous data fusion management described in the present invention, wherein: accessing multi-source heterogeneous data through an intelligent data source adapter, extracting semantic feature vectors and technical feature sets, including the following steps,

[0013] Automatically scan multi-source heterogeneous data configuration information sets through intelligent data source adapters, identify data protocols and output protocol identifiers;

[0014] Load the data parser according to the protocol identifier and convert the multi-source heterogeneous data into a structured record set;

[0015] Perform field-level semantic analysis on structured record sets to extract semantic feature vectors and technical feature sets.

[0016] As a preferred solution of the intelligent storage system supporting multi-source heterogeneous data fusion management described in the present invention, the semantic feature vector and technical feature set are input into the neural symbolic hybrid reasoning engine, and the similarity matrix between the multi-source heterogeneous data fields is calculated through the BERT-based neural network, including the following steps:

[0017] The semantic feature vector is input into the BERT-based neural network and the enhanced semantic representation is generated through multi-layer Transformer encoding;

[0018] Normalize the enhanced semantic representation to obtain a unit vector, and calculate the cosine similarity matrix of the unit vector;

[0019] The technical feature set is imported into the similarity correction module, and the conflicting items in the cosine similarity matrix are adjusted through the rule engine to obtain the similarity matrix.

[0020] As a preferred solution of the intelligent storage system supporting multi-source heterogeneous data fusion management described in the present invention, the following steps are included: importing the technical feature set into the domain knowledge graph constructed by Neo4j for symbolic logic verification to generate a unified metadata model:

[0021] Convert the technical feature set into Neo4j Cypher statements to build the initial knowledge graph;

[0022] Execute predefined symbolic reasoning rules on the initial knowledge graph, mark logical conflicts and generate conflict reports;

[0023] Correct the conflicting nodes and relationships in the initial knowledge graph based on the conflict report to obtain a conflict-free knowledge graph;

[0024] Extract globally consistent field mapping rules from the conflict-free knowledge graph to generate a unified metadata model.

[0025] As a preferred solution of the intelligent storage system supporting multi-source heterogeneous data fusion management described in the present invention, wherein: marking business tags in the unified metadata model includes the following steps:

[0026] Extract business semantic descriptions and associations of multi-source heterogeneous data fields from the unified metadata model to generate an original tag set;

[0027] Perform standardized matching on the original tag set through business classification rules to obtain a standardized tag candidate set;

[0028] Use the domain knowledge graph to verify the normalized label candidate set and generate a valid label set.

[0029] Bring the valid tag set into the extended attribute layer of the unified metadata model to form business tags.

[0030] As a preferred solution of the intelligent storage system supporting multi-source heterogeneous data fusion management described in the present invention, the following steps are included: using the LSTM-Hawkes hybrid model to calculate the expected access frequency of multi-source heterogeneous data fields and generate a storage configuration list.

[0031] Time series data collected from field-level operation records in DataWorks task logs and Hologre;

[0032] Input the time series data into the LSTM module to obtain the basic visit rate prediction;

[0033] Input the time series data into the Hawkes process module, calculate the event trigger intensity α and the attenuation coefficient β, and obtain the short-term burst access correction term;

[0034] The basic access rate forecast and the short-term burst access correction term are combined to generate a comprehensive access frequency forecast;

[0035] Match comprehensive access frequency predictions with business tags to generate field-level configuration items;

[0036] Aggregate field-level configuration items to generate a storage configuration list.

[0037] As a preferred solution of the intelligent storage system supporting multi-source heterogeneous data fusion management described in the present invention, wherein: according to the storage configuration, the DataWorks workflow is sent to the Hologres execution engine, logical tablespaces and physical storage partitions are created, and a storage topology diagram is obtained, including the following steps:

[0038] Parse the field-level storage policy in the storage configuration list and generate a set of Hologres-compatible DDL statements;

[0039] The Hologres execution engine is called through the DataWorks workflow to create a logical tablespace based on the DDL statement set.

[0040] Create partitions in the Hologres physical storage layer according to the C_storage partitioning rules and build an index structure that matches the logical tablespace.

[0041] Collect metadata about logical tablespaces, partitions created at the physical storage layer, and index structures to generate a storage topology diagram.

[0042] As a preferred solution of the intelligent storage system supporting multi-source heterogeneous data fusion management described in the present invention, wherein: monitoring the performance indicators of the storage topology map to generate a performance report includes the following steps:

[0043] Extract the monitoring target list from the storage topology map, configure the Hologres session activity status view and storage monitoring view monitoring items, and obtain the data collection flow;

[0044] Aggregate the data collection stream into time windows to generate a structured indicator set;

[0045] Compare the structured indicator set with the indicators of the storage configuration list to detect abnormal deviations and obtain the abnormal event set;

[0046] Generate visual performance reports based on structured indicator sets and abnormal event sets.

[0047] As a preferred solution of the intelligent storage system supporting multi-source heterogeneous data fusion management described in the present invention, the storage topology map and the unified metadata model are combined through the data service view layer of Hologres, and the service directory of the security watermark includes the following steps.

[0048] Create virtualized view mapping rules in Hologres, associate the physical tables in the storage topology diagram with the logical fields of the unified metadata model, and generate a view definition set;

[0049] Construct a cross-source unified view layer according to the view definition set, and inject the GAN watermark into the output stream of the cross-source unified view layer;

[0050] Access the protocol and watermark verification interface in the cross-source unified view layer to generate a service catalog of secure watermarks.

[0051] The beneficial effects of the present invention are as follows: through the neural-symbolic hybrid reasoning engine, the semantic similarity matrix calculated by the BERT-based neural network is combined with the symbolic logic verification of the Neo4j knowledge graph, thereby realizing the automatic semantic alignment and logical consistency verification of multi-source heterogeneous data, solving the limitation of the traditional method that relies on manual rule configuration, and finally generating a globally consistent unified metadata model to ensure the accuracy of data fusion and business interpretability. Through the LSTM-Hawkes hybrid model, the time series prediction and event triggering mechanism are integrated to dynamically capture the long-term trend and short-term burst pattern of data access, generate an accurate storage configuration list, realize the adaptive hierarchical optimization of storage resources, and avoid the resource waste or performance bottleneck of static strategies. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 Schematic diagram of an intelligent storage system that supports fusion management of multi-source heterogeneous data.

[0054] Figure 2 A diagram showing the visual performance report. DETAILED DESCRIPTION

[0055] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0056] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0057] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0058] Reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, provides an intelligent storage system that supports multi-source heterogeneous data fusion management, including the following steps:

[0059] The intelligent data source adapter module configures an intelligent data source adapter in the DataWorks data integration module, accesses multi-source heterogeneous data through the intelligent data source adapter, and extracts semantic feature vectors and technical feature sets.

[0060] Configure the intelligent data source adapter in the DataWorks data integration module.

[0061] Create a data integration task in the DataWorks console and select Intelligent Data Source Adapter as the data source type.

[0062] To proceed further, log in to the DataWorks console, go to the Data Integration page, and click the New Task button. On the Task Configuration page, enter basic information such as the Task Name and Description. Select the Smart Data Source Adapter option from the Data Source Type drop-down menu. Configure the Smart Data Source Adapter connection information, including the connection address, port number, database name, username, and password. After successfully testing the connection, set the data synchronization direction, select the source and target table mappings, ensure that the source and target table field types match when configuring field mappings, set the task scheduling properties, including the scheduling period and scheduling dependencies, save the task configuration, and submit it for publication.

[0063] Enable the intelligent protocol identification function and configure the intelligent data source adapter.

[0064] Going further, go to the Smart Data Source Adapter configuration page and select the Enable Smart Protocol Identification option in the Advanced Settings section. Smart Protocol Identification will automatically analyze the data source connection protocol type, such as MySQL, Oracle, or PostgreSQL. Configure the Smart Protocol Identification parameters, including the protocol detection timeout (60 seconds in this example) and the protocol matching threshold (0.8 in this example). After saving the configuration, the Smart Data Source Adapter will automatically identify the data source protocol type and establish the corresponding connection. When testing the connection, Smart Protocol Identification will verify the protocol matching result. Once the protocol identification result is correct, complete the final configuration of the Smart Data Source Adapter and submit it.

[0065] Access multi-source heterogeneous data through intelligent data source adapters to extract semantic feature vectors and technical feature sets.

[0066] Automatically scan multi-source heterogeneous data configuration information sets through intelligent data source adapters, identify data protocols and output protocol identifiers.

[0067] Furthermore, the data source connection configuration set to be scanned is loaded, and the intelligent data source adapter establishes a temporary connection with each data source in turn, sends a standard protocol detection request and collects response data, and the intelligent data source adapter analyzes the response data characteristics, matches the predefined protocol feature library (such as MySQL protocol features, Oracle protocol features, PostgreSQL protocol features), and calculates the protocol matching score. When the protocol matching score exceeds the set threshold (the example value is 0.75), the intelligent data source adapter determines the protocol type and generates a corresponding protocol identifier. The intelligent data source adapter associates the identified protocol identifier with the data source configuration information and stores it, and outputs a result set containing the data source name, connection address and protocol identifier. After the scan is completed, the intelligent data source adapter generates a scan report, recording the successfully identified data source protocols and the data source information that was not identified.

[0068] Load the data parser according to the protocol identifier and convert multi-source heterogeneous data into a structured record set.

[0069] Furthermore, the intelligent data source adapter loads the corresponding data parser from the parser repository based on the protocol identifier. The data parser establishes a connection with the data source according to the protocol specifications, reads the raw data stream or data file, analyzes the data format characteristics, identifies field delimiters, and disassembles the raw data according to the protocol-defined format, extracts field names and values, and maps them into structured fields. The data parser performs data type conversion, converting the source data format into the data type corresponding to the target structure. The data parser verifies each record, filters out abnormal records that do not meet the format requirements, and generates a structured record set containing complete metadata information.

[0070] Perform field-level semantic analysis on structured record sets to extract semantic feature vectors and technical feature sets.

[0071] Furthermore, a semantic feature dictionary is constructed based on field names and field values. The semantic analyzer scans the keywords in the field names and calculates the semantic relevance score in combination with the field value content. The semantic analyzer extracts field-level technical features, including data type (such as string type, numeric type, date type), length limit (example value is 255 bytes), precision range (example value is 2 decimal places) and value distribution characteristics. The semantic analyzer converts the field name into a standardized semantic identifier through stem extraction and synonym mapping. The semantic analyzer generates a field-level semantic feature vector, which contains quantitative indicators of name similarity and value domain matching (three dimensions. The technical feature set records the storage format of each field (such as fixed length, variable length), compression flag (such as yes / no) and encryption status (such as AES-256), and outputs a complete analysis result table containing field name, semantic feature vector and technical feature set.

[0072] The neural-symbolic hybrid reasoning module inputs the semantic feature vector and technical feature set into the neural-symbolic hybrid reasoning engine, forms a similarity matrix between multi-source heterogeneous data fields through the BERT-based neural network, imports the technical feature set into the domain knowledge graph built by Neo4j for symbolic logic verification, and generates a unified metadata model.

[0073] The semantic feature vector and technical feature set are input into the neural-symbolic hybrid reasoning engine, and the similarity matrix between multi-source heterogeneous data fields is calculated through the BERT-based neural network.

[0074] The semantic feature vector is input into the BERT-based neural network and the enhanced semantic representation is generated through multi-layer Transformer encoding.

[0075] Furthermore, each field name and example data of the structured record set is encoded based on the pre-trained BERT model to generate an initial semantic vector. The field context description is weighted and aggregated using the attention mechanism. For example, the field annotations or business definitions in the data dictionary are combined to enhance the distinctiveness of the semantic representation and form a semantic feature vector. The extraction of the technical feature set is achieved by parsing the metadata attributes of the structured record set, including field data types, constraints, index types, and technical parameters specific to the data source, such as the character set configuration of MySQL or the number of topic partitions of Kafka, and outputting a standardized technical feature set.

[0076] The enhanced semantic representation is normalized to obtain a unit vector, and the cosine similarity matrix of the unit vector is calculated.

[0077] Furthermore, the L2 normalization method is used to process the enhanced semantic representation output by the BERT-based neural network. The Euclidean length of each vector is scaled to 1, and unit vectors are generated. The dot product between all unit vectors is calculated through matrix multiplication. Since the dot product of unit vectors is equivalent to cosine similarity, the cosine similarity matrix between multi-source heterogeneous data fields is finally output. The matrix elements range from -1 to 1. The closer the value is to 1, the higher the semantic similarity of the field.

[0078] Specifically, the expression is,

[0079] S=U U T ;

[0080] Among them, S is the cosine similarity matrix, U is the unit vector matrix, and U T is the transposed matrix of the unit vector matrix U

[0081] The technical feature set is imported into the similarity correction module, and the conflicting items in the cosine similarity matrix are adjusted through the rule engine to obtain the similarity matrix.

[0082] Furthermore, the field constraints and data types recorded in the technical feature set are analyzed, and the conflict judgment logic is predefined in the rule engine. For example, when the two field data types are incompatible or the constraints conflict, the value of the corresponding position in the cosine similarity matrix is ​​forced to be adjusted to 0. All elements of the cosine similarity matrix are traversed, and the correction coefficient output by the rule engine is applied for weighted adjustment to generate a similarity matrix that ultimately eliminates technical conflicts.

[0083] The technical feature set is imported into the domain knowledge graph built by Neo4j for symbolic logic verification to generate a unified metadata model.

[0084] Convert the technical feature set into Neo4j Cypher statements to build the initial knowledge graph.

[0085] Furthermore, the field metadata in the technical feature set is parsed, including field name, data type, constraints and associations. According to the node-relationship model of the Neo4j graph database, each field is converted into a labeled node. The foreign key references or logical associations between fields are then converted into relationship edges. The generated Cypher statement set is executed to build an initial knowledge graph in Neo4j containing complete field nodes and associations.

[0086] Execute predefined symbolic reasoning rules on the initial knowledge graph, mark logical conflicts and generate conflict reports.

[0087] Furthermore, a predefined set of symbolic reasoning rules is loaded. The rules are expressed in Cypher syntax and are used to detect foreign key references with type mismatches. These rule queries are executed one by one on the initial knowledge graph, and the returned results are recorded as potential conflict instances. Each potential conflict instance is then given a manually preset severity rating, for example, data type conflicts are marked as severe, and naming inconsistencies are marked as warnings. A structured conflict report is generated based on the conflict type, involved fields, and severity. The report entries include conflict descriptions, related fields, and repair suggestions.

[0088] According to the conflict report, the conflicting nodes and relationships in the initial knowledge graph are corrected to obtain a conflict-free knowledge graph.

[0089] Furthermore, the conflicting entries recorded in the conflict report are parsed and processed according to the conflict type. For example, for field type mismatch conflicts, Cypher statements are executed to modify the node type attribute. For missing foreign key relationships, corresponding relationship edges are created. Batch update operations are performed in Neo4j to remove data type conflicts. The predefined symbolic reasoning rules are re-executed on the corrected knowledge graph to verify that all conflicting entries have been eliminated and no new conflicts have been introduced, thus generating a final conflict-free knowledge graph.

[0090] Extract globally consistent field mapping rules from the conflict-free knowledge graph to generate a unified metadata model.

[0091] Furthermore, predefined Cypher query statements are executed on the conflict-free knowledge graph to extract all field nodes and their associations. The query results are then converted into standardized mapping rules, including field naming rules (such as underscore naming) and data type conversion rules. Based on these rules, a structured definition of a unified metadata model is constructed. The unified metadata model contains core metadata attributes such as field name, data type, constraints, and cross-source associations. It is verified whether the unified metadata model completely covers all fields and relationships in the conflict-free knowledge graph to ensure the consistency and completeness of the model.

[0092] The dynamic storage optimization module marks business tags in the unified metadata model, uses the LSTM-Hawkes hybrid model to calculate the expected access frequency of multi-source heterogeneous data fields, and generates a storage configuration list.

[0093] Mark business tags in the unified metadata model.

[0094] Extract the business semantic descriptions and association relationships of multi-source heterogeneous data fields from the unified metadata model to generate the original tag set.

[0095] Furthermore, the extended attribute layer of each field in the unified metadata model is parsed, the business semantic description text is extracted, the relationship definitions between fields are traversed, the business-level relationship descriptions (such as customer-order or product-inventory) are recorded, the extracted semantic descriptions are segmented and keywords are extracted, the core business terms are retained after removing stop words, and the field name, business semantic description and relationship description are combined into a structured record to form the original tag set.

[0096] The original tag set is standardized and matched according to the business classification rules to obtain the normalized tag candidate set.

[0097] Furthermore, a predefined business classification rule base is loaded. The rules adopt the mapping table form of "business term → standard label". Term matching is performed on each business semantic description in the original label set, and the corresponding standard term is found in the business classification rule base using the longest prefix matching algorithm. Then, the successfully matched business semantic description is assigned a standard label, and the unmatched description retains the original text and is marked for review. Finally, the standardized results are merged with the field identifier and the association relationship description to generate a normalized label candidate set containing the standard label, the original description and the matching status.

[0098] Use the domain knowledge graph to verify the normalized label candidate set and generate a valid label set.

[0099] Furthermore, the business entity nodes related to the standardized label candidate set are located in the domain knowledge graph, such as core concepts such as customer and order. Then, a Cypher query is executed to verify the semantic consistency of the standard label, confirm whether the label belongs to the correct business entity, check the contextual relevance of the unmatched original description, and determine whether a new label should be created through relationship path analysis (such as "customer-order-order"). Finally, the verified standard labels and the reviewed new labels are screened, and duplicate or conflicting label items are eliminated to form a valid label set containing field identifiers, standard labels and semantic context.

[0100] Bring the valid tag set into the extended attribute layer of the unified metadata model to form business tags.

[0101] Furthermore, the field identifiers in the valid tag set are parsed, the corresponding field definition nodes are located in the unified metadata model, and business tag attributes are created in the extended attribute layer of the field definition node. The attribute value adopts the structured format of "tag type: tag value". The standard tags recorded in the valid tag set are written into the value domain of the business tag attribute, and the semantic context is attached as an annotation. All field nodes in the unified metadata model are traversed to ensure that the business tag attributes completely cover all fields defined in the valid tag set, and generate standardized business tags.

[0102] The LSTM-Hawkes hybrid model is used to calculate the expected access frequency of multi-source heterogeneous data fields and generate a storage configuration list.

[0103] Time series data collected from field-level operation records in DataWorks task logs and Hologre.

[0104] Furthermore, we extract field-level operation records from DataWorks task logs, including SELECT and UPDATE operation types, operation timestamps, and involved field names. We also obtain query performance metrics from the Hologre execution engine monitoring interface, recording field access time, response time, and concurrent access volume. We then align and aggregate the two types of data according to time windows to generate structured time series data containing timestamps, field identifiers, operation types, access counts, and performance metrics. We then verify the integrity and continuity of the time series data to ensure there are no missing time windows or abnormal breakpoints.

[0105] Input the time series data into the LSTM module to obtain the basic visit rate prediction.

[0106] Furthermore, the time series data is standardized, and features such as the number of visits and response time are scaled to the same dimension. The standardized time series data is then grouped by field identifier to form an input sequence in time steps (for example, 24 consecutive hours of data). The input sequence is then input into a pre-trained LSTM module. The LSTM module captures long-term dependencies through a forget gate, input gate, and output gate mechanism, and outputs a basic visit rate prediction value for the next N time windows. The prediction result is then denormalized and restored to the actual visit volume range, generating a basic visit rate prediction result that includes field identifiers, predicted time points, and predicted visits.

[0107] The time series data is input into the Hawkes process module, and the event trigger intensity α and attenuation coefficient β are calculated to obtain the short-term burst access correction term.

[0108] Furthermore, the timestamp sequence of field access events is extracted from the time series data to identify clusters of burst access events. Maximum likelihood estimation is then used to fit the Hawkes process parameters, calculating the event triggering intensity α (which represents the strength of the external event's impact) and the decay coefficient β (which represents the decay rate of the historical event's impact). Based on the fitted parameters α and β, the conditional intensity function of the Hawkes process is used to calculate the additional accesses triggered by burst access events in the future time window. This additional accesses is converted into an incremental ratio relative to the base access rate, generating a short-term burst access correction term consisting of the field identifier, time point, and correction coefficient.

[0109] Specifically, the expression is,

[0110]

[0111] Where K is the burst access correction term, α is the event trigger intensity, and t i is the occurrence time of the i-th historical event, s is the integral variable, t is the current time, Δt is the future time window, ds is the integral differential element, and i is the historical event index.

[0112] The basic access rate prediction and the short-term burst access correction term are combined to generate a comprehensive access frequency prediction.

[0113] Furthermore, the basic access rate prediction results and the short-term burst access correction items are aligned according to the field identifiers and time points, and then fused using a weighted superposition method. The basic access rate prediction provides a benchmark value, and the short-term burst access correction item provides an incremental adjustment. The weight coefficient is dynamically calculated based on the historical prediction error (for example, the basic prediction weight is 0.7, and the burst correction weight is 0.3). The fusion result is non-negatively constrained to ensure that the predicted access volume is not less than zero, and a structured output containing field identifiers, time points and comprehensive prediction values ​​is generated. The comprehensive prediction value reflects both long-term trends and short-term fluctuation characteristics.

[0114] Match the comprehensive access frequency prediction with the business tags to generate field-level configuration items.

[0115] Furthermore, the business tag attributes of the fields in the unified metadata model are parsed, and the key tag values ​​of data classification and business importance are extracted. The storage policy rules are determined according to the business tag values. For example, data classification sensitive information corresponds to encrypted storage, business importance, and the core corresponds to a high-performance storage layer. The comprehensive access frequency prediction value is mapped to specific storage parameters according to the preset interval. For example, if the predicted access volume is 100-500 times / hour, 2 read replicas are configured accordingly. The policy determined by the business tag and the parameters of the access frequency mapping are merged to generate field-level configuration items that include complete parameters such as field identifier, storage type, number of replicas, indexing strategy, etc.

[0116] Aggregate field-level configuration items to generate a storage configuration list.

[0117] Furthermore, field-level configuration items are grouped according to data source and table name, field configurations belonging to the same physical table are merged, and configuration conflict detection and resolution are performed. For example, when encrypted and non-encrypted fields appear in the same table, the encrypted configuration is prioritized. Table-level storage parameters are aggregated, and the maximum number of replicas is calculated as the final configuration. For example, field-level replicas 2 and 3 are aggregated into a table-level replica 3. The parsed parameters such as table name, storage type, number of replicas, encryption flag, etc. are structured and output in a Hologres-compatible JSON format to generate a storage configuration list containing a complete storage policy definition.

[0118] The storage topology construction module sends data to the Hologres execution engine through the DataWorks workflow based on the storage configuration, creates logical tablespaces and physical storage partitions, and obtains a storage topology diagram.

[0119] Based on the storage configuration, the data is sent to the Hologres execution engine through the DataWorks workflow to create logical tablespaces and physical storage partitions, and obtain a storage topology diagram.

[0120] Parse the field-level storage policy in the storage configuration list and generate a set of Hologres-compatible DDL statements.

[0121] Furthermore, it reads the JSON format configuration definition in the storage configuration list, extracts the table name, field definition and storage parameters, maps the field data type to the type supported by Hologres, and generates a complete set of DDL statements.

[0122] The Hologres execution engine is called through the DataWorks workflow to create a logical tablespace according to the DDL statement set.

[0123] Furthermore, a Hologres execution node is configured in the DataWorks workflow. The DDL statement set is sorted by table creation dependency to form a directed acyclic graph. A session with the Hologres execution engine is established through a JDBC connection pool. The transaction isolation level is set to READCOMMITTED. DDL statements are executed in batches in sequence. The metadata verification result returned by Hologres is immediately obtained after each CREATE TABLE statement is executed. Finally, the name of the successfully created table and the error information of the failed statement are recorded, and a logical tablespace creation report is generated that includes the execution status, timestamp, and error details.

[0124] According to the partitioning rules of C_storage, partitions are created in the Hologres physical storage layer, and an index structure that matches the logical tablespace is established.

[0125] Furthermore, the partition key and range values ​​defined in the C_storage partitioning rule are parsed, and the ALTER TABLE statement is executed on the Hologres physical storage layer to add partitions. The corresponding index is created according to the index strategy in the storage configuration list. The CREATE INDEX statement is executed for the frequently queried fields to verify the physical storage status of the partitions and indexes, ensure that the partition metadata is consistent with the logical tablespace definition, and ensure that the index structure matches the query pattern.

[0126] Collect metadata about logical tablespaces, partitions created at the physical storage layer, and index structures to generate a storage topology diagram.

[0127] Furthermore, the Hologres system table pg_tables is queried to obtain the logical tablespace definition, including the table name, field list, and constraints. Partition metadata is collected from pg_partitions, recording the partition key, boundary value, and physical storage location. Index information is extracted from pg_indexes, including index name, field combination, and storage parameters. The three types of metadata are modeled according to the hierarchical relationship of logical table-partition-index, converted into graph structure data containing nodes (table / partition / index) and edges (inclusion / reference relationships), and the storage topology is output in JSON format.

[0128] The performance monitoring module monitors the performance indicators of the storage topology map to generate performance reports, combines the storage topology map and the unified metadata model through the data service view layer of Hologres, and provides a secure watermarked service catalog.

[0129] Monitor the performance indicators of the storage topology diagram and generate performance reports.

[0130] Extract the monitoring target list from the storage topology map, configure Hologres' session activity status view and storage monitoring view monitoring items, and obtain the data collection flow.

[0131] Furthermore, the graph data structure nodes of the logical tablespace definition and physical partition information in the storage topology diagram are parsed, and the table names and partition names that need to be monitored are extracted to form a target list. A session activity status view is created in Hologres, and the session ID, query statement, and execution status are collected by querying the session activity status view. Then, the storage monitoring view is configured to monitor the storage usage of tablespaces and indexes from the session activity status view and the storage monitoring view. The monitoring results of the two types of views are aligned by timestamps to generate a time series data stream containing the monitoring target name, session metrics (such as the number of active sessions), and storage metrics (such as the table size in MB). The data stream is recorded in CSV format, and each record is marked with the data source view name and collection time.

[0132] Aggregate the data collection stream in time windows to generate a structured indicator set.

[0133] Furthermore, the time series records in the data collection stream are grouped according to fixed time windows (for example, 5 minutes), and the window alignment adopts a sliding window mechanism (the example value window size is 5 minutes and the sliding step size is 1 minute). In each group of time windows, statistical values ​​(maximum value, average value) are obtained for session indicators (such as the number of active sessions), and the latest value at the end of the window is taken for storage indicators (such as the table size in MB). The statistical results are combined with the monitoring target name and the start and end timestamps of the time window to generate a structured record containing the indicator name, statistical value, time range and target identifier. The integrity of the indicator value is verified to ensure that the data point coverage in each time window reaches the threshold (the example value is ≥90%), and the output is a structured indicator set that meets the quality requirements.

[0134] The structured indicator set is compared with the indicators of the storage configuration list to detect abnormal deviations and obtain the abnormal event set.

[0135] Furthermore, the performance thresholds defined in the storage configuration list are analyzed and compared item by item with the statistical values ​​recorded in the structured indicator set. The three-sigma principle is used to identify statistical anomalies. The historical mean and standard deviation of the indicator are calculated (for example, the data of the past 30 days are taken). When the current value exceeds the range of ±3 times the standard deviation of the mean, it is marked as an anomaly. Then, for events of threshold violation and statistical anomaly, the anomaly type, anomaly degree (and related configuration items) are recorded, and sorted by the time the anomaly occurred. This generates an anomaly event set containing the anomaly timestamp, target identifier, anomaly description, and the original indicator value.

[0136] Generate visual performance reports based on structured indicator sets and abnormal event sets.

[0137] Furthermore, the time series data of the structured indicator set are grouped according to the monitoring target, and a line graph is drawn to show the indicator change trend (for example, the fluctuation of the number of active sessions over time). The threshold line in the storage configuration list is superimposed as a reference, and the time point and abnormal description of the abnormal event set are marked in the graph, and the abnormal location is marked with a striking symbol (such as a red triangle); then the type distribution and frequency of abnormal events are counted, and a pie chart is generated to show the proportion of each type of abnormality. The trend chart, abnormality markers and statistical charts are integrated, and the time range description and key indicator summary are added. The output is a PDF format visual performance report containing charts, legends and explanatory text.

[0138] The storage topology map and the unified metadata model are combined through the data service view layer of Hologres to create a secure watermarked service catalog.

[0139] Create virtualized view mapping rules in Hologres, associate the physical tables in the storage topology diagram with the logical fields of the unified metadata model, and generate a view definition set.

[0140] Furthermore, the physical table structure definition of the Table node in the storage topology diagram is parsed, the table name, field name and data type are extracted, the business labels and mapping rules of the corresponding logical fields in the unified metadata model are matched, a mapping relationship between physical fields and logical fields is established, and the generated view definitions are stored by business subject classification to form a view definition set containing view names, mapping rules and creation statements.

[0141] A cross-source unified view layer is constructed according to the view definition set, and the GAN watermark is injected into the output stream of the cross-source unified view layer.

[0142] Furthermore, the CREATE VIEW statement recorded in the view definition set is executed in Hologres to create a virtual view based on the physical table but with logical field naming. An interceptor for the view output stream is then configured to call a pre-trained GAN model before the data is returned to the client. This model injects slight perturbations (e.g., random offsets within ±0.1%) into numeric fields and adds invisible characters (e.g., Unicode zero-width spaces) to text fields. The model then records a watermark injection log containing the original value, perturbation amount, and timestamp. Finally, the system verifies that the watermarked data maintains business availability, for example, the error rate of the numerical summation result is less than 0.5% (for example), and the length and encoding validity of the text field remain unchanged.

[0143] Access the protocol and watermark verification interface in the cross-source unified view layer to generate a service catalog of secure watermarks.

[0144] Furthermore, the watermark verification interface is called through the REST API, and the data sample to be verified and the corresponding timestamp range are submitted. The watermark verification interface uses the same GAN model parameters as during injection to detect whether there is watermark perturbation in the expected pattern in the data, and returns the verification result (including the probability of watermark existence and perturbation characteristics). It then parses the access protocol of the cross-source unified view layer, extracts security attributes such as protocol type (such as HTTPS), authentication method, and access frequency limit, merges the verification results with the protocol security attributes, and classifies them according to the view name to generate a JSON format security watermark service directory containing view identifier, watermark feature description, protocol configuration, and verification timestamp.

[0145] In summary, the present invention combines the semantic similarity matrix calculated by the BERT-based neural network with the symbolic logic verification of the Neo4j knowledge graph through a neural-symbolic hybrid reasoning engine, thereby realizing the automatic semantic alignment and logical consistency verification of multi-source heterogeneous data, solving the limitation of traditional methods that rely on manual rule configuration, and ultimately generating a globally consistent unified metadata model to ensure the accuracy of data fusion and business interpretability. Through the LSTM-Hawkes hybrid model, the time series prediction and event triggering mechanism are integrated to dynamically capture the long-term trend and short-term burst pattern of data access, generate an accurate storage configuration list, realize adaptive hierarchical optimization of storage resources, and avoid resource waste or performance bottlenecks of static strategies.

[0146] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An intelligent storage system that supports fusion management of multi-source heterogeneous data, characterized by: include, The intelligent data source adapter module configures an intelligent data source adapter in the DataWorks data integration module, accesses multi-source heterogeneous data through the intelligent data source adapter, and extracts semantic feature vectors and technical feature sets; The neural-symbolic hybrid reasoning module inputs the semantic feature vector and technical feature set into the neural-symbolic hybrid reasoning engine. Using the BERT-based neural network, it calculates the similarity matrix between multi-source heterogeneous data fields. It then imports the technical feature set into the domain knowledge graph built by Neo4j for symbolic logic verification and generates a unified metadata model. The dynamic storage optimization module annotates business tags in a unified metadata model, uses an LSTM-Hawkes hybrid model to calculate the expected access frequency of multi-source heterogeneous data fields, and generates a storage configuration list. The storage topology construction module sends the data to the Hologres execution engine through the DataWorks workflow based on the storage configuration, creates logical tablespaces and physical storage partitions, and obtains the storage topology map. The performance monitoring module monitors the performance indicators of the storage topology map to generate performance reports, combines the storage topology map and the unified metadata model through the data service view layer of Hologres, and provides a secure watermarked service catalog.

2. The intelligent storage system supporting multi-source heterogeneous data fusion management according to claim 1, characterized in that: Configure the intelligent data source adapter in the DataWorks data integration module. The following steps are included: Create a data integration task in the DataWorks console and select Intelligent Data Source Adapter as the data source type. Enable the intelligent protocol identification function and configure the intelligent data source adapter.

3. The intelligent storage system supporting multi-source heterogeneous data fusion management according to claim 2, characterized in that: Access multi-source heterogeneous data through intelligent data source adapters to extract semantic feature vectors and technical feature sets, including the following steps: Automatically scan multi-source heterogeneous data configuration information sets through intelligent data source adapters, identify data protocols and output protocol identifiers; Load the data parser according to the protocol identifier and convert the multi-source heterogeneous data into a structured record set; Perform field-level semantic analysis on structured record sets to extract semantic feature vectors and technical feature sets.

4. The intelligent storage system supporting multi-source heterogeneous data fusion management according to claim 3, characterized in that: The semantic feature vector and technical feature set are input into the neural symbolic hybrid reasoning engine, and the similarity matrix between multi-source heterogeneous data fields is calculated through the BERT-based neural network, including the following steps: The semantic feature vector is input into the BERT-based neural network and the enhanced semantic representation is generated through multi-layer Transformer encoding; Normalize the enhanced semantic representation to obtain a unit vector, and calculate the cosine similarity matrix of the unit vector; The technical feature set is imported into the similarity correction module, and the conflicting items in the cosine similarity matrix are adjusted through the rule engine to obtain the similarity matrix.

5. The intelligent storage system supporting multi-source heterogeneous data fusion management according to claim 4, characterized in that: Import the technical feature set into the domain knowledge graph built by Neo4j for symbolic logic verification and generate a unified metadata model, including the following steps: Convert the technical feature set into Neo4j Cypher statements to build the initial knowledge graph; Execute predefined symbolic reasoning rules on the initial knowledge graph, mark logical conflicts and generate conflict reports; Correct the conflicting nodes and relationships in the initial knowledge graph based on the conflict report to obtain a conflict-free knowledge graph; Extract globally consistent field mapping rules from the conflict-free knowledge graph to generate a unified metadata model.

6. The intelligent storage system supporting multi-source heterogeneous data fusion management according to claim 5, characterized in that: Marking business tags in the unified metadata model includes the following steps: Extract business semantic descriptions and associations of multi-source heterogeneous data fields from the unified metadata model to generate an original tag set; Perform standardized matching on the original tag set through business classification rules to obtain a standardized tag candidate set; Use the domain knowledge graph to verify the normalized label candidate set and generate a valid label set. Bring the valid tag set into the extended attribute layer of the unified metadata model to form business tags.

7. The intelligent storage system supporting multi-source heterogeneous data fusion management according to claim 6, characterized in that: The LSTM-Hawkes hybrid model is used to calculate the expected access frequency of multi-source heterogeneous data fields and generate a storage configuration list, including the following steps: Time series data collected from field-level operation records in DataWorks task logs and Hologre; Input the time series data into the LSTM module to obtain the basic visit rate prediction; Input the time series data into the Hawkes process module, calculate the event trigger intensity α and the attenuation coefficient β, and obtain the short-term burst access correction term; The basic access rate forecast and the short-term burst access correction term are combined to generate a comprehensive access frequency forecast; Match comprehensive access frequency predictions with business tags to generate field-level configuration items; Aggregate field-level configuration items to generate a storage configuration list.

8. The intelligent storage system supporting multi-source heterogeneous data fusion management according to claim 7, characterized in that: According to the storage configuration, the data is sent to the Hologres execution engine through the DataWorks workflow to create logical tablespaces and physical storage partitions, and obtain the storage topology diagram, including the following steps: Parse the field-level storage policy in the storage configuration list and generate a set of Hologres-compatible DDL statements; The Hologres execution engine is called through the DataWorks workflow to create a logical tablespace based on the DDL statement set. Create partitions in the Hologres physical storage layer according to the C_storage partitioning rules and build an index structure that matches the logical tablespace. Collect metadata about logical tablespaces, partitions created at the physical storage layer, and index structures to generate a storage topology diagram.

9. The intelligent storage system supporting multi-source heterogeneous data fusion management according to claim 8, characterized in that: Monitoring the performance indicators of the storage topology to generate a performance report includes the following steps: Extract the monitoring target list from the storage topology map, configure the Hologres session activity status view and storage monitoring view monitoring items, and obtain the data collection flow; Aggregate the data collection stream into time windows to generate a structured indicator set; Compare the structured indicator set with the indicators of the storage configuration list to detect abnormal deviations and obtain the abnormal event set; Generate visual performance reports based on structured indicator sets and abnormal event sets.

10. The intelligent storage system supporting multi-source heterogeneous data fusion management according to claim 9, characterized in that: Combining the storage topology map and the unified metadata model through the data service view layer of Hologres and the secure watermark service catalog includes the following steps. Create virtualized view mapping rules in Hologres, associate the physical tables in the storage topology diagram with the logical fields of the unified metadata model, and generate a view definition set; Construct a cross-source unified view layer according to the view definition set, and inject the GAN watermark into the output stream of the cross-source unified view layer; Access the protocol and watermark verification interface in the cross-source unified view layer to generate a service catalog of secure watermarks.

Images