Data weaving method for integration and treatment of multi-source heterogeneous data

Abstract

The invention provides a multi-source heterogeneous data integration and governance-oriented data weaving method, which comprises the following steps of: performing data acquisition from an accessed multi-source heterogeneous data source to generate an original multi-source heterogeneous data stream; performing standardization processing on the original multi-source heterogeneous data stream to generate a standardized multi-source heterogeneous data set; performing active content scanning processing on the standardized multi-source heterogeneous data set to determine business metadata, and performing consanguinity tracking processing on the business metadata to generate enhanced business metadata; calling a domain ontology framework to carry out standardized constraint on the enhanced service metadata to obtain standardized service metadata without cross-data source semantic ambiguity, and carrying out implicit association mining processing on the standardized service metadata based on a graph neural network to generate a semantic knowledge graph containing core entities and relationships; and performing logic abstraction processing on the distributed data resources according to the semantic knowledge graph to generate a unified data access interface.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data management, in particular to a data weaving method for multi-source heterogeneous data integration and governance. BACKGROUND

[0002] In the process of enterprise digital transformation, the integration and governance of multi-source heterogeneous data have become the core demand to support business analysis and decision optimization. Currently, enterprises usually deploy ERP systems, CRM systems, Internet of Things sensing platforms and other types of business systems, which generate different types of data sources such as relational database data, unstructured document data and time series sensing data, forming a multi-source heterogeneous data environment; in order to fully tap the value of these data, the data scattered in various systems need to be effectively integrated, while the semantic consistency and accessibility of the data are guaranteed to meet the needs of business end for unified data query, analysis and application.

[0003] The implementation idea of the existing multi-source heterogeneous data integration and governance scheme is as follows: first, the original data is extracted from each multi-source heterogeneous data source through an ETL tool, and the original data in different formats is converted into intermediate data in a preset format; then, technical personnel manually define the field correspondence between the intermediate data and the target data model based on business experience to complete the integration processing of the data; finally, the integrated data is stored in a centralized data warehouse, and the business system obtains the data by developing a customized access interface for the data warehouse.

[0004] In the above existing scheme, since the field mapping relationship is manually defined, it is not only impossible to dynamically record the flow trajectory of the data from the original data source to the data warehouse, making it difficult to track the source and processing process of the data during data tracing, but also different business systems need to develop exclusive customized interfaces for the data warehouse, which cannot form a unified data access portal, increasing the system development and maintenance cost; at the same time, the semantic differences between the data sources are not eliminated through a systematic constraint mechanism, and the same business concept may be expressed inconsistently in different data sources, which may cause business personnel to have a biased understanding of the data, affecting the accuracy of data application. SUMMARY

[0005] To solve the above technical problems, the present application provides a data weaving method for multi-source heterogeneous data integration and governance to at least alleviate the above technical problems.

[0006] The technical scheme provided by the embodiments of the present application is as follows: A data weaving method for multi-source heterogeneous data integration and governance, comprising: Step 1, calling a multi-source data acquisition component to collect data from the accessed multi-source heterogeneous data sources to generate an original multi-source heterogeneous data stream; Step 2, call the multi-source data standardization component to standardize the original multi-source heterogeneous data stream to generate a standardized multi-source heterogeneous data set; Step 3, call the data enhancement component to actively scan the standardized multi-source heterogeneous data set to determine business metadata, and perform bloodline tracking processing on the business metadata to generate enhanced business metadata; Step 4, call the graph generation component, call the domain ontology framework to normalize the enhanced business metadata to eliminate semantic ambiguity across data sources, and perform implicit association mining on the normalized business metadata based on graph neural networks to generate a semantic knowledge graph containing core entities and relationships. Step 5, call the interface generation component to perform logical abstraction processing on distributed data resources based on the semantic knowledge graph to generate a unified data access interface.

[0007] The technical solution in this application has the following technical benefits: 1. Relieve the problem of "relying on manual definition of field mapping, unable to dynamically record data flow track, difficult data traceability" The traditional scheme in the background art relies on technical personnel to manually define the field correspondence between intermediate data and target models, and has no mechanism to dynamically record data flow, making it difficult to track the source and processing process when data traceability is performed. The present scheme, through step 3 "calling the data enhancement component to actively scan the standardized multi-source heterogeneous data set to determine business metadata, and performing bloodline tracking processing on the business metadata to generate enhanced business metadata", differs from the traditional manual recording method: the componentized bloodline tracking can dynamically follow the data from collection to standardization The flow process will associate the data with the data source and the processing node in the enhanced business metadata, without the need for manual recording; this processing allows the flow track to be directly queried from the enhanced business metadata when data traceability is performed, which is significantly lower than the traditional manual method of no traceable, and is more suitable for the scene needs of frequent data flow of enterprise multi-source data.

[0008] 2. Relieve the problem of "need to develop customized interfaces, no unified data access entry, high system development and maintenance cost" The traditional scheme in the background art needs to develop a customized interface for the data warehouse for each business system, and the interface types are scattered and difficult to maintain. The scheme is different from the traditional customized interface design: the semantic knowledge graph has integrated the core entities and relationships of multi-source data, and logical abstraction based on it can generate a unified interface that adapts to multiple business systems, without the need to develop a single system separately; compared with the traditional multi-interface parallel maintenance method, this processing reduces the workload of interface development, and the unified interface reduces the complexity of subsequent maintenance, which is more suitable for the scene needs of enterprise multi-business system sharing multi-source data.

[0009] 3. Alleviate the problem of "no systematic semantic constraint across data sources, inconsistent expression of the same business concept, affecting data understanding" The traditional scheme in the background art has no systematic semantic constraint mechanism, and relies on manual calibration of semantic differences across data sources, making consistency difficult to guarantee. The scheme is different from the traditional manual calibration method: the domain ontology framework provides a unified semantic standard (such as unified entity naming and relationship definition) for business metadata across data sources, which can systematically eliminate the problem of "different expressions of the same business concept in different data sources"; at the same time, the semantic knowledge graph generated based on the graph neural network to mine implicit associations further strengthens the semantic association of entities and relationships, which improves the consistency of multi-source data semantics compared with the traditional manual fragmented calibration method, helping business personnel to more clearly understand the logical relationship between multi-source data, and adapting to the scene needs of enterprise cross-system data semantic unification. BRIEF DESCRIPTION OF DRAWINGS

[0010] Figure 1 A flowchart of a data weaving method for multi-source heterogeneous data integration and governance according to an embodiment of the present application.

[0011] Figure 2 A structural diagram of a data weaving device for multi-source heterogeneous data integration and governance according to an embodiment of the present application. DETAILED DESCRIPTION

[0012] Figure 1 A flowchart of a data weaving method for multi-source heterogeneous data integration and governance according to an embodiment of the present application. As shown in Figure 1As shown, it includes: step 1, calling a multi-source data acquisition component to collect data from accessed multi-source heterogeneous data sources to generate an original multi-source heterogeneous data stream; step 2, calling a multi-source data standardization component to standardize the original multi-source heterogeneous data stream to generate a standardized multi-source heterogeneous data set; step 3, calling a data enhancement component to actively scan and process the standardized multi-source heterogeneous data set to determine business metadata and perform bloodline tracking processing on the business metadata to generate enhanced business metadata; step 4, calling a graph generation component, calling a domain ontology framework to normalize the enhanced business metadata to obtain normalized business metadata that eliminates semantic ambiguity across data sources, and based on a graph neural network, to mine implicit correlations of the normalized business metadata to generate a semantic knowledge graph containing core entities and relationships; and step 5, calling an interface generation component to perform logical abstraction processing on distributed data resources based on the semantic knowledge graph to generate a unified data access interface.

[0013] Optionally, the step 1 specifically includes the following steps: Step 11: calling a dynamic data source compatibility module of the multi-source data acquisition component, loading a preset data source configuration template, and adapting to the type of the accessed multi-source heterogeneous data source, wherein the multi-source heterogeneous data source includes a structured database, a semi-structured mark file, and an unstructured media stream; Step 12: selecting a collection mode based on the type of the data source, using incremental snapshot collection for the structured database to obtain structured data, using batch parsing collection for the semi-structured mark file to obtain semi-structured data, using real-time stream collection for the unstructured media stream to obtain unstructured data, and performing alignment processing on the structured data, the semi-structured data, and the unstructured data to generate initial collection data; Step 13: calling a built-in running state monitoring module of the component to perform integrity verification and exception detection on the initial collection data, to eliminate invalid data with missing fields or format errors, to retain valid data, and to integrate the valid data into the original multi-source heterogeneous data stream.

[0014] Optionally, the step 11 specifically includes the following steps: Step 111: the dynamic data source compatibility module of the multi-source data acquisition component reads data source access protocol information and matches it with a preset interface adaptation library to generate a dedicated access link for different target data sources; Step 112: based on the dedicated access link, loading a corresponding data source configuration template, wherein the data source configuration template contains field identification rules, data reading frequency, and data sharding strategies.

[0015] Preferably, the specific implementation process of step 111 is as follows: first, the dynamic data source compatibility module of the multi-source data collection component is called to initiate a protocol detection request for the accessed multi-source heterogeneous data sources (such as structured databases, semi-structured mark files, and unstructured media streams), and read the data source access protocol information (including protocol types such as JDBC protocol for structured databases, HTTP / HTTPS protocol for semi-structured files, and RTSP protocol for unstructured media streams, and protocol parameters such as port number, authentication method, and data transmission format) returned by each data source; then, the read data source access protocol information is matched with the interface adaptation library preset in the dynamic data source compatibility module, the interface adaptation library is stored in a three-dimensional structure of “protocol type-adaptation rule-link parameter” (the row represents the protocol type, such as JDBC, HTTP, and RTSP; the column represents the adaptation dimension, such as connection timeout threshold, data packet size, and authentication encryption method; and the element at the intersection of the row and the column represents the standard parameter value of the corresponding protocol type in the adaptation dimension, such as 30 seconds for the connection timeout threshold of the JDBC protocol and 1024 KB for the data packet size of the RTSP protocol), the protocol type in the data source access protocol information is compared with the row identifier of the interface adaptation library to determine the corresponding adaptation rule and link parameter; and finally, based on the determined adaptation rule and link parameter, an exclusive access link containing “protocol matching identifier- exclusive connection parameter-data transmission verification rule” is generated for each data source to ensure that data sources of different protocols can be stably accessed to the collection component through the corresponding exclusive access link. This technology is different from the traditional fixed link design. The traditional design only presets a fixed link for a single protocol, and cannot adapt to the different protocol requirements of multi-source heterogeneous data sources. However, the present application realizes dynamic adaptation of multi-protocol data sources through the process of “protocol detection-three-dimensional library matching-exclusive link generation”, solves the problem of low adaptability of the traditional design, and is more suitable for the scene of parallel access of enterprise multi-source heterogeneous data sources.

[0016] Preferably, in the multi-source heterogeneous data source high concurrency access scene, step 112 is specifically implemented as follows: first, link attribute analysis is performed on the exclusive access link generated in step 111 to extract the “protocol matching identifier” (such as “JDBC-001” and “RTSP-002”) in the link, which directly corresponds to the type of data source (such as “JDBC-001” corresponding to a structured database and “RTSP-002” corresponding to an unstructured media stream); then, based on the parsed protocol matching identifier, the corresponding data source configuration template is loaded from the template library of the dynamic data source compatibility module. The templates for different types of data sources contain differentiated core configuration items: for the template for a structured database, the field identification rule is “parsing the primary key field and the associated field according to the table structure metadata”, the data reading frequency is “set according to the transaction log update frequency (such as once every 5 minutes)”, and the data sharding strategy is “sharding according to the primary key range (such as ID 1-10000 for one shard)”; for the template for a semi-structured mark file, the field identification rule is “parsing the key field according to the XML / JSON tag level”, the data reading frequency is “triggered according to the file modification time (such as reading within 1 minute after the file is updated)”, and the data sharding strategy is “sharding according to the file size (such as 500 MB for one shard)”; for the template for an unstructured media stream, the field identification rule is “parsing the frame identification field according to the media encoding format”, the data reading frequency is “real-time stream pushing (no fixed interval)”, and the data sharding strategy is “sharding according to the time slicing (such as 10 seconds for one shard)”; by loading the corresponding data source configuration template, adaptive configuration basis is provided for subsequent collection and processing of different types of data sources, so as to avoid problems such as low collection efficiency or data loss caused by unmatched configuration. This technology is different from the traditional general template design. The traditional template uses unified configuration items, which cannot meet the collection needs of different types of data sources (such as real-time stream reading in real time, while the fixed frequency reading of the traditional template will cause stream data accumulation), and the application binds the “link identifier-differentiated template” to ensure that the configuration items are highly adaptive to the type of data source, thereby improving the stability and efficiency of multi-source data collection.

[0017] Optionally, the step 12 specifically comprises the following steps: Step 121, when using incremental snapshot collection for a structured database, calling the change capture submodule of the multi-source data collection component, identifying the change operation records since the last collection based on the database transaction log, and assigning a change marker type and a change timestamp to each change record to generate a structured incremental data set containing the change track; Step 122, when using batch analysis collection for a semi-structured mark file, calling the nested structure analysis submodule of the multi-source data collection component to automatically identify the nested levels in the semi-structured mark file and expand the nested structure according to the “flattening rule” to generate a semi-structured analysis data set; Step 123, when real-time stream collection is adopted for the unstructured media stream, the stream data encapsulation submodule of the multi-source data collection component is called to bind a collection timestamp, a collection device unique identifier and a stream transmission status code for each piece of media data, so as to generate an unstructured time-series data set; Step 124, the structured incremental data set, the semi-structured parsed data set and the unstructured time-series data set are aligned and encapsulated into a triple format, so as to generate initial collection data.

[0018] Preferably, the specific implementation process of step 121 is as follows: the change capture submodule of the multi-source data collection component first accesses the transaction log of the structured database (such as the binlog of MySQL or the redo log of Oracle), extracts the SQL operation instructions (including INSERT, UPDATE and DELETE) and the corresponding data row identifiers in the log through a log parser; then, the change capture submodule calls a timestamp comparison unit to compare the execution time of each operation instruction with the last collection termination timestamp, filters out the change operation records with a timestamp within a threshold range, and forms an original change data set; then, each record in the original change data set is assigned a change marker type (insert is represented by "INS", update is represented by "UPD" and delete is represented by "DEL"), and an accurate timestamp (accurate to millisecond level) at the time of operation is bound; finally, the marked change record is associated with the corresponding data row snapshot, and a structured incremental data set containing "data row identifier-change type-change timestamp-change before and after value" is generated. This technique is different from the traditional full snapshot collection, and the traditional method needs to read the data of the whole table regularly, which is easy to cause resource waste and data redundancy. However, the application realizes incremental capture through transaction log analysis, only records change data and retains complete change track, and is more suitable for efficient data collection in the high-frequency update scenario of enterprise core business library.

[0019] Preferably, in the semi-structured data (such as multi-layer nested XML / JSON file) acquisition scenario, step 122 is specifically implemented as follows: the nested structure parsing submodule first reads the root node information of the semi-structured marked file, traverses the entire file structure through a label level detector, and generates a nested structure tree with "node path-level depth-subnode number" as the dimension (for example, the "user.info.address" in the JSON file corresponds to a level depth of 3); then, a flattening rule engine is called, which includes preset "level splicing rules" (splicing multi-level labels with separators, such as "address.city") and "array expansion rules" (distinguishing array type nodes by index suffix, such as "order[0].id" converted to "order_0_id"); then, the nested structure tree is expanded layer by layer according to the above rules, and multi-layer nested fields are converted into flat field names while retaining the original field values; finally, the flattened fields after expansion are combined (the same actual field is repeatedly named due to path parsing), and a semi-structured parsed dataset with unique field names and flat structure is generated. This technology is different from the traditional fixed path parsing, which requires manual presetting of parsing paths and cannot adapt to dynamically changing nested structures. However, the present application can adapt to semi-structured files with different nesting depths and structures through dynamic level detection and rule-based expansion, improving the flexibility of parsing.

[0020] Preferably, in the specific technical implementation of step 123: the stream data packaging submodule first accesses unstructured media streams (such as monitoring video streams, industrial sensor waveform streams) through the RTSP / RTP protocol, starts a stream slicer to slice the continuous media stream according to a preset time length (such as 10 seconds per slice), and generates independent media data segments; then, three types of metadata are bound for each media data segment: acquisition timestamp (UTC time when the media data is generated), acquisition device unique identifier (such as device MAC address hash value), and stream transmission status code (0-5 representing transmission quality, 0 representing no packet loss, and 5 representing severe packet loss); then, a media data verification unit is called to verify the integrity of the media data segment with bound metadata (such as video frame continuity detection, audio sampling rate consistency verification); finally, the media data segment that passes the verification is combined with its metadata into an encapsulation structure of "metadata header + media data body", and an unstructured time series dataset is generated in chronological order. This technology is different from the traditional unmarked stream acquisition, which only stores the original media data and cannot trace the data source and transmission quality. However, the present application realizes the time series management of unstructured data through metadata binding, providing a basis for subsequent quality evaluation and traceability, and is more suitable for scenarios such as industrial Internet of Things that require high time series requirements.

[0021] Preferably, the specific implementation process of step 124 is as follows: the data alignment processing unit first extracts the timestamp field of the structured incremental data set, the semi-structured parsed data set and the unstructured time series data set, establishes a global time axis based on the timestamp (the precision is unified to millisecond level), then performs slicing aggregation on the three types of data sets according to the time axis, groups different types of data in the same time slice (such as 1 minute unit) into a group, and forms a time-aligned mixed data group; then, a triple encapsulator is called to convert each mixed data group into a triple format of "entity identifier-attribute key-attribute value": for structured data, directly map the table primary key as the entity identifier, the field name as the attribute key, and the field value as the attribute value; for semi-structured data, split and combine the flat field name as the attribute key (such as "user_name" corresponds to "user:name"); for unstructured data, take the device identifier as the entity identifier, the media type as the attribute key, and the media data segment storage path as the attribute value; finally, perform format checking (ensure that the entity identifier is unique and the attribute key format is uniform) on all triples, and integrate the triples that pass the checking to form the initial collection data. This technology is different from the traditional type storage, which independently stores different types of data and is difficult to establish time correlation. However, the present application realizes the correlation and integration of multi-source heterogeneous data through time alignment and unified triple format, and lays a foundation for subsequent standardized processing.

[0022] Optionally, the step 13 specifically includes the following steps: Step 131, calling the multi-dimensional integrity checking unit of the running state monitoring module to perform field integrity checking, record integrity checking and association integrity checking on the initial collection data to determine the default data therein, and performing completion on the completable data therein, and marking the non-completable data as invalid data to generate intermediate data set that passes the integrity checking; Step 132, starting the anomaly detection submodule of the running state monitoring module to perform anomaly detection on the intermediate data set that passes the integrity checking to determine the abnormal data therein; Step 133, for the detected abnormal data, if it is a repairable abnormality, repair it, if it is an unrepairable abnormality, mark it as invalid data and remove it, to form valid data, and further store and archive the valid data according to "data source type-collection time" to integrate into the original multi-source heterogeneous data stream.

[0023] Preferably, the specific implementation process of step 131 is as follows: the multi-dimensional integrity verification unit of the running state monitoring module first loads the data source configuration template corresponding to the initial collection data, extracts the list of mandatory fields defined in the template (such as the primary key field in structured data, the core tag in semi-structured data); then, perform field integrity verification, check line by line whether the fields of the initial collection data contain all items in the list of mandatory fields, and mark the records missing mandatory fields as default data; then, perform record integrity verification, detect whether the records exist by hash value comparison (such as incomplete lines caused by file transmission interruption), mark the records with unmatched hash values as default data; then, perform association integrity verification, check the reference relationship across records based on the preset entity association rule (such as "order record must be associated with customer ID"), mark the records with invalid references as default data; then, call the completion engine for the marked default data, perform completion for the default data that can be interpolated by historical data (such as missing values of time series data) or derived by associated data (such as completing customer ID by customer name), and mark the default data that cannot be completed (such as missing primary key and no derivation basis) as invalid data; finally, integrate the records that pass the verification and completion to generate an intermediate data set that passes the integrity verification. This technology is different from traditional single-dimensional verification, which only checks whether the field exists, ignoring record integrity and association integrity. The three-dimensional collaborative verification of the present application is more suitable for the integrity requirements of multi-source data association analysis scenarios.

[0024] Preferably, in the multi-type data mixed verification scene, when step 132 is specifically implemented: the anomaly detection submodule first shards the intermediate data set that passes the integrity verification according to data types (structured, semi-structured, unstructured); for structured data shards, call statistical anomaly detection rules (such as Z-score threshold range for numerical fields, time reasonableness verification for date fields), mark records that exceed the threshold range as anomaly data; for semi-structured data shards, start the syntax rule engine, check the closeness of the marked syntax (such as whether XML tags are paired), attribute value format (such as regular matching of mobile phone number, email), mark records with syntax errors or format inconsistencies as anomaly data; for unstructured data shards, call media feature verifier, detect media data encoding consistency (such as frame rate mutation of video stream), data integrity (such as audio file header check code), mark media segments with feature anomalies as anomaly data; then, aggregate anomaly data from each shard to generate an anomaly data list containing "data ID- anomaly type- anomaly feature value". This technology is different from traditional general anomaly detection, which uses a unified threshold to detect all types of data, which is easy to cause misjudgment of unstructured data. The present application improves the pertinence and accuracy of anomaly recognition through type-based sharding and exclusive detection rules.

[0025] Preferably, in the specific technical implementation of step 133: the abnormal data processing unit first reads the abnormal data list, calls the abnormal type parser for each abnormal data, and judges whether the abnormal type belongs to the repairable category (such as format error date, out of range but normalizable value); for repairable abnormal data, start the repair engine: for format class exceptions (such as "2023 / 13 / 01" invalid date), convert to standard format (such as "2024 / 01 / 01") through date parsing rules; for value class exceptions (such as sensor readings beyond reasonable range), normalize based on the distribution characteristics of similar data; for unrepairable abnormal data (such as damaged media files, unresolvable encrypted fields), directly mark as invalid data and record the reason for rejection; then, integrate the repaired valid data and the non-abnormal data that pass the original integrity check to form a unified valid data set; finally, store and archive the valid data in a hierarchical structure according to "data source type-collection time" (such as "structured database / 20240903 / xxx.csv"), and generate an index file containing the storage path, data volume, and check code, and finally integrate the original multi-source heterogeneous data stream. This technology is different from the traditional single processing. The traditional method only rejects abnormal data, wasting repairable data resources. The present application improves resource utilization while ensuring data quality, and is more suitable for the fine needs of enterprise-level data governance.

[0026] Optionally, the step 2 specifically comprises the following steps: Step 21: calling the multi-format parsing engine of the multi-source data standardization component to identify the format of the original multi-source heterogeneous data stream, automatically parsing the table structure of structured data, the tag syntax of semi-structured data, and the media encoding format of unstructured data to obtain the parsing result; Step 22: based on the constructed business semantic dictionary, performing semantic enhancement on the parsing result to obtain semantic enhanced parsing data; Step 23: calling the semantic driven standard schema mapping engine to map the semantic enhanced parsing data to the preset standard data model, generating a standardized multi-source heterogeneous data set with unified field format and consistent semantics.

[0027] Optionally, the step 22 specifically comprises the following steps: Step 221: calling the dynamic semantic matching sub-module of the business semantic dictionary, adding semantic description tags to the core business fields in the parsing result to generate field-level semantic annotation data; Step 222: calling the semantic enhancement sub-module, performing business correlation feature enhancement processing on the field-level semantic annotation data to generate feature enhanced semantic data; Step 223: calling a cross-type semantic consistency calibration submodule, performing disambiguation processing on the feature-enhanced semantic data based on the context semantic rules of the semantic dictionary to generate semantic-enhanced parsed data.

[0028] Preferably, the specific implementation process of step 221 is as follows: the dynamic semantic matching submodule of the business semantic dictionary first loads the parsed results output by the multi-source data standardization component, extracts the core business fields (such as “order ID” and “user ID” in structured data, “product code” label in semi-structured data, and “device model” metadata associated with unstructured data) in the parsed results through the field recognition engine; then, calling the semantic dictionary retrieval unit, the extracted core business fields are semantically matched with the standard field library in the business semantic dictionary, which is stored in a four-dimensional structure of “field name-business meaning-field of origin-associated field” (for example, “order ID” corresponds to the business meaning of “unique identification code of order”, the field of origin is “transaction”, and the associated fields are “user ID” and “payment amount”); then, according to the matching result, an exclusive semantic description label (the label format is “business meaning|field of origin”, such as “order ID” adding “unique identification code of order|transaction” label) is added for each core business field; finally, the core business field after adding the label is integrated with other fields in the parsed results to generate field-level semantic annotation data. This technology is different from the traditional fixed label annotation, and the traditional method adds a uniform label to all fields of the same type, which cannot adapt to the differences in business scenarios of fields in multi-source data (such as different meanings of “code” in order data and device data), while the present application realizes the customization of field labels in scenarios through dynamic semantic matching, improving the fit of semantic annotation.

[0029] Preferably, in cross-data source business association scenarios (such as the association between e-commerce orders and logistics data), step 222 is specifically implemented as follows: The semantic enhancement submodule first reads the field-level semantic annotation data generated in step 221, calls the association feature extraction engine, and extracts cross-field business association features from the annotation data based on the association rules in the business semantic dictionary (such as "order ID needs to be associated with logistics tracking number" and "user ID needs to be associated with delivery address"). (e.g., "order ID=O20240901001" is associated with "logistic tracking number=SF123456789" and "user ID=U1001" is associated with "delivery address=XX province XX city"). Then, the extracted business association features are structurally encapsulated to generate association feature entries of "core field - associated field - association rule". Then, the association feature entries are embedded in the corresponding field-level semantic annotation data so that the annotation data not only contains the semantic labels of a single field, but also carries cross-field business association information. Finally, the format of the annotation data with embedded association features is standardized (to ensure that the correspondence between association information and field data is clear) to generate feature-enhanced semantic data. This technology differs from traditional single-field semantic enhancement. Traditional methods only strengthen the semantic information of a single field and ignore the business relationships between fields. However, this application embeds related features to enable data to have semantic relationships at the business link level, which is more suitable for the scenario requirements of joint analysis of multi-source data.

[0030] Preferably, in the specific technical implementation of step 223: the cross-type semantic consistency calibration submodule first loads the feature-enhanced semantic data generated in step 222, shards by data type (structured, semi-structured, unstructured), and starts the corresponding semantic disambiguation rules for different types of data; for structured data shards, call the field ambiguity detection rule (such as detecting whether the unit of the "amount" field is uniform, distinguishing the semantic difference between "yuan" and "ten thousand yuan"), and perform semantic calibration on the fields with ambiguity according to the standard unit (such as "yuan") in the semantic dictionary; for semi-structured data shards, start the label ambiguity resolution engine to handle labels with the same name but different meanings (such as "status" in order data as "payment status" and in logistics data as "transportation status"), and eliminate ambiguity by adding domain suffixes (such as "status_payment" and "status_transportation"); for unstructured data shards, call the media semantic association rule to check the semantic consistency of the media data metadata (such as whether the "device model" metadata is consistent with the device model identified in the media content), and correct the inconsistent items according to the standard semantic dictionary; then, aggregate the disambiguation results of each shard and check the semantic consistency across shards (such as whether the semantic description of "order ID" in order data and logistics data is uniform); finally, integrate the disambiguation data that passes the consistency check to generate semantic-enhanced parsed data. This technology is different from traditional single-type disambiguation, which only handles semantic ambiguity of the same type of data and ignores semantic consistency across types. However, this application ensures semantic consistency across multiple sources of heterogeneous data at the global level through dual processing of type-specific disambiguation and cross-type calibration, avoiding semantic confusion in subsequent standardization processing.

[0031] Optionally, the step 23 specifically comprises the following steps: Step 231: calling the semantic disassembly submodule in the standard schema mapping engine to perform standard data model core field disassembly on the semantic-enhanced parsed data, and to perform semantic association matching between the parsed data and the standard field semantics to generate a semantic-enhanced field-standard model field mapping table; Step 232: calling the encoding submodule in the standard schema mapping engine to convert the semantic-enhanced field-standard model field mapping table to a semantic vector through semantic description vector conversion, and to calculate the similarity between the corresponding semantic vectors of the associated fields to generate a field semantic matching vector; Step 233: calling the data integration submodule in the standard schema mapping engine to automatically convert the semantic-enhanced parsed data to semantic vector adaptive format data based on the field semantic matching vector according to the field format in the standard model, and to perform business anchoring processing on the semantic vector adaptive format data according to the semantic business anchoring rules in the standard model, to generate a standardized multi-source heterogeneous data set with uniform field format and consistent semantics.

[0032] Preferably, the specific implementation process of step 231 is as follows: the semantic disassembly submodule of the standard schema mapping engine first loads the semantic enhanced parsed data and the preset standard data model, extracts the core field list of the standard data model (such as the mandatory fields “user ID”, “registration time”, “area of belonging” and the optional fields “gender”, “age” in the “user basic information” model) through the model parser; then, field-level semantic disassembly is performed on the semantic enhanced parsed data, and the basic attributes (name, data type) and semantic enhancement information (semantic label, associated feature) of each field are separated; then, the semantic association matching engine is started, the semantic of the disassembled parsed data field is compared with the preset semantic description (such as the preset semantic of “user ID” is “string code for uniquely identifying a user”) of the core field of the standard model in a cross-dimension manner, and a preliminary matching result is generated; then, the preliminary matching result is filtered under multiple conditions (such as excluding matching items with incompatible data types), and finally a semantic enhanced field-standard model field mapping table containing “parsed data field name-standard model field name-semantic matching degree-association feature basis” is formed. This technology is different from the traditional direct mapping based on field name. The traditional method ignores the semantic difference of the fields (such as the “code” fields with the same name but different meanings), while the present application improves the accuracy of cross-data source field mapping through semantic disassembly and association matching, and is more suitable for the fusion scene of multi-source heterogeneous data.

[0033] Preferably, in the cross-field data semantic matching scenario (such as medical data and social security data fusion), step 232 is specifically implemented as follows: the coding submodule first reads the semantic enhanced field-standard model field mapping table, calls the semantic vector generator, and converts each "parsed data field semantic description" and "standard model field semantic description" in the table into a semantic vector - each dimension of the vector corresponds to a basic semantic feature (such as "identification type", "data format", "business field", etc.), and the dimension value is the feature consistency between 0 and 1 (for example, the "identification type" dimension value of "user ID" is 0.95, indicating that it is highly consistent with the "unique identification" feature); then, the cosine similarity algorithm is used to calculate the similarity of the semantic vectors corresponding to each pair of associated fields (the range is 0-1, and the higher the value, the higher the semantic fit degree); then, the calculated similarity value is weighted and fused with the "semantic matching degree" in the mapping table (the weight is dynamically adjusted according to the business scenario, such as higher similarity weight of core fields), to generate the final field semantic matching vector - the vector is a two-dimensional matrix, the rows represent the parsed data fields, the columns represent the standard model fields, and the matrix elements are the fusion matching degrees of the corresponding field pairs; finally, the matching vector is normalized (to ensure that all element values are in the range of 0-1), and the standardized field semantic matching vector is output. This technology is different from the traditional single-dimension matching degree calculation, and the traditional method only depends on the field name similarity, while the present application constructs and weights the multi-dimensional semantic vector, which more accurately quantifies the semantic association degree of cross-field.

[0034] Preferably, in the specific implementation of step 233: the data integration sub-module first loads the field semantic matching vector and the semantic enhanced parsed data, sets a matching degree threshold (such as 0.7), and screens out field pairs with element values higher than the threshold in the vector as effective mappings; for the parsed data fields corresponding to the effective mapping pairs, the format conversion engine is called to automatically convert according to the format requirements of the standard model fields (such as uniformly converting date fields into "YYYY-MM-DD" format and uniformly retaining two decimal places for numerical value fields), to generate semantic vector adaptive format data; then, the business anchoring processor is started, cross-field association verification is performed on the semantic vector adaptive format data according to the semantic business anchoring rules in the standard model (such as "order amount" must be associated with the "currency" field and "diagnosis result" must be associated with the "diagnosis time" field), and missing associated fields are supplemented (such as automatically matching the "currency" information in the same record for "order amount"); then, global consistency checking is performed on the data set anchored by the business (such as ensuring that the "user ID" of the same entity is uniformly formatted in all records); finally, all data that passes the checking is integrated to generate a standardized multi-source heterogeneous data set with uniform field format and consistent semantics. This technology is different from the traditional standardization method that only does format conversion. The traditional method ignores business association constraints, resulting in broken business logic in the standardized data. However, the present application combines semantic vector driven format conversion and business anchoring to ensure format uniformity while strengthening the business integrity of the data, which is more in line with actual analysis needs.

[0035] Optionally, the step 3 specifically comprises the following steps: Step 31: calling a multi-modal semantic recognition module of the data enhancement component to perform active content scanning on the standardized multi-source heterogeneous data set to identify business entities, object entities and event entities; Step 32: calling a dynamic blood relationship link analysis module of the data enhancement component to perform entity-data source-processing node association tracing processing on the business entities, object entities and event entities to determine the full-link blood relationship of the entities; Step 33: performing blood semantic annotation and entity attribute binding processing on the full-link blood relationship of the entities to generate enhanced business metadata.

[0036] Optionally, the step 31 specifically comprises the following steps: Step 311: calling a domain adaptation model in the multi-modal semantic recognition module to perform type slicing and feature adaptation processing on the standardized multi-source heterogeneous data set and generate type-sliced entity information; Step 312: calling an entity semantic aggregation sub-module in the multi-modal semantic recognition module to perform cross-modal entity attribute association completion and conflict resolution processing on the type-sliced entity information to generate cross-modal association completed entity data; Step 313: Call the entity type dynamic decision sub-module in the multi-modal semantic recognition module to perform entity feature-type mapping processing on the cross-modal association completion entity dataset to determine business entities, object entities, and event entities.

[0037] Preferably, the specific implementation process of step 311 is as follows: the multi-modal semantic recognition module first loads a domain adaptation model matched with the business scenario (such as loading a "user-account-transaction" adaptation model in the financial field and a "patient-diagnosis-drug" adaptation model in the medical field), which contains feature extractors for different data types (structured data feature extractor, semi-structured token parser, and unstructured media feature encoder); then, type slicing is performed on the standardized multi-source heterogeneous dataset, and the data is split into three data slices according to structured (such as database table records), semi-structured (such as JSON logs), and unstructured (such as medical record documents and monitoring video frames); then, feature adaptation processing is started, the structured data slice extracts field-level features (such as the numerical range of "transaction amount" and the enumeration value of "user level") through the feature extractor, the semi-structured data slice extracts key-value pair features (such as the format feature of "diagnosis time" and the text description feature of "device status") through the token parser, and the unstructured data slice extracts content features (such as the keyword vector of the document and the frame picture feature of the video) through the media feature encoder; finally, the three types of feature adaptation results are integrated, "data type-core feature-source identifier" metadata is added to each data unit, and type-specific entity information is generated. This technology is different from the traditional single model processing method, which uses a general model to process all types of data, resulting in insufficient feature extraction pertinence. However, the present application combines the domain adaptation model with type slicing, making the feature adaptation more suitable for the heterogeneous characteristics of multi-source data.

[0038] Preferably, in the cross-modal entity association scenario (such as the association of the "user" entity of the e-commerce platform with the order record and the customer service voice), when step 312 is implemented: the entity semantic aggregation submodule first reads the sub-type entity information, calls the cross-modal association engine, and establishes an association index between different data shards based on the entity unique identifier (such as user ID, product code) (such as associating "user ID = U1001" in the structured order table with "user ID = U1001" in the semi-structured chat record, and "belonging ID = U1001" in the unstructured avatar picture); then, the entity attributes after association are subjected to completion processing, "registration time", "delivery address" and other attributes are extracted from structured data, "consultation content", "feedback label" and other attributes are extracted from semi-structured data, and "avatar style", "voice emotion" and other attributes are extracted from unstructured data, and are integrated into a complete attribute set; then, a conflict resolution mechanism is started, different values of the same attribute (such as "user gender = male" in structured data and "gender = female" in unstructured voice recognition) are weighted and judged according to the preset data source credibility weight (such as the real-name authentication data weight being higher than the voice recognition) and the time freshness (such as the data in the last 30 days having a higher weight), and the attribute value with a higher comprehensive score is retained; finally, the results after completion and disambiguation are packaged into a "entity ID-cross-modal attribute set-association credibility" structure, and cross-modal association completion entity data is generated. This technology is different from the traditional single-modal entity processing, and the traditional method only processes the entity attributes in the same type of data, resulting in a one-sided entity portrait, while the present application constructs a more comprehensive entity attribute system through cross-modal association and weighted disambiguation.

[0039] Preferably, in the specific technical implementation of step 313: the entity type dynamic decision sub-module first loads the cross-modal association complement entity data, calls the feature-type mapping engine, which contains a preset type feature library (business entity features such as "containing transaction amount, contract number, and other business attributes", object entity features such as "containing physical specifications, device models, and other attributes", and event entity features such as "containing timestamp, participants, and action description attributes"); then, the feature matching is performed on the cross-modal attribute set of each entity, and the matching degrees of the features in the attribute set with the three types of entity type features are calculated (for example, if an entity contains "transaction amount" and "contract number", the matching degree with the business entity feature is 0.85, and the matching degree with the object entity feature is 0.2); then, a type judgment threshold (such as 0.6) is set, if the highest matching degree exceeds the threshold, the entity is directly judged as the type; if the highest matching degree is lower than the threshold or the matching degrees of multiple types are close (difference <0.1), the dynamic decision mechanism is started, and the domain rule library (such as "entities containing the keyword 'order' are preferentially judged as business entities") is called for secondary verification; finally, an explicit type label ("business entity", "object entity", or "event entity") is added to each entity, and an entity type judgment result containing entity ID, type label, and core attribute is generated. This technology is different from the traditional fixed rule classification. The traditional method relies on static rules and is difficult to deal with new entities. Through the feature-type dynamic mapping and the secondary verification of the domain rules, the adaptability of entity type judgment is improved, which is especially suitable for the emerging new entity scenarios in multi-source data.

[0040] Optionally, the step 32 specifically comprises the following steps: Step 321: calling the fingerprint dynamic matching sub-module in the dynamic blood relationship link analysis module, performing cross-dimension association mapping processing of entity core features and data source unique fingerprints on the business entity, object entity, and event entity to generate entity-data source association index; Step 322: calling the weight calculation sub-module in the dynamic blood relationship link analysis module, performing node dependence analysis and influence degree weight assignment processing on the entity-data source association index based on the entity attribute transmission path to generate weight-based time sequence link data; Step 323: calling the blood relationship gap sub-module in the dynamic blood relationship link analysis module, performing entity attribute flow traceback and link gap dynamic completion on the weight-based time sequence link data to generate entity full-link blood relationship.

[0041] Preferably, the specific implementation process of step 321 is as follows: the fingerprint dynamic matching submodule of the dynamic blood link analysis module first reads the cross-modal attribute set of the business entity, object entity and event entity, extracts the entity core features (such as "transaction number" "contract unique code" of the business entity, "device serial number" "product batch number" of the object entity, "event ID" "occurrence timestamp" of the event entity) from it; then, load the data source unique fingerprint library, which contains the static fingerprints (such as database table structure hash value, file format signature) and dynamic fingerprints (such as real-time data stream transmission session identifier, API interface call token) of each data source; then, start the cross-dimension association mapping engine, and establish the association relationship between the entity and the data source through the "entity core feature-data source fingerprint" bidirectional matching algorithm (such as accurate matching "transaction number = TX2023001" with "association number = TX2023001" in the database transaction log, fuzzy matching "device serial number = SN123" with "device identifier = SN123" in the Internet of Things gateway log); finally, the matching result is packaged as a "entity ID-data source fingerprint-matching confidence-association field" four tuple, and an entity-data source association index is generated. This technology is different from the traditional association method based on a single identifier. The traditional method only relies on fixed field matching, and it is difficult to deal with the problem of different identification formats in multi-source data. However, the present application improves the coverage of cross-data source entity association through dynamic fingerprint library and bidirectional matching algorithm.

[0042] Preferably, in the financial data bloodline tracing scenario (such as full-link tracking of credit approval data), step 322 is specifically implemented as follows: the weight calculation submodule first loads the entity-data source association index, calls the path analysis engine, and restores the transmission path of the entity attribute (such as the complete path of “user income proof” from PDF file parsing to structured data table storage to risk control model calculation to approval result generation) based on the data processing pipeline log (such as ETL job record, API call chain); then, each processing node (such as parsing node, storage node, calculation node) in the path is subjected to dependency analysis to identify the parent-child relationship between the nodes (such as “storage node” depends on the output of “parsing node”); then, the influence weight assignment mechanism is started, and each node is assigned an influence weight in the interval [0, 1] according to three dimensions: node type (core processing node weight is higher than auxiliary node), attribute importance (such as “repayment ability score” attribute related node weight is higher than “application channel” attribute node), and data flow frequency (high-frequency access node weight adjustment coefficient is higher); finally, the node links are arranged in chronological order, the influence weight is embedded into the link data, and the weight time sequence link data of “node ID-parent node ID-influence weight-processing timestamp” is generated. This technology is different from the traditional equal weight link analysis. The traditional method cannot reflect the importance difference of nodes in the business flow, while the present application makes the bloodline link more consistent with the actual business influence transmission rule through multi-dimensional dynamic weighting.

[0043] Preferably, in the specific technical implementation of step 323: the blood relationship gap sub-module first reads the weighted time sequence link data, calls the flow trace tracer, and restores the details of the flow of entity attributes between nodes (such as the "user ID card number" is desensitized to "XXXXXX" in the parsing node and is associated to "user ID" in the storage node) based on node processing logs (such as data conversion records, field mapping relationships); then, through time continuity checking (such as checking whether the timestamp difference of adjacent nodes is within a reasonable threshold) and attribute consistency checking (such as checking whether the attribute hash values before and after the flow are matched), the gaps in the link are identified (such as node breakage caused by missing some processing records, and abnormal gaps caused by unrecorded attribute value mutations); then, a dynamic completion mechanism is started, for time continuity gaps, a reasonable node is calculated according to the average processing time of historical links of the same type, and a "speculation" label is marked; for attribute consistency gaps, a similar entity link library (such as the same type of entity processing link of the same data source) is called for pattern matching, the most possible flow record is completed, and a "completed" label is marked; finally, the original link and the completed content are integrated to generate an entity full-link blood relationship containing "complete node link-attribute flow track-gap completion explanation". This technology is different from the traditional way of only recording explicit links. The traditional way is powerless to the implicit flow and record loss in the data processing process, while the present application improves the completeness of the blood link through trace tracking and dynamic completion, especially suitable for complex data scenarios with multi-node collaborative processing.

[0044] Optionally, the step 33 specifically comprises the following steps: Step 331: calling the blood relationship node operation association submodule of the data enhancement component to perform operation association type semantic annotation processing on the entity full-link blood relationship based on entity attribute flow traces to generate blood link data with operation association semantic annotations; Step 332: calling the blood relationship judgment submodule of the data enhancement component to perform directional association of entity core attributes and blood relationship nodes on the blood link data with operation association semantic annotations to generate entity-blood relationship binding data; Step 333: calling the structured encapsulation submodule of the data enhancement component to perform structured encapsulation processing on the entity-blood relationship binding data to generate enhanced business metadata.

[0045] Preferably, the specific implementation process of step 331 is as follows: the blood relation node operation association submodule of the data enhancement component first loads the entity full-link blood relation, which contains complete node link and entity attribute flow track; then, the operation trace extraction engine is called to extract the operation type (such as "field desensitization", "format conversion", "attribute splitting", "data aggregation"), operation execution time and operation execution module information corresponding to each blood relation node from the data processing log (such as ETL conversion log, field cleaning record, model calculation log); then, the semantic annotation mapping mechanism is started to bind the extracted operation information to the corresponding blood relation node, and add semantic annotations in the format of "operation type | execution time | execution module" (such as "field desensitization | 2024-09-03 10:15:30 | data security module" and "format conversion | 2024-09-03 10:18:20 | standardization processing module") to each node; finally, the annotated blood relation node and the original link structure are integrated to ensure that the annotation information corresponds to the node one by one, and the blood link data with operation association semantic annotation is generated. This technology is different from the traditional blood relation description which only records the node topology. The traditional method cannot reflect the specific data processing actions corresponding to the node, which makes it difficult to locate the operation responsibility party during subsequent tracing. However, the application realizes operation association semantic annotation, so that the blood link has dual information of "structure" and "operation", which is more suitable for multi-module collaborative processing of multi-source data management scenarios.

[0046] Preferably, in the entity core attribute traceability scenario (such as the full-link association of the "payment amount" attribute of the e-commerce order), when step 332 is implemented: the blood relationship determination submodule first loads the blood relationship link data with operation association semantic annotation, calls the core attribute extractor, filters the core attributes (such as "order ID" and "payment amount" of the business entity, "device model" and "production batch" of the object entity, and "event ID" and "impact range" of the event entity) from the entity cross-modal attribute set, and forms a core attribute list; then, based on the entity attribute flow track, the transfer path of each core attribute in the blood relationship link is tracked (such as the complete path of "payment amount" from the "order data table" node→"format conversion" node→"aggregation calculation" node→"risk control verification" node); then, a directional association mechanism is started, each core attribute is bound with all blood relationship nodes on its flow path, and the change details of the attribute at each node are recorded (such as "payment amount" in the "format conversion" node is converted from "fen" unit to "yuan" unit, and in the "aggregation calculation" node, it is combined with "discount amount" into "actual payment amount"); finally, the structured data of "entity ID-core attribute-association node list-attribute change details", that is, the entity-blood relationship binding data, is generated. This technology is different from the traditional fuzzy association of attributes and links. The traditional method only shows that the attribute exists in the link, and cannot clearly show the corresponding relationship between the attribute and the specific node. However, through directional association and change recording, the application improves the association accuracy of the core attribute and the blood relationship node.

[0047] Preferably, in the specific technical implementation of step 333: the structured packaging submodule of the data enhancement component first loads the entity-kinship binding data, calls the packaging structure definition engine, loads the preset enhanced business metadata structure template, which contains four fixed field groups: "entity basic information" (entity ID, entity type, data source belonging to), "core attribute set" (core attribute name, attribute type, attribute initial value), "full-link kinship information" (node link with operation annotation, core attribute-node association relationship), and "metadata generation information" (generation time, generation module, version number); then, the entity-kinship binding data is filled in according to the template field group to ensure that the information of each field group is complete and uniform in format (for example, the "full-link kinship information" field group stores the node link and the association relationship in JSON format); then, the packaging verification mechanism is started to check whether the filled template has problems such as field missing, format error, and inconsistent association (such as core attribute-node association relationship breakage), and the parts with problems are returned to the previous step for reprocessing, and the template that passes the verification is executed for structured coding; finally, the structured data conforming to the preset format standard (such as XML or JSON-LD) is generated, that is, the enhanced business metadata. This technology is different from the traditional unstructured metadata storage. In the traditional way, the metadata information is scattered and the format is not uniform, which makes it difficult for the subsequent graph generation component to efficiently parse. However, by standardizing the structured packaging, the enhanced business metadata has high parsability, directly meets the needs of ontology alignment and semantic verification of the graph generation component in step 4, and reduces the subsequent data conversion cost.

[0048] Optionally, step 4 specifically comprises the following steps: Step 41: calling the ontology constraint module of the graph generation component, loading the preset domain ontology framework to perform ontology alignment on the enhanced business metadata and generating ontology-aligned business metadata; Step 42: performing semantic verification on the ontology-aligned business metadata based on the domain ontology framework, correcting entity naming ambiguity and relationship definition deviation, and obtaining standardized business metadata that eliminates cross-data-source semantic ambiguity; Step 43: calling the graph neural network module, mapping the entities in the standardized business metadata to graph nodes and the explicit relationships to initial edges, mining the implicit associations between entities through a graph attention model, completing the graph edge relationships, and generating a semantic knowledge graph.

[0049] Optionally, step 41 specifically comprises the following steps: Step 411: the ontology constraint module loads the domain ontology knowledge base; Step 412: extracting the entity list and the relationship list from the enhanced business metadata, and comparing them with the standard naming of the standard entities and the standard relationship paradigm in the domain ontology knowledge base; Step 413: standardize the entity naming in the entity list that does not meet the standard naming, and redefine the entity relationship in the relationship list that does not meet the standard relationship paradigm, to generate the ontology alignment business metadata.

[0050] Preferably, the specific implementation process of step 411 is as follows: the ontology constraint module of the graph generation component first reads the preset field ontology knowledge base configuration file, which contains the field identification (such as "e-commerce transaction", "industrial Internet of Things", "financial risk control") of the knowledge base, the storage path and the version information; then, according to the current multi-source heterogeneous data business scene (such as matching the "e-commerce transaction" field identification when processing e-commerce order and logistics data integration), the ontology knowledge base of the corresponding field is loaded; the field ontology knowledge base is organized in four layers of "field classification-standard entity set-standard relationship paradigm-semantic constraint rule": the standard entity set contains the standard naming of the entity (such as "user", "order", "logistics order"), the core attribute (such as "user" contains "user ID", "registration time"); the standard relationship paradigm defines the standard relationship between entities (such as "user-creates-order", "order-associates-logistics order"); the semantic constraint rule specifies the value range of the entity attribute (such as "order status" is only allowed to be "to be paid", "paid", "delivered"); finally, the loaded field ontology knowledge base is checked for integrity (checking whether the standard entity, relationship and rule are missing), and after confirming that there is no error, the field ontology knowledge base that can be used for subsequent comparison is formed. This technology is different from the traditional general ontology knowledge base design. The traditional knowledge base is not subdivided by field, resulting in lower adaptability to specific scenes, while the application loads the field knowledge base to make the ontology constraint more in line with the semantic needs of the current business data.

[0051] Preferably, in the cross-data-source ontology alignment scenario (such as the fusion of e-commerce platform data and third-party payment data), step 412 is specifically implemented as follows: the ontology constraint module first loads the domain ontology knowledge base obtained in step 411, and then reads the enhanced business metadata; next, the metadata parsing engine is called to extract the entity list (including entity name and core attribute) and the relationship list (including associated entity pair and relationship description) from the enhanced business metadata, for example, extracting the "buyer" and "order number" entities from the e-commerce data, extracting the "payer" and "transaction number" entities from the payment data, and extracting the "buyer-payout-order number" and "payer-complete-transaction number" associations from the relationship list; then, a multi-dimensional comparison mechanism is started: at the entity level, the extracted entity name and core attribute are compared with the standard entity set in the domain ontology knowledge base for semantic comparison (such as the semantic similarity comparison of "buyer" and "payer" with the standard entity "user", and the comparison of "order number" and "transaction number" with the core attribute "order ID" of the standard entity "order"); at the relationship level, the extracted relationship description is compared with the standard relationship paradigm for structural comparison (such as the relationship subject, object, and relationship word matching degree comparison of "buyer-payout-order number" and the standard paradigm "user-create-order"); finally, the comparison results are recorded, and the three types of difference items of "inconsistent entity naming", "missing entity attribute", and "inconsistent relationship paradigm" are marked, and a preliminary comparison result table is generated. This technique is different from the traditional single-dimensional comparison, which only compares entity names and ignores attribute and relationship structure differences. The present application can more accurately locate the semantic differences between cross-data sources through entity-relationship multi-dimensional collaborative comparison.

[0052] Preferably, in the specific technical implementation of step 413: the ontology constraint module first reads the preliminary comparison result table generated in step 412, and screens out the difference items of “inconsistent entity naming” and “inconsistent relationship paradigm”; for the difference items of inconsistent entity naming, the semantic matching engine is called to calculate the semantic similarity of non-standard entity names and standard entity names (such as the similarity of “buyer” and “user” is 0.92, and the similarity of “transaction number” and “order ID” is 0.88), when the similarity exceeds the preset threshold (such as 0.8), the standardization replacement (replace “buyer” with “user”, and replace “transaction number” with “order ID”) is performed, and the original name is retained as a note (such as “user [original name: buyer]”) at the same time, which is convenient for subsequent tracing; for the difference items of inconsistent relationship paradigm, the standard relationship paradigm of the domain ontology knowledge base is used for redefinition (such as “buyer-payout-order number” is redefined as “user-create-order”, and “payout person-complete-transaction number” is redefined as “user-pay-order”), to ensure that the subject, object and relationship word of the relationship completely comply with the standard; then, the replaced entity list and the redefined relationship list are subjected to secondary verification (to check whether there are still items inconsistent with the standard); finally, the entity list and the relationship list that pass the verification are integrated with the blood relationship information in the enhanced business metadata to generate ontology-aligned business metadata with unified structure and standardized semantics. This technology is different from the traditional manual alignment method, which relies on technical personnel to manually modify the naming and relationship, and is low in efficiency and prone to errors. The present application improves the efficiency and consistency of ontology alignment through semantic-driven automatic replacement and definition, and is more suitable for the scene of high-frequency update of multi-source data.

[0053] Optionally, the step 42 specifically comprises the following steps: Step 421: processing the ontology-aligned business metadata to determine entity semantic anchoring differences and relationship topology deviations through entity semantic feature cross-source anchoring and relationship constraint topology mapping; Step 422: dynamically calculating the deviation influence weight of the entity semantic anchoring differences and the relationship topology deviations to determine the flow transfer influence weight of the entity in the business link, and performing topology constraint adaptation processing on the relationship topology deviations based on the flow transfer influence weight to correct the entity naming ambiguity and the relationship definition deviation and obtain calibrated business metadata; Step 423: performing cross-source semantic deduction processing on the calibrated business metadata to generate standardized business metadata that eliminates cross-data-source semantic ambiguity.

[0054] Preferably, the specific implementation process of step 421 is as follows: the semantic verification module of the graph generation component first loads the ontology alignment business metadata, calls the entity semantic feature extraction engine to extract the core semantic features of each entity (such as the features of “transaction amount”, “payment status”, “belonging user” of the “order” entity) from the metadata, and groups and marks them according to the data source origin (such as “e-commerce platform data source-order” and “third-party payment data source-order”); then, the entity semantic feature cross-source anchoring mechanism is started, the core semantic features of entities with the same name or similar semantics (such as “e-commerce order” and “payment order”) in different data sources are compared one by one, and through calculating the feature coincidence degree (such as the definition coincidence degree of the “transaction amount” field in two data sources is 0.85, and the enumeration value coincidence degree of “payment status” is 0.7), the entity semantic anchoring differences such as inconsistent feature definitions (such as the “transaction amount” unit is “yuan” and “fen” respectively) and feature missing (such as “e-commerce order” has “logistics status” while “payment order” does not) are marked; at the same time, the relationship topology mapping engine is called to construct the relationship topology graph according to the “entity A-relation-entity B” triple structure of the entity relationship in the ontology alignment business metadata, and compare it with the standard relationship topology graph (such as “user-create-order-include-goods”) in the domain ontology framework to mark the relationship topology deviations such as incorrect relationship direction (such as “order-create-user”) and relationship redundancy (such as repeatedly defining “order-associate-user” and “order-belong-to-user”); finally, the entity semantic anchoring differences and the relationship topology deviations are integrated to form the entity semantic anchoring differences and the relationship topology deviations. This technology is different from the traditional semantic verification within a single data source. The traditional method ignores the feature and topology differences across data sources, while the present application accurately locates the deep semantic conflicts in multi-source data fusion through cross-source anchoring and topology mapping.

[0055] Preferably, in the financial data cross-source integration scenario (such as bank core system and credit system data fusion), step 422 is specifically implemented as follows: the semantic verification module first loads the entity semantic anchor difference and relationship topology deviation generated in step 421, calls the deviation influence weight calculation engine, and assigns an influence weight value (range 0-1) to each deviation item. The weight calculation is based on: the frequency of entity flow in the business link (such as the "loan contract" entity appearing frequently in the approval, lending, and repayment link, the weight is high), the number of associated entities (such as the "customer" entity is associated with "loan contract", "repayment record", "credit report", and other entities, the weight is high), and the deviation type (such as the core feature definition deviation weight is higher than the non-core feature missing deviation). For example, the "ID card number" field format deviation of the "customer" entity (one is 18 characters, and the other contains space separation) is set to 0.9, and the "attachment description" field missing deviation of the "loan contract" is set to 0.3. Then, based on the flow influence weight, the relationship topology deviation is processed for topology constraint adaptation: for high weight (such as >0.7) relationship direction error, directly correct the direction according to the standard topology (such as correcting "loan contract-apply-customer" to "customer-apply-loan contract"); for medium weight (0.3-0.7) relationship redundancy, keep the relationship closest to the standard norm (such as keeping "loan contract-associated-customer", and deleting "loan contract-attribute-customer"); for low weight (<0.3) secondary deviation, record it as a tolerable deviation and add an explanation; then, integrate the corrected entity semantic features and relationship topology to generate calibrated business metadata; finally, perform weight backtracking verification on the calibration result (check whether all high weight deviations have been corrected), to ensure that the correction priority meets the business impact. This technology is different from the traditional equal weight correction method, which may waste resources on low-impact deviations. The present application realizes differentiated correction through dynamic weight calculation, improving the efficiency and business fit of semantic calibration.

[0056] Preferably, in the specific implementation of step 423: the semantic verification module first loads the calibrated business metadata generated in step 422, calls the cross-source semantic deduction engine, and loads the semantic reasoning rule library in the domain ontology framework (such as “if entity A contains entity B, and entity B is associated with entity C, then entity A is indirectly associated with entity C” “when the units of numerical fields are inconsistent, conversion to standard units is required”); Then, start multi-step semantic deduction: first, perform cross-source consistency deduction on the calibrated entity features, for example, the “account opening date” of the “customer” entity in the bank system and the “first application date” in the credit system, verify and correct abnormal values through the rule “the first application date should be later than or equal to the account opening date” (such as correcting records with “first application date earlier than account opening date” to be consistent with the account opening date); second, perform cross-source transitivity deduction on entity relationships, for example, deduce the implicit relationship “customer-owns-mortgage property” from “customer-application-loan contract” and “loan contract-mortgage-property”, and verify whether the relationship conforms to the standard paradigm; third, perform integrity deduction on the entity set across data sources, check whether there are entity pairs that should be associated but are not associated (such as a “loan contract” has a record in the bank system but no associated “repayment plan” in the credit system), and supplement the association mark; then, return the new deviations found in the deduction process to step 422 for recalibration, forming a “verification-correction-deduction-reverification” closed loop; finally, when the deduction result has no new deviation for two consecutive times, output the standardized business metadata that eliminates the semantic ambiguity across data sources. This technology is different from the traditional one-time verification method, which cannot find implicit semantic conflicts. Through multi-step closed-loop deduction, this application deeply excavates the potential semantic inconsistencies of cross-source data, significantly improving the semantic consistency of metadata.

[0057] Optionally, the step 43 specifically comprises the following steps: Step 431: initialize the graph structure of the graph neural network module, taking the entities in the standardized business metadata as nodes and the entity attributes as node attributes, and taking the explicit relationships as initial edges; Step 432: train the initial graph using a graph convolution network, calculate the association weight between nodes, filter out potential associations with weights exceeding a threshold as implicit relationships, and complete the graph edges; Step 433: call the graph self-optimization module to monitor the matching degree of new business metadata and the existing graph, and if there are new entities or relationship changes, automatically update the graph nodes and edges to maintain the timeliness of the semantic knowledge graph.

[0058] Preferably, the specific implementation process of step 431 is as follows: the graph neural network module first loads the normalized business metadata that eliminates cross-data-source semantic ambiguity, calls the entity extractor to extract all independent business entities (such as “user”, “goods”, “order”, “payment record” in the e-commerce scenario) from the normalized business metadata, and assigns a unique identifier (such as “user_001”, “goods_002”) to each entity; then, the core attributes of each entity (such as “registration time” and “belonging area” of “user”, “category ID” and “unit price” of “goods”) are extracted, these attributes are stored in a “attribute name-attribute value-data type” triple format, and are used as attribute information of the corresponding entity node; then, explicit relationships (such as “user-order-order”, “order-contain-goods”, and “order-associate-payment record”) explicitly recorded in the metadata are extracted, these relationships are defined as initial edges between entity nodes, and a relationship type label (such as “order”, “contain”, and “associate”) is added to each edge; finally, the above-mentioned entity nodes, node attributes, and initial edges are encapsulated using a graph data structure to form an initial graph containing a basic topology. This technology is different from the traditional knowledge graph initialization method that only records entities and relationships. The present application provides more rich semantic features for subsequent implicit relationship mining by structuring the storage of entity attributes, so that the graph has a deeper business meaning.

[0059] Preferably, in the scenario of smart retail data association (such as data fusion of online mall and offline store), step 432 is implemented as follows: the graph neural network module first loads the initial graph generated in step 431 to construct an adjacency matrix of the graph, where the rows and columns of the matrix correspond to the entity nodes in the initial graph (such as row "online order" and column "offline member"), and the matrix element values represent the initial association strength between the two entities (0.3 as the basic value if there is an explicit relationship, and 0 if not); then, the improved graph convolution network is used to train the initial graph: the first layer of the network encodes the node attributes (such as "payment channel" and "product list" of "online order", and "consumption frequency" and "preferred brand" of "offline member") to generate entity feature vectors; the middle layer calculates the attention weights between nodes (such as "online order" and "offline member" get higher weights due to high overlap of "product list" and "preferred brand") through graph attention mechanism, and updates the node features based on the adjacency matrix and the attention weights; the output layer calculates the association weight (range 0-1) between any two nodes, which integrates the node attribute similarity, co-occurrence frequency and path transferability (such as the indirect path influence of "online order - product - offline inventory"); then, set the association weight threshold (such as 0.6), and select the potential associations with weight exceeding the threshold as implicit relationships (such as "online order - belongs to - offline member" and "product - substitutes - similar product"); finally, add these implicit relationships as new edges to the initial graph to complete the graph edge relationship. This technology is different from the traditional method of mining implicit relationships based on entity co-occurrence only. By fusing the graph attention mechanism of node attributes and graph structure features, the implicit relationship mining of the present application is more consistent with the business logic, and the calculation of the association weight is more scenario-explaining.

[0060] Preferably, in the specific technical implementation of step 433: the graph self-optimization module first sets the monitoring period of the newly added business metadata (such as once an hour), reads the newly added normalized business metadata (such as the newly added "promotion activity" entity and "user-participation-promotion activity" relationship) in each period; then, the entity matching engine is called to calculate the semantic similarity between the newly added entity and the existing graph node (such as the similarity between "promotion activity" and the existing "marketing activity" node is 0.89), if the similarity exceeds the matching threshold (such as 0.8), the newly added entity attribute is merged into the existing node; if the similarity is lower than the threshold, a new node is created and added to the graph; at the same time, the type of the newly added relationship is checked to determine whether it conforms to the existing relationship paradigm (such as whether "user-participation-promotion activity" conforms to the standard paradigm of "user-participation-activity"), if it does, it is directly added as a new edge, if not, it is added after being modified according to the paradigm; then, the graph consistency check is started to check whether the newly added node or edge causes existing topology conflict (such as the contradiction between "commodity A-replacement-commodity B" and "commodity A-complementary-commodity B"), if there is a conflict, the relationship with higher weight is retained based on the associated weight; finally, the graph update log (recording the newly added node ID, edge relationship type and conflict processing result) is generated, and the dynamic update of the semantic knowledge graph is completed. This technology is different from the traditional knowledge graph which needs manual intervention for updating. Through the automatic monitoring, matching and checking mechanism, the incremental self-optimization of the graph is realized, which can quickly respond to the dynamic changes of multi-source data and maintain the timeliness and consistency of the graph.

[0061] Optionally, the step 5 specifically comprises the following steps: Step 51: based on the physical characteristics of the data source, a dynamic logical data layer is constructed and the underlying physical difference processing is shielded to generate a dynamic logical data layer shielding the physical difference; Step 52: according to the semantic knowledge graph, the dynamic logical data layer shielding the physical difference is mapped to a structured logical data model, and the logical query syntax and data access basic rules adapted to the structured logical data model are defined synchronously to generate a logical data model containing query syntax and access rules; Step 53: the logical data model containing query syntax and access rules is subjected to interface generation adaptation processing based on the general query language, a unified data access interface supporting the general query language is generated, and the permission control logic based on the data security label is embedded to generate a secure and compliant unified data access interface with permission control.

[0062] Optionally, the step 51 specifically comprises the following steps: Step 511: physical characteristic extraction processing is performed on the underlying multi-source heterogeneous data sources associated with the semantic knowledge graph, and the physical characteristics of each data source are extracted, including the physical storage location and access protocol; Step 512, based on the extracted physical characteristics, constructing a dynamic logical data layer shielding physical differences through a preset "data source type-logical adaptation template".

[0063] Preferably, the specific implementation process of step 511 is as follows: the physical characteristic extraction module of the interface generation component first loads the data source association information (such as "user data" is associated with "MySQL database", "MongoDB cluster", and "log file server") recorded in the semantic knowledge graph, calls the multi-source probe engine to send lightweight probe requests (such as database connection test, file metadata reading request) to each associated data source; then, the probe response data is parsed to extract physical characteristics: for structured databases (such as MySQL), IP address, port number, database instance name, and table space distribution (such as "user table stored in shard1 node") are extracted; for semi-structured file servers, storage path (such as " / data / logs / 2023 / ") and file system type (such as HDFS, local FS) are extracted; for unstructured media stream sources (such as cameras, sensors), stream protocol (such as RTSP, MQTT), push frequency, and data cache location are extracted; then, a physical characteristic description matrix is constructed - the matrix rows represent data source types (such as row 1 for "relational database"), the columns represent characteristic categories (such as column 1 for "network location" and column 2 for "access method"), and the matrix elements are specific characteristic values (such as "192.168.1.100:3306" and "JDBC driver"); finally, consistency check is performed on the extracted physical characteristics (such as verifying the connectivity of IP address and port), invalid characteristic values are removed, and a complete data source physical characteristic list is generated. This technology is different from the traditional static configuration of physical characteristics. The present application can automatically adapt to new data source types through dynamic detection and matrix description, and the characteristic records are more comprehensive, providing accurate input for subsequent logical abstraction.

[0064] Preferably, in the cross-departmental integration scenario of government data (such as the integration of public security, civil affairs, and social security system data), when step 512 is implemented, the logical layer construction module of the interface generation component first loads the data source physical property list generated in step 511, calls the template matching engine to match the type of each data source (such as "public security population database" for Oracle database, "civil affairs low-income list" for Excel file, and "social security payment record" for REST API service) with the preset "data source type-logical adaptation template" library; then, based on the matched template, hierarchical logical abstraction is performed: the connection layer template converts the physical access protocol (such as OCI of Oracle, POI of Excel, and HTTPS of API) into a unified logical connection interface (such as the "getConnection()" method), shielding the protocol differences; the data operation layer template maps the physical operation syntax (such as SELECT of SQL, cell reading of Excel, and GET request of API) into standardized logical operations (such as "query()" and "read()"), unifying the operation semantics; the metadata layer template converts the physical storage structure (such as database table structure, Excel column index, and API response JSON structure) into logical data description (such as "[name] string (20), [ID number] string (18)"), eliminating the structural differences; then, the topology relationship graph of the dynamic logical data layer is constructed, and the mapping relationship between the logical interface and the underlying data source is recorded (such as "logical user interface" associated with "Oracle table t_person", "Excel table Sheet1", and "API path / user"); finally, the logical layer verification mechanism is started, and the logical interface is tested whether it can correctly route to the corresponding data source and return a unified format result through simulated queries (such as "query the record with ID number 110XXXX"), and after verification, the dynamic logical data layer shielding the physical differences is generated. This technology is different from the traditional fixed logical layer which only supports a limited number of data source types. Through the extensible adaptation template and hierarchical abstraction architecture, the present application can flexibly access different types of data sources, and the logical interface automatically adapts to changes in the underlying physical properties (such as the connection layer template automatically updating when the data source IP changes), significantly improving the flexibility of multi-source data integration.

[0065] Optionally, the step 52 specifically comprises the following steps: Step 521, extracting entities and inter-entity relationships consistent with the semantic knowledge graph from the dynamic logical data layer; Step 522, mapping the extracted entities and inter-entity relationships into a structured logical data model, which includes entity attribute fields, field type constraints, and entity relationship constraints; Step 523, defining an adapted logical query syntax and data access basic rules for the structured logical data model to generate a logical data model containing the query syntax and access rules.

[0066] Preferably, the specific implementation process of step 521 is as follows: the entity relationship extraction module of the interface generation component loads the dynamic logical data layer shielding physical differences, while importing the defined core entities (such as "customer", "account", "transaction" in the financial scenario) and entity relationship (such as "customer-owns-account", "account-generates-transaction") in the semantic knowledge graph; the semantic mapping engine is called to perform semantic similarity calculation on the logical entities (such as "personal information", "funding account", "transfer record" in the logical layer) in the dynamic logical data layer and the knowledge graph entities (such as the semantic similarity between "personal information" and "customer" is 0.92), and the logical entities with a similarity exceeding a threshold value (such as 0.85) are screened as the entities to be extracted; then, based on the relationship types (such as "owns", "generates") in the knowledge graph, the association links (such as "personal information-associates-funding account", "funding account-triggers-transfer record") between the logical entities in the dynamic logical data layer are identified, and the inter-entity relationship to be extracted is determined through relationship type matching (such as "association" matching "owns", "trigger" matching "generates"); finally, the extracted entities to be extracted and the inter-entity relationship to be extracted are combined to generate an entity-relationship set consistent with the semantic knowledge graph. This technology is different from the traditional way of directly extracting entity relationship from the data layer. By dynamically aligning with the knowledge graph, the extracted entity relationship is ensured to be consistent with the underlying data and the business semantics, solving the problem of relationship extraction deviation caused by "different names for the same thing" in multi-source data.

[0067] Preferably, in the smart medical data integration scenario (such as the fusion of electronic medical records, examination reports, and medication records system), when step 522 is implemented: the model construction module loads the entity-relationship set generated in step 521 (such as entities "patient", "electronic medical record", "examination report", and relationships "patient-generates-electronic medical record" and "patient-receives-examination report"); attribute mapping is performed for each entity: the logical attributes of the entity (such as the logical attributes of "patient" include "name", "ID number", and "date of birth") are extracted from the dynamic logical data layer, the attribute definitions of the corresponding entity in the semantic knowledge graph are referred to (such as "patient" needs to include "unique identifier", "basic information", and "treatment record"), and the logical attributes are mapped to structured fields (such as "ID number" is mapped to the "unique identifier" field, and "name" and "date of birth" are combined into the "basic information" nested field); then, the field type constraints are defined: the "unique identifier" is set to have a string type and length constraint (such as 18 digits), the "treatment time" is set to have a date and time type and format constraint (such as "YYYY-MM-DD HH:MM:SS"), and the "examination value" is set to have a numerical type and range constraint (such as the white blood cell value of blood routine is 3.5-10.5) ); then, define entity relationship constraints: set one-to-many constraint for "patient-generate-electronic medical record" (one patient can generate multiple electronic medical records), set one-to-one constraint for "electronic medical record-associate-examination report" (one electronic medical record corresponds to one initial examination report); finally, encapsulate the above contents in the entity-relation-constraint triple format to generate a structured logical data model containing entity attribute fields, field type constraints, and entity relationship constraints. This technology is different from the traditional manual definition of data model. The present application combines dynamic mapping with knowledge graph constraints, so that the model can accurately reflect the data characteristics and reflect the business rules, and the field constraints automatically adapt to the medical terminology standard (such as ICD-10 coding update) as the knowledge graph updates.

[0068] Preferably, in the specific technical implementation of step 523: the rule definition module loads the structured logical data model generated in step 522 (such as "goods" and "order" models in e-commerce scenarios); design a logical query syntax for the model: the core structure of the syntax adopts the format of "operator + entity + condition + association" (such as "query [operator] goods [entity] where unit price > 100 [condition] associated order [association]"), supports cross-entity nested queries (such as "query user information of users registered in 2023 in the associated orders of the users"), and defines syntax parsing rules (such as the "association" keyword is used to trigger entity relationship traversal); then, define data access basic rules: set access frequency limits based on entity importance (such as "order" entity maximum query 100 times per minute, "goods" entity no frequency limit), set access dependency rules based on entity association relationship (such as querying "order details" needs to verify the validity of "order ID" first), and set visibility rules based on field sensitivity (such as "user mobile phone" field returns only the first 3 digits and the last 4 digits); then, build a syntax-rule association matrix: the rows of the matrix represent the operation types of the logical query syntax (such as "query" "insert" "update"), the columns represent the data access rule types (such as "frequency limit" "dependency rule"), and the matrix elements are specific association relationships (such as "query-order" corresponds to "frequency limit: 100 times / minute"); finally, encapsulate the logical query syntax, data access basic rules and syntax-rule association matrix to generate a logical data model containing query syntax and access rules. This technology is different from the traditional fixed query syntax and independent access rule design. The present application associates the matrix of the query and the rule, so that the query operation automatically adapts to the corresponding access limit, which improves the query flexibility and ensures the normativity of data access.

[0069] Optionally, the step 53 specifically comprises the following steps: Step 531, calling an interface syntax conversion module to perform general query language adaptation processing on the logical data model containing query syntax and access rules to determine the logical query language of the adapted model. Step 532, based on the adapted query language, generate a preliminary unified data access interface supporting general query language; Step 533, obtain the security label of each entity from the semantic knowledge graph, bind the security label with the permission control logic, embed the bound permission control logic into the preliminary unified data access interface to generate a security compliance unified data access interface with permission control.

[0070] Preferably, in the financial data cross-system query scene (such as bank core system, credit system, risk control system data integration), when step 531 is implemented: the interface syntax conversion module loads the logical data model containing query syntax and access rules (such as "customer", "loan contract", "repayment record" model and matching query rules); call the syntax mapping engine to parse the core structure of the logical query syntax (such as "query [operator] customer [entity] where credit score > 600 [condition] associated loan contract [association]"), which is decomposed into four elements: operation type (query), target entity (customer), filter condition (credit score > 600), and association relationship (associated loan contract); then, build a logical-general syntax mapping table, each row in the table corresponds to a logical syntax structure, and each column corresponds to a syntax element of a general query language (such as SQL, GraphQL), and the intersection is a mapping rule (such as "association" keyword is mapped to SQL JOIN operation, "credit score > 600" is mapped to WHERE clause); then, based on the mapping table, convert the decomposed logical syntax elements into general query language fragments (such as converting the above example to "SELECT FROM customer JOIN loan contract ON customer.ID = loan contract.customerID WHERE customer.credit score > 600"); finally, verify the semantic consistency of the conversion result and the original logical query (such as checking whether the association relationship is accurately mapped), and determine the logical query language of the adaptation model as SQL and GraphQL dual mode (SQL for structured query, GraphQL for entity relationship query) after verification. This technology is different from the traditional single language adaptation method. The present application supports multi-language automatic conversion through multi-dimensional mapping table, and can automatically recommend the optimal query language according to the query complexity (such as recommending SQL for simple query and recommending GraphQL for complex association query), thereby improving the flexibility of the interface.

[0071] Preferably, the specific implementation process of step 532 is as follows: the interface generation module loads the logical query language (such as SQL and GraphQL in the financial scenario) of the adaptation model determined in step 531; design the interface basic framework based on the RESTful architecture, define the interface path rules (such as “ / api / v1 / entity name / operation type”), request methods (such as GET corresponds to query, POST corresponds to add), parameter passing format (such as JSON); Next, generate a special interface function for each entity operation: generate a “getCustomer(queryParams)” function for “customer query”, encapsulating the SQL query logic; generate a “getCustomerWithLoans(customerId)” function for “customer-loan contract association query”, encapsulating the GraphQL query logic; At the same time, embed the request parser in the interface, automatically identify the type of input query language and route to the corresponding processing logic; Then, build an interface-entity association matrix, the rows of the matrix represent the interface functions (such as “getCustomer” “getLoans”), the columns represent the entity types (such as “customer” “loan contract”), and the matrix elements are the operation permissions of the interface to the entity (such as “read” “write” “association query”); Finally, simulate multi-language query requests (such as SQL statement “SELECT FROM customer WHERE region = 'East China'” and GraphQL statement “{customer(region: "East China") {ID, name, loan contract {contract number}}} ”) through the interface test engine, verify whether the interface can correctly return the response result in a unified format, and generate a preliminary unified data access interface supporting general query languages after passing the test. This technology is different from the traditional manual interface writing method. The present application ensures the accurate correspondence of interface functions and entity operations through matrix association and automatic generation, and automatically generates supporting interfaces when new entities are added, significantly improving the interface development efficiency.

[0072] Preferably, in the specific technical implementation of step 533: the security interface generation module loads the preliminary unified data access interface generated in step 532, while extracting the security labels of each entity from the semantic knowledge graph (for example, in the financial scenario, "customer ID number" is marked as "highly sensitive", "loan amount" is marked as "moderately sensitive", and "customer name" is marked as "low sensitivity"); call the permission logic generator to convert the security labels into corresponding permission control rules: "highly sensitive" corresponds to the "only administrator role + encrypted transmission" rule, "moderately sensitive" corresponds to the "department manager and above role + operation log recording" rule, and "low sensitivity" corresponds to the "all authenticated users can access" rule; then, construct a label-permission mapping matrix, where the rows represent security label types (high / medium / low sensitivity), the columns represent permission control dimensions (role restriction, transmission encryption, log recording), and the matrix elements are specific control strategies (for example, "highly sensitive-role restriction" corresponds to "administrator"); then, embed the permission control logic defined by the matrix into the request processing link of the preliminary unified data access interface: add a security label identifier at the interface entrance to parse the entity types in the request and match the security labels; add a rule executor at the permission verification link to verify the access permission of the visitor according to the label-permission mapping matrix; add a data desensitization processor at the response link to perform partial masking processing on highly sensitive fields (such as displaying the ID number as "3****************X"); finally, verify the effectiveness of the permission control through security penetration testing (such as whether an ordinary user is denied access to highly sensitive data), and generate a secure and compliant unified data access interface with permission control after testing. This technology is different from the traditional interface and permission separation design. The present application realizes the deep integration of security logic and interface function through the label-permission matrix, and the permission rules are automatically synchronized with the update of the security labels in the knowledge graph, solving the problem of permission control lagging behind the change of data sensitivity level.

[0073] Figure 2 A structure schematic diagram of a data weaving device for multi-source heterogeneous data integration and governance according to an embodiment of the present application is shown in FIG. 1. As shown in FIG. 1, it comprises: Figure 2 A multi-source data acquisition unit for calling a multi-source data acquisition component to acquire data from the accessed multi-source heterogeneous data sources to generate an original multi-source heterogeneous data stream; A multi-source data standardization unit for calling a multi-source data standardization component to standardize the original multi-source heterogeneous data stream to generate a standardized multi-source heterogeneous data set; A data enhancement unit for calling a data enhancement component to actively scan and process the standardized multi-source heterogeneous data set to determine business metadata, and to perform bloodline tracking processing on the business metadata to generate enhanced business metadata; ​The semantic knowledge graph generation unit is configured to invoke a graph generation component, invoke a domain ontology framework to perform normalized constraint on the enhanced business metadata to obtain normalized business metadata eliminating semantic ambiguity across data sources, and perform implicit association mining processing on the normalized business metadata based on a graph neural network to generate a semantic knowledge graph containing core entities and relationships. The unified data access interface generation unit is configured to invoke an interface generation component, perform logical abstraction processing on distributed data resources according to the semantic knowledge graph, and generate a unified data access interface.

[0074] The above merely provides preferred embodiments of the present application but is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A data weaving method for multi-source heterogeneous data integration and governance, characterized in that, Comprise: Step 1, calling a multi-source data acquisition component to collect data from accessed multi-source heterogeneous data sources to generate original multi-source heterogeneous data streams; Step 2, calling a multi-source data standardization component to standardize the original multi-source heterogeneous data streams to generate a standardized multi-source heterogeneous data set; Step 3, calling a data enhancement component to actively scan and process the standardized multi-source heterogeneous data set to determine business metadata and to process the business metadata to generate enhanced business metadata; Step 4, calling a graph generation component to call a domain ontology framework to normalize the enhanced business metadata to eliminate semantic ambiguity across data sources, to mine implicit associations in the normalized business metadata based on a graph neural network, and to generate a semantic knowledge graph containing core entities and relationships; Step 5, calling an interface generation component to perform logical abstraction of distributed data resources based on the semantic knowledge graph to generate a unified data access interface.

2. The method of claim 1, wherein, The step 2 specifically comprises the following steps: Step 21: calling a multi-format parsing engine of the multi-source data standardization component to identify the format of the original multi-source heterogeneous data streams, automatically parse the table structure of structured data, the markup syntax of semi-structured data, and the media encoding format of unstructured data to obtain a parsing result; Step 22: based on a constructed business semantic dictionary, performing semantic enhancement on the parsing result to obtain semantic enhanced parsing data; Step 23: calling a semantic driven standard schema mapping engine to map the semantic enhanced parsing data to a pre-set standard data model to generate a standardized multi-source heterogeneous data set with uniform field format and consistent semantics.

3. The method of claim 2, wherein, The step 23 specifically comprises the following steps: Step 231: calling a semantic disassembly submodule in the standard schema mapping engine to disassemble the semantic enhanced parsing data, parse the data semantics, and match the standard field semantics to generate a semantic enhanced field-standard model field mapping table; Step 232: calling an encoding submodule in the standard schema mapping engine to convert the semantic enhanced field-standard model field mapping table to a semantic vector and calculate the similarity between the corresponding semantic vectors of the associated fields to generate a field semantic matching vector; Step 233: calling a data integration submodule in the standard schema mapping engine to automatically convert the semantic enhanced parsing data to a semantic vector adaptive format data based on the field semantic matching vector according to the field format in the standard model, and to perform business anchoring processing on the semantic vector adaptive format data according to the semantic business anchoring rules in the standard model to generate a standardized multi-source heterogeneous data set with uniform field format and consistent semantics.

4. The method of claim 1, wherein, The step 3 specifically comprises the following steps: Step 31: calling a multi-modal semantic recognition module of the data enhancement component to actively scan the standardized multi-source heterogeneous data set to identify business entities, object entities, and event entities; Step 32: Call the dynamic blood link analysis module of the data enhancement component to perform entity-data source-processing node association tracing processing on business entities, object entities, and event entities to determine entity full-link blood relationships; Step 33: Perform blood semantic annotation and entity attribute binding processing on the entity full-link blood relationships to generate enhanced business metadata.

5. The method of claim 4, wherein, The step 32 specifically includes the following steps: Step 321: Call the fingerprint dynamic matching submodule in the dynamic blood link analysis module to perform cross-dimension association mapping processing of entity core features and data source unique fingerprints on business entities, object entities, and event entities to generate entity-data source association indexes; Step 322: Call the weight calculation submodule in the dynamic blood link analysis module to perform node dependency analysis and influence weight assignment processing based on entity attribute transmission paths on the entity-data source association indexes to generate weighted time sequence link data; Step 323: Call the blood gap submodule in the dynamic blood link analysis module to perform entity attribute flow trace tracing and link gap dynamic completion on the weighted time sequence link data to generate entity full-link blood relationships.

6. The method of claim 4, wherein, The step 33 specifically includes the following steps: Step 331: Call the blood node operation association submodule of the data enhancement component to perform operation association type semantic annotation processing based on entity attribute flow traces on the entity full-link blood relationships to generate blood link data with operation association semantic annotation; Step 332: Call the blood judgment submodule of the data enhancement component to perform directional association of entity core attributes and blood nodes on the blood link data with operation association semantic annotation to generate entity-blood binding data; Step 333: Call the structured encapsulation submodule of the data enhancement component to perform structured encapsulation processing on the entity-blood binding data to generate enhanced business metadata.

7. The method of claim 1, wherein, The step 4 specifically includes the following steps: Step 41: Call the ontology constraint module of the graph generation component to load the pre-set domain ontology framework to perform ontology alignment on the enhanced business metadata and generate ontology-aligned business metadata; Step 42: Perform semantic verification on the ontology-aligned business metadata based on the domain ontology framework to correct entity naming ambiguity and relationship definition deviation, and obtain standardized business metadata that eliminates cross-data source semantic ambiguity; Step 43: Call the graph neural network module to map entities in the standardized business metadata to graph nodes and explicit relationships to initial edges, mine implicit associations between entities through a graph attention model, complete graph edge relationships, and generate a semantic knowledge graph.

8. The method of claim 7, wherein, The step 41 specifically includes the following steps: Step 411: The ontology constraint module loads the domain ontology knowledge base; Step 412: Extract the entity list and relationship list from the enhanced business metadata and compare them with the standard naming of standard entities and the standard relationship paradigm in the domain ontology knowledge base; Step 413: Standardize the entity naming of entities in the entity list that do not conform to the standard naming, and redefine the entity relationships in the relationship list that do not conform to the standard relationship paradigm to generate ontology-aligned business metadata.

9. The method of claim 7, wherein, The step 42 specifically comprises the following steps: Step 421: Perform entity semantic feature cross-source anchoring and relationship constraint topology mapping processing on the ontology alignment business metadata to determine entity semantic anchoring differences and relationship topology deviations; Step 422: Perform deviation influence weight dynamic calculation on the entity semantic anchoring differences and relationship topology deviations to determine the flow transfer influence weight of the entity in the business link, and perform topology constraint adaptation processing on the relationship topology deviations based on the flow transfer influence weight to correct the entity naming ambiguity and relationship definition deviation and obtain calibrated business metadata; Step 423: Perform cross-source semantic reasoning processing on the calibrated business metadata to generate standardized business metadata that eliminates cross-data-source semantic ambiguity.

10. The method of claim 1, wherein, The step 5 specifically comprises the following steps: Step 51: Based on the physical characteristics of the data source, construct a dynamic logical data layer and shield the underlying physical difference processing to generate a dynamic logical data layer that shields the physical difference; Step 52: According to the semantic knowledge graph, map the dynamic logical data layer that shields the physical difference to a structured logical data model, and simultaneously define a logical query syntax and data access basic rule adapted to the structured logical data model to generate a logical data model containing a query syntax and access rule; Step 53: Perform general query language-based interface generation adaptation processing on the logical data model containing the query syntax and access rule, generate a unified data access interface supporting the general query language, and embed permission control logic based on data security labels to generate a secure and compliant unified data access interface with permission control.

Images