Multi-source heterogeneous data access method and system based on intelligent mapping and dynamic interface

Abstract

The invention belongs to the technical field of data processing, and particularly discloses a multi-source heterogeneous data acquisition access method and system based on intelligent mapping and a dynamic interface, and the method comprises the steps: registering an input data source class, generating a data source identifier corresponding to the data type, and storing the data source identifier; respectively executing meta-model analysis operation to generate a standardized meta-model associated with the data source identifier and a container instance template; generating a mapping rule set through a twin neural network architecture with multi-modal feature fusion and a preset mapping rule; a middle layer API interface is constructed through interface logic, and real-time conversion and access between input and output data are achieved; and realizing containerized parallel access based on the container instance template and the containerized interface service instance. According to the method, data source change is dynamically adapted, the real-time performance and accuracy of the mapping rule are ensured, and meanwhile, multi-source heterogeneous data can be accessed efficiently in real time in combination with containerized parallel processing and a data source change monitoring mechanism.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of artificial intelligence, and particularly relates to the technical field of data governance, and is a multi-source heterogeneous data access method and system based on intelligent mapping and dynamic interface. BACKGROUND

[0002] With the continuous development of information technology, it is necessary to integrate massive heterogeneous data from different business systems, different formats and different protocols. Efficient and accurate collection and access of data from various forms of structured, semi-structured and unstructured data sources such as databases, API interfaces, log files, sensor streams and documents have become the key foundation for building a data center and realizing data-driven decision-making.

[0003] In a data collection and access system, the traditional method relies on manual configuration of data source connection information, structure mapping rules and customized development of adaptive interfaces. This approach has significant shortcomings when facing frequent changes in schema (Schema) or multiple types of data sources: First, the manual mapping cost is high and the flexibility is poor: different multi-source heterogeneous data sources need to be manually analyzed for their structure and matched with the target Schema to establish field-level mapping rules . This process is time-consuming and labor-intensive, and is difficult to adapt to dynamic changes in data source Schema, making it difficult to maintain mapping rules and prone to semantic ambiguity and mapping errors.

[0004] Second, the interface adaptability and scalability are weak: in traditional data collection and access systems, dedicated data reading interfaces (APIs) need to be developed for different data sources. When new data source types are added or existing data source structures are changed, the interface needs to be redeveloped and deployed, resulting in long system access cycles, poor scalability and inability to quickly respond to business needs.

[0005] Third, the parallel access efficiency of heterogeneous data is low: traditional serial or simple concurrent processing mechanisms cannot effectively configure multi-channel throughput strategies, making it difficult to ensure the real-time and integrity access of large amounts of, high-speed, heterogeneous data streams, and causing synchronization inconsistencies when data is frequently updated.

[0006] Fourth, the response to changes is lagging: the current system lacks an effective monitoring mechanism for changes in data source Schema structure. When the source data structure changes (such as adding or deleting fields, modifying types), the existing system often cannot timely detect and adjust the mapping rules and interface logic, resulting in subsequent data access failure or data quality degradation, disrupting the consistency and continuity of data access. SUMMARY

[0007] To solve the above problems, the application provides a multi-source heterogeneous data access method and system based on intelligent mapping and dynamic interface, which introduces an intelligent Schema source data mapping engine to automatically generate mapping rules, uses an interface logic automatic generation module to build a dynamically adaptive interface, and combines containerized parallel processing and data source change listening mechanism to realize real-time and efficient access to multi-source heterogeneous data.

[0008] To achieve the above purpose, the application provides the following scheme: In a first aspect, the application provides a multi-source heterogeneous data access method based on intelligent mapping and dynamic interface, comprising the following steps: Step one. Receive the data source type defined by user input or configuration file to register the connection parameters, generate the data source identifiers corresponding to the structured data source, semi-structured data source and unstructured data source, and store them; Step two. Perform meta-model analysis operation on structured data source, semi-structured data source and unstructured data source to generate standardized meta-model associated with the data source identifier, and generate container instance template to be deployed after automatically selecting pre-packaged container image; Step three. Analyze the target Schema data source, establish the mapping relationship between the standardized meta-model and the mapping rule set through the multi-modal feature fusion twin neural network architecture and the preset mapping rule, and generate the mapping rule set; Step four. Build an intermediate layer API interface through interface logic, realize real-time conversion and access between the input of the mapping rule set and the data source identifier and the output of the deployable containerized interface service instance through the intermediate layer API interface; Step five. Realize containerized parallel access based on the container instance template and containerized interface service instance.

[0009] Further, step one specifically includes: According to the unified registration interface, receive the data source type defined by user input or configuration file to register the connection parameters, wherein, The connection parameters of the structured data source include connection string, authentication information and target object identifier; The connection parameters of the semi-structured data source include endpoint URL, request method and example response body; the connection parameters of the unstructured data source include storage path mode and file meta attribute extraction rule; After registration is completed, the data source identifiers corresponding to the structured data source, semi-structured data source and unstructured data source are generated And persistent storage.

[0010] Further, step two specifically includes: For structured data sources, parse the meta-model description by executing DESCRIBE or querying system tables through JDBC / ODBC connection : For semi-structured data sources, parse the meta-model description by syntax analysis and schema inference : For unstructured data sources, parse the meta-model description according to predefined rules : After completing the meta-model parsing, perform standardization processing to generate a standardized meta-model : Automatically select a pre-packaged container image and inject connection parameters and to generate a container instance template to be deployed after the generation; In the above formula, is the field name, is the data type, is the constraint, is the data path, is the inferred data type, is the instance value, is the metadata key, is the metadata value.

[0011] Further, step three specifically includes: Parse the target Schema data source, denoted as : and based on the verified rule set , the standardized meta-model , perform mapping knowledge base construction: Based on the knowledge base and through a multi-modal feature fusion twin neural network architecture, perform mapping function training, and use the joint similarity score of the source-target field pair to train: where, is the target field name, is the target data type, is the target constraint, ​To control the adjustable weight of structure and semantic trade-off, The field name, data type and constraint encoding of and are represented as a feature vector, The semantic vector is generated by encoding the field description text using pre-trained word embedding. The trained mapping function is used to establish a mapping relationship based on the preset mapping rule, and a mapping rule set is generated.

[0012] Further, the step of presetting the mapping rule specifically includes: The standardized meta-model is used to each source field in , all candidate target fields that meet are retrieved by performing candidate matching, and if there is only one that meets: a direct mapping rule first subset is generated: ; If there are multiple high-similarity candidate target fields , an attention-enhanced context discriminator is started to generate a mapping rule second subset with confidence : then , ; If there is no candidate field , an empty mapping marker subset is generated; The mapping rule set is generated from the mapping rule first subset, the mapping rule second subset, and the empty mapping marker subset ; wherein, is a similarity threshold, is a context encoding function implemented by a neural network.

[0013] Further, step four specifically includes: According to the data source type, a pre-defined basic interface template is selected; The mapping rules in the mapping rule set are compiled into template executable logic units; According to the completed registration connection parameters, a complete executable interface code is generated; The runtime dependencies of the executable interface code are scanned statically to generate a lightweight dependency list; Then the optimal base image is selected according to the data source type; The executable interface code and the lightweight dependency list are injected into a build instruction, a containerized build command is executed, a portable container image is generated, an interface instance is output after deployment of the container image, and meta information of the interface instance is input into a service directory to complete construction and deployment of the intermediate layer API interface.

[0014] Further, the step of selecting a predefined base interface template according to the data source type specifically comprises: for a structured source, using a SQL template engine to generate a parameterized query statement; for a semi-structured source, using a JSON / XML conversion template engine to generate a data extraction conversion script; and for an unstructured source, using a file metadata extraction template engine to generate a binary stream processing pipeline.

[0015] Further, the step of realizing containerized parallel access based on the container instance template and the containerized interface service instance further comprises: Real-time monitoring is performed on whether the data source is changed, if changed, the processes of steps one and two are triggered automatically to obtain an updated standardized meta model, and then the mapping relationship of step three is called to generate a new mapping rule set and trigger the interface reconstruction process of step four.

[0016] Further, the step of real-time monitoring whether the data source is changed further comprises using a differential monitoring strategy according to different data source types, periodically polling a database system table or subscribing to a DDL event log for a structured data source; regularly pulling an API instance response for a semi-structured data source, and detecting whether a field is changed through structural hash comparison; and monitoring whether a file meta attribute is changed or whether a timestamp modification event occurs for an unstructured source.

[0017] In a second aspect, the present application further provides a multi-source heterogeneous data access system based on intelligent mapping and dynamic interface, to perform the access method as described in the first aspect, and the multi-source heterogeneous data access system comprises: A connection parameter registration module is configured to receive a data source type defined by a user input or a configuration file to register connection parameters, generate data source identifiers corresponding to structured data sources, semi-structured data sources and unstructured data sources, and store the data source identifiers; An instance template generation module is configured to perform a meta model analysis operation on the structured data sources, the semi-structured data sources and the unstructured data sources to generate standardized meta models associated with the data source identifiers, and generate a container instance template to be deployed after automatically selecting a pre-packaged container image; The mapping rule set generation module is configured to parse a target Schema data source, establish a mapping relationship between the target Schema data source and the standardized meta-model through a multi-modal feature fusion twin neural network architecture and a preset mapping rule, and generate a mapping rule set. The API interface construction module is configured to construct an intermediate layer API interface through interface logic, and realize real-time conversion and access between input of the mapping rule set and a data source identifier and output of a deployable containerized interface service instance through the intermediate layer API interface. The data parallel access module is configured to realize containerized parallel access based on the container instance template and the containerized interface service instance.

[0018] Advantages of the present application: 1. The present application automatically generates field-level rules through a machine learning driven mapping engine, significantly reduces manual costs, avoids semantic ambiguity, dynamically adapts to data source Schema changes, and ensures real-time and accuracy of mapping rules.

[0019] 2. The present application automatically generates a unified adaptive intermediate layer API based on rules, realizes "one development, multiple reuse", supports quick access to new data sources, and containerizes and encapsulates interface logic to realize second-level deployment and elastic scaling of services.

[0020] 3. The present application uses a containerized parallel framework combined with a multi-channel strategy (such as streaming / batch hierarchical processing) to optimize the real-time and integrity of heterogeneous data streams; a change listening mechanism is used to link rules and automatic update of interfaces to ensure zero interruption and high availability of data access services in a dynamic environment. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 A multi-source heterogeneous data access method flowchart based on intelligent mapping and dynamic interface is provided for the present application; Figure 2 A whole architecture schematic diagram in a multi-source heterogeneous data access method based on intelligent mapping and dynamic interface is provided for the present application; Figure 3 A knowledge base mapping and rule generation flowchart in a multi-source heterogeneous data access method based on intelligent mapping and dynamic interface is provided for the present application; Figure 4 A dynamic interface construction and deployment flowchart in a multi-source heterogeneous data access method based on intelligent mapping and dynamic interface is provided for the present application; Figure 5 An interface containerized parallel access and listening framework diagram in a multi-source heterogeneous data access method based on intelligent mapping and dynamic interface is provided for the present application.

[0022] Figure 6A multi-source heterogeneous data access system framework based on intelligent mapping and dynamic interface is provided for the present application. DETAILED DESCRIPTION

[0023] The present application will be further described in detail below with reference to the accompanying drawings. It is necessary to point out here that the following detailed description is only used to further illustrate the present application and cannot be understood as limiting the scope of protection of the present application. Those skilled in the art can make some non-essential improvements and adjustments to the present application according to the above application content.

[0024] The present application provides a multi-source heterogeneous data access method based on intelligent mapping and dynamic interface. The multi-source heterogeneous data set After obtaining the standardized meta-model through the distributed data scheduling module and registering the data source, the standardized meta-model obtains the rule set through the intelligent mapping engine, then the mapping rule obtained is obtained through the interface logic automatic generation module, and finally the containerized parallel access module and the intelligent monitoring module are used to ensure that the API is maintained and updated in time.

[0025] When it is necessary to access the core metadata table (structured, Schema ) of the scientific research literature system, the JSON data stream returned by the open source dynamic public opinion API (semi-structured, Schema ) and the scientific research report, popular audio and video (unstructured, no Schema). In the access stage, the intelligent Schema source data mapping engine first automatically parses the meta-model information of each data source (such as database table structure, JSON field definition, file meta attribute), and combines the pre-defined target reading data Schema S, uses the built-in machine learning mapping function to dynamically generate accurate field-level mapping rule set to solve the semantic mismatch and data structure difference. According to the rule set , the interface logic automatic generation module automatically constructs an intermediate layer unified data reading API that can adapt to the characteristics of each data source and execute necessary conversion logic. When the data flows in, the system uses the black box access technology to encapsulate the access task of each data source as an independent containerized processing unit, and configures a multi-channel throughput strategy (such as assigning a high priority channel to high-frequency transaction data and a high-bandwidth channel to large files) through a distributed framework (such as Kubernetes) to realize parallel and synchronous access of heterogeneous data, and guarantee real-time and integrity. At the same time, the embedded data source change monitoring mechanism continuously monitors the source Schema change (such as adding a field in the scientific research literature system, changing the field name of the open source dynamic public opinion API). Once the change is detected, the mapping rule It also automatically generates and updates the corresponding API logic in conjunction with the interface logic, ensuring that the data content is consistent and the access pipeline remains available.

[0026] To more clearly illustrate the technical solution of this invention, the following detailed description is provided.

[0027] like Figures 1-2 As shown in the figure, this embodiment proposes a method for accessing multi-source heterogeneous data based on intelligent mapping and dynamic interfaces, which specifically includes the following steps: Step 1. Receive user input or data source type defined in the configuration file to register connection parameters, generate and store data source identifiers corresponding to structured data sources, semi-structured data sources and unstructured data sources.

[0028] like Figure 3 As shown, in this embodiment, before accessing multi-source heterogeneous data, it is necessary to complete the registration of the data source and the parsing of its meta-model. Given the target data schema... and a collection of heterogeneous data sources to be connected (such as database tables, API endpoints, and file storage paths). Each data source Associate its metamodel information The purpose of data source registration and metamodel parsing is to extract... The vector representation of .

[0029] Based on a unified registration interface, it receives connection parameters from user input or data source types defined in the configuration file. For registration of structured data sources (such as RESTful APIs), connection parameters include: connection string, authentication information, and target object identifier; connection parameters for semi-structured data sources include: endpoint URL, request method, and sample response body; connection parameters for unstructured data sources (such as image / document storage) include: storage path pattern and file meta-attribute extraction rules. After registration, data source identifiers corresponding to structured, semi-structured, and unstructured data sources are generated. And persist the storage.

[0030] Step 2. Perform metamodel parsing operations on structured data sources, semi-structured data sources, and unstructured data sources respectively to generate standardized metamodels associated with the data source identifiers, and generate container instance templates to be deployed after automatically selecting pre-packaged container images.

[0031] In this embodiment, when parsing the data source, according to Differentiated operations are performed on the data source type to generate a standardized metamodel. .

[0032] For structured data sources, resolve the meta-model description by executing DESCRIBE or querying system tables (such as INFORMATION_SCHEMA.COLUMNS) via JDBC / ODBC connection : For semi-structured data sources, perform syntax parsing and schema inference to resolve the meta-model description : For unstructured data sources, resolve the meta-model description based on predefined rules : After completing the meta-model resolution, the meta-model needs to be standardized and stored, converting heterogeneous data sources into an intermediate representation (such as Protobuf / JSON Schema format) to generate a standardized meta-model , which contains field names, data types, constraints, path mappings, meta attributes, and other key information, and is associated with and stored in the meta-model repository: According to the type and protocol of , automatically select a pre-packaged container image (such as jdbc-ingestor , rest- scraper , file-watcher ), inject connection parameters and to generate a container instance template to be deployed . This template will be scheduled for execution in subsequent steps, realizing the parallelization basis of black box data collection.

[0033] In the above formula, is the field name, is the data type, is the constraint, is the data path, is the inferred data type, is the instance value, is the metadata key, is the metadata value.

[0034] It should be noted that in the foregoing expression, JDBC is generally represented as Java Database Connectivity, ODBC is generally represented as Open Database Connectivity, DESCRIBE is generally represented as describe or view structure, which is used to obtain the structure information of a table, INFORMATION_SCHEMA.COLUMNS is generally represented as information schema table or data dictionary table.

[0035] Step three. Analyzing the target Schema data source, establishing a mapping relationship between the standardized meta-model and the standardization meta-model through the multi-modal feature fusion twin neural network architecture and the preset mapping rule, and generating a mapping rule set.

[0036] In the present embodiment, step three is the core function of the intelligent Schema source data mapping engine, which aims to automatically establish the precise semantic association between the standardized meta-model and the target Schema through the mapping function driven by machine learning, and generate an executable field-level mapping rule set . The input is the standardized meta-model output in stage S1 and the predefined target Schema , and the output is the rule set .

[0037] Analyzing the target Schema data source, loading the machine-readable definition (such as Avro Schema, Protobuf Descriptor) of the target Schema , is represented as: and based on the verified rule set , the standardized meta-model , the mapping knowledge base is constructed: Based on the knowledge base and through the multi-modal feature fusion twin neural network architecture, the mapping function is trained, and the joint similarity score of the source-target field pair is used for training: wherein, is the target field name, is the target data type, is the target constraint, ​to control the adjustable weight of structure and semantic trade-off, representing generating a feature vector of field name, data type and constraint encoding of and , representing encoding field description text using pre-trained word embedding to generate semantic vector; the mapping function after training is used to establish a mapping relationship based on the preset mapping rule, and a mapping rule set is generated.

[0038] In the embodiment, the step of presetting the mapping rule specifically includes: to the standardized meta-model , to each source field in , perform candidate matching to retrieve all candidate target fields in that meet : (1) if there is only one that meets: then generate a direct mapping rule first subset : ; (2) if there are multiple high-similarity candidate target fields , then start the attention-enhanced context discriminator to generate a mapping rule second subset with confidence : then , ; In the embodiment, represents a similarity threshold value, and the attention weight is represented as: is the dimension of , is a context encoding function realized by a neural network, which introduces global semantic context information for field mapping decision.

[0039] (3) if there is no candidate field , then generate an empty mapping marker subset ; generate a mapping rule set from the mapping rule first subset, the mapping rule second subset, and the empty mapping marker subset.

[0040] Step four. Build an intermediate layer API interface through interface logic, and realize real-time conversion and access between input of the mapping rule set and data source identifier and output of a deployable containerized interface service instance through the intermediate layer API interface.

[0041] As Figure 4 shown in the present embodiment, step four is to automatically generate a unified and adaptive intermediate layer API interface based on the mapping rule set in step three through the interface logic automatic generation module, realizing real-time conversion and access from heterogeneous data sources to the target Schema. The input is the rule set and data source identification , and the output is a deployable containerized interface service instance .

[0042] According to the data source type, a predefined basic interface template is selected. For structured sources, a SQL template engine is used to generate parameterized query statements; for semi-structured sources, a JSON / XML conversion template engine is used to generate data extraction and conversion scripts; for unstructured sources, a file metadata extraction template engine is used to generate binary stream processing pipelines.

[0043] The mapping rules in the mapping rule set are compiled into template executable logic units: wherein are the compilation operators of rule to SQL field mapping, jq filter, and metadata extraction function, respectively.

[0044] According to the completed registration connection parameters (e.g. ), a complete executable interface code is generated, and then containerization packaging and deployment are performed. Through standardized container technology, agile delivery of interface services is realized. First, the runtime dependencies (such as Python third-party libraries and Java JAR packages) of are statically scanned to generate a lightweight dependency list ; then the optimal basic image is selected according to the data source type (e.g., jdk17-base for structured sources and python3.9-jq for semi-structured sources), and the interface code is injected into the build instruction; then the containerization build command (docker instruction) is executed to generate a portable container image ; finally, deployment is performed on the Kubernetes cluster , dynamic resource quota configuration is performed, and the service endpoint is exposed , and finally the interface instance is output.

[0045] Finally, the meta information of the interface instance is written into the service directory: Meanwhile, to support real-time response in step five, a Webhook callback is registered to the data source change monitoring system, and when a source schema change event is detected, the interface reloading process is automatically triggered.

[0046] Through the above process, the executable interface code and the lightweight dependency list are injected into the build instruction, the containerized build command is executed, the portable container image is generated, the interface instance is output after the container image is deployed, and the meta information of the interface instance is input into the service directory to complete the construction and deployment of the intermediate layer API interface.

[0047] Step five. Realize containerized parallel access based on the container instance template and the containerized interface service instance.

[0048] As shown in Figure 6 , in this embodiment, step five realizes efficient access of multi-source heterogeneous data through a containerized parallel processing framework, and establishes a real-time monitoring mechanism to dynamically respond to data source structure changes.

[0049] Containerized parallel access is based on the container instance template generated in step two and the interface service instance built in step four, and dynamically schedules data access tasks through a distributed container orchestration platform (such as Kubernetes). Each data source corresponds to an independent containerized processing unit to perform data collection, conversion, and writing. Data collection refers to the container instance extracting raw data from the data source (such as executing SQL queries, calling APIs, and monitoring file directories); data conversion refers to sending raw data to the interface instance through the service endpoint , and converting it into a standardized format according to the mapping rule set of the target Schema .

[0050] At the same time, the system configures differential resource strategies according to the characteristics of the data sources to realize multi-channel throughput strategies. High-frequency streaming data (such as transaction logs) is allocated to a high-priority channel (CPU-intensive container, enabling stream processing mode) and a low-latency transmission strategy (such as gRPC protocol) is set; large-volume unstructured data (such as images / PDF) is allocated to a high-bandwidth channel (high network I / O quota, parallel sharding transmission), and a compression algorithm is enabled to reduce transmission overhead; batch structured data enables batch processing channel (memory-optimized container) to improve throughput efficiency through window aggregation strategy.

[0051] In this embodiment, step five also includes monitoring the data source. Data source change monitoring and adaptive updates are implemented through the system's built-in change monitoring module, employing differentiated monitoring strategies for different data source types. For structured sources, the system periodically polls database system tables (such as INFORMATION_SCHEMA) or subscribes to DDL event logs (such as MySQLBinlog). For semi-structured sources, it periodically pulls API sample responses and detects field changes through structure hash comparison (such as MD5 fingerprinting of JSON Schema). For unstructured sources, it monitors file meta-attribute change events (such as the inotify mechanism) or compares file modification timestamps.

[0052] When a change to the source schema is detected (such as the addition of a new field `new_column`), the change monitoring module generates a change event, automatically triggering steps one and two of the process, and re-parses the data source. Get the updated metamodel Then, the intelligent mapping engine from step three is invoked, based on... and Generate a new rule set Then, the interface logic automatic generation module in step four is triggered to rebuild the interface instance. And inject new rules New instance After the health check, the traffic was gradually switched away from the old instance. Once the old instance completes its existing tasks, it terminates promptly to ensure zero data loss. If rule generation fails (e.g., semantic conflicts prevent automatic matching), the system automatically rolls back to the previous available version and issues an alert.

[0053] In this embodiment, during the entire system operation, metrics such as container resource usage (CPU / memory), data throughput, and end-to-end latency are collected in real time. Based on historical load prediction, container resource quotas are automatically adjusted. At the same time, all schema change events, rule update versions, and interface release logs are recorded to support fault tracing.

[0054] like Figure 6 As shown, the multi-source heterogeneous data access system based on intelligent mapping and dynamic interfaces in this embodiment includes: The connection parameter registration module is used to receive user input or data source type defined in the configuration file to register connection parameters, generate and store data source identifiers based on structured data sources, semi-structured data sources and unstructured data sources; The instance template generation module is used to perform metamodel parsing operations on structured data sources, semi-structured data sources and unstructured data sources respectively to generate a standardized metamodel associated with the data source identifier, and generate a container instance template to be deployed after automatically selecting a pre-packaged container image. The mapping rule set generation module is configured to parse a target Schema data source, establish a mapping relationship between the target Schema data source and the standardized meta-model through a multi-modal feature fusion twin neural network architecture and preset mapping rules, and generate a mapping rule set. The interface logic automatic generation module is configured to build an intermediate layer API interface through interface logic construction, and realize real-time conversion and access between an input of the mapping rule set and a data source identifier and an output of a deployable containerized interface service instance through the intermediate layer API interface. The containerized parallel access module is configured to realize containerized parallel access based on the container instance template and the containerized interface service instance.

[0055] It should be noted that, as shown in Figure 2 The multi-source heterogeneous data access system based on intelligent mapping and dynamic interface can further include a distributed data scheduling module configured to schedule the received multi-source heterogeneous data set to a corresponding data type processing module for data registration or processing after meta-model analysis. Meanwhile, the multi-source heterogeneous data access system based on intelligent mapping and dynamic interface can further include a change monitoring module configured to monitor whether the data source is changed in real time. The step of monitoring whether the data source is changed in real time includes adopting a differential monitoring strategy according to different data source types, periodically polling a database system table or subscribing to a DDL event log for a structured data source; regularly pulling an API instance response for a semi-structured data source, and detecting whether a field is changed through structural hash comparison; monitoring whether a file meta attribute is changed or whether a timestamp modification event occurs for an unstructured source, and judging whether to change according to the monitoring result. The specific process has been described in the foregoing and will not be repeated here.

[0056] In the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting", "fixing" and the like should be understood in a broad sense. For example, it can be fixed connection, or detachable connection, or integral; it can be mechanical connection, or electrical connection; it can be direct connection, or indirect connection through an intermediate medium; it can be internal communication of two elements or interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0057] In the present application, unless otherwise explicitly specified and limited, the first feature is "on" or "under" the second feature can be that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, the first feature is "above", "over" and "on" the second feature can be that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is horizontally higher than the second feature. The first feature is "below", "under" and "under" the second feature can be that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is horizontally lower than the second feature.

[0058] The above-described embodiments only express several implementation manners of the present application, which are described in a more specific and detailed manner, but cannot be understood as a limitation on the patent scope of the present application. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, which all belong to the protection scope of the present application.

Claims

1. A method for accessing multi-source heterogeneous data based on intelligent mapping and dynamic interfaces, characterized in that, Includes the following steps: Step 1. Receive user input or data source type defined in the configuration file to register connection parameters, generate and store data source identifiers corresponding to structured data sources, semi-structured data sources and unstructured data sources; Step 2. Perform metamodel parsing operations on structured data sources, semi-structured data sources, and unstructured data sources respectively to generate standardized metamodels associated with the data source identifiers, and generate container instance templates to be deployed after automatically selecting pre-packaged container images; Step 3. Parse the target schema data source, establish a mapping relationship between it and the standardized meta-model through a Siamese neural network architecture with multimodal feature fusion and preset mapping rules, and generate a mapping rule set; Step 4. Construct an intermediate layer API interface through interface logic, and realize real-time conversion and access between the input of the mapping rule set and data source identifier and the output of a deployable containerized interface service instance through the intermediate layer API interface; Step 5. Implement containerized parallel access based on the container instance template and containerized interface service instance.

2. The multi-source heterogeneous data access method based on intelligent mapping and dynamic interface according to claim 1, characterized in that, Step one specifically includes: Based on a unified registration interface, it registers connection parameters by receiving user input or data source types defined in the configuration file. The connection parameters of the structured data source include: connection string, authentication information, and target object identifier; The connection parameters for the semi-structured data source include: endpoint URL, request method, and sample response body; the connection parameters for the unstructured data source include: storage path mode and file meta-attribute extraction rules. After registration, data source identifiers are generated based on structured data sources, semi-structured data sources, and unstructured data sources. And persist the storage.

3. The multi-source heterogeneous data access method based on intelligent mapping and dynamic interface according to claim 1, characterized in that, Step two specifically includes: For structured data sources, the metamodel description is parsed by executing DESCRIBE or querying the system tables via JDBC / ODBC connection. : Syntax parsing and pattern inference are performed on semi-structured data sources to extract meta-model descriptions. : Parse the metamodel description from unstructured data sources according to predefined rules. : After metamodel parsing is completed, standardization is performed to generate a standardized metamodel. : Automatically select a pre-packaged container image and inject connection parameters. and Then, a container instance template to be deployed is generated; In the above formula, For field names, For data types, For constraints, For data path, To infer the data type, For instance values, For metadata keys, This is a metadata value.

4. The multi-source heterogeneous data access method based on intelligent mapping and dynamic interface according to claim 3, characterized in that, Step three specifically includes: Parse the target schema data source, represented as : And based on the validated rule set Standardized metamodel Mapping knowledge base Build: Based on knowledge base The mapping function is trained using a Siamese neural network architecture that integrates multimodal feature fusion, and the source-target fields are used to perform mapping. Training is performed using the joint similarity scores: in, For the target field name, For the target data type, As a constraint on the goal, To control the adjustable weights of the structural and semantic tradeoffs, Indicates will and The feature vector generated by encoding the field names, data types, and constraints. This indicates that pre-trained word embedding encoding fields are used to describe the text, generating semantic vectors; The trained mapping function establishes a mapping relationship based on preset mapping rules and generates a mapping rule set.

5. The multi-source heterogeneous data access method based on intelligent mapping and dynamic interface according to claim 4, characterized in that, The specific steps for setting up mapping rules include: For standardized meta-model China Each source field Perform candidate matching and retrieval. All of the above satisfy Candidate target fields If there exists a unique satisfy: Then the first subset of direct mapping rules is generated. : ; If there are multiple candidate target fields with high similarity Then, the attention-enhanced context discriminator is activated to generate a confidence-based discriminator. The second subset of mapping rules :but , ; If there are no candidate fields Then an empty mapping subset is generated. ; A mapping rule set is generated from the first subset of mapping rules, the second subset of mapping rules, and the empty mapping tag subset. ; in, For similarity threshold, It is a context encoding function implemented using a neural network.

6. The multi-source heterogeneous data access method based on intelligent mapping and dynamic interface according to claim 5, characterized in that, Step four specifically includes: Select a predefined basic interface template based on the data source type; The mapping rule set The mapping rules are compiled into template-executable logical units; Generate complete executable interface code based on the connection parameters after registration; Static scan of executable interface code runtime dependencies generates a lightweight dependency list; Then select the optimal base image based on the data source type; The executable interface code and lightweight dependency manifest are injected into the build command, the containerized build command is executed, a portable container image is generated, the interface instance is output after the container image is deployed, and the meta information of the interface instance is input into the service directory to complete the construction and deployment of the middle-layer API interface.

7. The multi-source heterogeneous data access method based on intelligent mapping and dynamic interface according to claim 6, characterized in that, The steps for selecting a predefined basic interface template based on the data source type specifically include: for structured sources, using an SQL template engine to generate parameterized query statements; for semi-structured sources, using a JSON / XML conversion template engine to generate data extraction and conversion scripts; and for unstructured sources, using a file metadata extraction template engine to generate binary stream processing pipelines.

8. The multi-source heterogeneous data access method based on intelligent mapping and dynamic interface according to claim 7, characterized in that, The steps for implementing containerized parallel access based on the container instance template and containerized interface service instance also include: The system monitors the data source for changes in real time. If a change occurs, it automatically triggers steps one and two to obtain the updated standardized meta-model. Then, it calls the mapping relationship in step three to generate a new mapping rule set and triggers the interface reconstruction process in step four.

9. The multi-source heterogeneous data access method based on intelligent mapping and dynamic interface according to claim 7, characterized in that, The steps for real-time monitoring of data source changes also include adopting differentiated monitoring strategies based on different data source types. For structured data sources, this involves periodically polling database system tables or subscribing to DDL event logs. For semi-structured data sources, this involves periodically pulling API sample responses and detecting field changes through structure hash comparison. For unstructured sources, this involves monitoring for changes to file metadata attributes or comparing for timestamp modifications.

10. A multi-source heterogeneous data access system based on intelligent mapping and dynamic interfaces, for executing the access method as described in any one of claims 1-9, characterized in that, The multi-source heterogeneous data access system includes: The connection parameter registration module is used to receive user input or data source type defined in the configuration file to register connection parameters, generate and store data source identifiers based on structured data sources, semi-structured data sources and unstructured data sources; The instance template generation module is used to perform metamodel parsing operations on structured data sources, semi-structured data sources and unstructured data sources respectively to generate a standardized metamodel associated with the data source identifier, and generate a container instance template to be deployed after automatically selecting a pre-packaged container image. The mapping rule set generation module is used to parse the target schema data source, establish a mapping relationship between the target schema data source and the standardized meta-model through a multimodal feature fusion Siamese neural network architecture and preset mapping rules, and generate a mapping rule set. The interface logic automatic generation module is used to construct intermediate layer API interfaces through interface logic, and realize real-time conversion and access between the input of the mapping rule set and data source identifier and the output of a deployable containerized interface service instance through the intermediate layer API interfaces; The containerized parallel access module is used to implement containerized parallel access based on the container instance template and the containerized interface service instance.

Images