Data integration method and apparatus

Abstract

This invention provides a data source integration method and apparatus, relating to the field of data processing technology, to effectively isolate raw data from integrated data while ensuring data access flexibility, reducing metadata management complexity, and providing a more efficient and reliable data integration solution. If the access mode is read-only, the data source to be accessed is mapped to at least one logical read-only subspace on a two-dimensional rational number data plane based on its original data format. If the access mode is read-write, the dataset with read permissions from the data source to be accessed is mapped to at least one logical read-only subspace on a two-dimensional rational number data plane based on its original data format, and the storage space with write permissions from the data source to be accessed is mapped to at least one logical write subspace on a two-dimensional rational number data plane. The positions of data in the logical read-only subspace and logical write subspace are represented by two-dimensional coordinate points.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data source integration method and apparatus. Background Technology

[0002] With the rapid development of information technology and the deepening of digital transformation, enterprises and organizations are facing increasing data management challenges. In today's big data era, data has become one of the most important strategic assets for enterprises. However, this data is often scattered across different systems, platforms, and data sources, forming numerous "data silos." These heterogeneous data sources may include various forms such as relational databases, NoSQL databases, data warehouses, cloud storage services, file systems, and API interfaces, and they differ significantly in terms of data format, access protocols, and query languages.

[0003] Both PostgreSQL's Foreign Data Wrapper (FDW) and DataFabric technologies are based on relational schemas for construction and management, and both involve detailed documentation of relational schema metadata. This design approach presents the following common challenges: 1. Metadata bloat problem: When the number of data sources increases, the metadata that needs to be maintained expands rapidly, including table structure definitions, field mapping relationships, data type conversion rules, access permission configurations, etc., which leads to an exponential increase in system management complexity.

[0004] 2. Difficulty in schema evolution: When the source data schema changes, all related mapping definitions and metadata need to be updated synchronously, which is costly to maintain and can easily introduce inconsistencies.

[0005] 3. Limited performance optimization: Mapping based on a fixed pattern is difficult to dynamically optimize for different query patterns and cannot fully utilize the native optimization capabilities of the source data system.

[0006] 4. Insufficient real-time performance: Traditional integration methods often require pre-planning.

[0007] Therefore, there is an urgent need for a data source integration method that can effectively isolate the original data from the integrated data while ensuring data access flexibility, reduce the complexity of metadata management, and provide a more efficient and reliable data integration solution. Summary of the Invention

[0008] This invention provides a data source integration method and apparatus to ensure data access flexibility while effectively isolating raw data from integrated data, reducing metadata management complexity, and providing a more efficient and reliable data integration solution.

[0009] In a first aspect, the present invention provides a data source integration method, the method comprising: Receive data sources to be connected; Determine the access mode of the data source to be accessed; If the access mode of the data source to be accessed is read-only mode, then based on the original data format of the data source to be accessed, the data source to be accessed is mapped to at least one logical read-only subspace on the two-dimensional rational number data plane. If the access mode of the data source to be accessed is read-write mode, then based on the original data format of the data source to be accessed, the dataset with read permission of the data source to be accessed is mapped to at least one logical read-only subspace on the two-dimensional rational number data plane, and the storage space with write permission of the data source to be accessed is mapped to at least one logical write subspace on the two-dimensional rational number data plane. The positions of data in the logical read-only subspace and the logical write subspace are represented by two-dimensional coordinate points.

[0010] Optionally, the method further includes: If a write operation is performed on an already connected data source, the data to be written is added to an isolation identifier and then stored in the logical write subspace and physical storage corresponding to the already connected data source; the isolation identifier is used to distinguish between newly written data and original data.

[0011] Optionally, the method further includes: The metadata of the logical read-only subspace of the data source to be accessed includes the boundary and access mode of the logical read-only subspace on the two-dimensional rational number data plane. The metadata of the logical write subspace of the data source to be accessed includes the boundary of the logical write subspace on the two-dimensional rational number data plane, the distribution of the data source to be accessed in the physical storage space, and the access mode.

[0012] Optionally, the method further includes: The query statement is parsed into a read operation of the corresponding value of the two-dimensional coordinate point of the data to be queried in the logical read-only subspace and / or logical write subspace corresponding to the data source that has been connected; Based on the reading operation of the corresponding values ​​of the two-dimensional coordinate points of the data to be queried, the original data of the data to be queried is obtained through the abstract storage interface; the abstract storage interface encapsulates atomic operations oriented towards the two-dimensional rational number data plane; Based on a standardized data flow component, the raw data of the data to be queried is converted into two-dimensional coordinate points and returned.

[0013] Optionally, if a write operation is performed on an already connected data source, the data to be written is added to an isolation identifier and then stored in the physical storage corresponding to the data source, including: Based on standardized data flow components, the two-dimensional coordinates of the data to be written are added with isolation markers to form the original data format of the data source that has been connected. Write the data with the added isolation identifier into the physical storage of the connected data source.

[0014] Optionally, based on the original data format of the data source to be accessed, the data source to be accessed is mapped to at least one logical read-only subspace or logical write subspace on the two-dimensional rational number data plane, including: If the original data format of the data source to be accessed is a file format, then for each file in the file directory, the file path of the file is converted to obtain the X-axis coordinate; the file content of the file is converted to obtain the Y-axis coordinate; based on the X-axis coordinate, the Y-axis coordinate and the corresponding file sub-content, a set of data point triples for the file is generated; based on the set of data point triples for all the files, the point set of the file directory on the data plane is determined. If the original data format of the data to be accessed is a nested data structure, then for any first element in the nested data structure, the path dimension encoding of the first element is determined based on all the key names in the complete path from the root node to the first element; the position dimension encoding of the first element is determined based on the order in which the first element appears in the same path; and each first element in the nested data structure is mapped to a two-dimensional coordinate form on a two-dimensional rational number plane. If the original data format of the data source to be accessed is a relational data format, then the primary key or row number of the relational data is used as the X-axis coordinate, and the Y-axis coordinate of the relational data is determined according to the database, table name and column name where the relational data is stored, so as to convert the relational data into the form of two-dimensional coordinate points on a two-dimensional rational number data plane.

[0015] Optionally, the method further includes: When a new data source with a read-write access mode is accessed, at least one target subspace is selected from the existing logical write subspaces. The at least one target subspace is divided to generate at least one first new logical write subspace containing all existing data within the target subspace, and at least one second new logical write subspace that does not contain any existing data at the time of division. The first new logic write subspace is reassigned to the physical storage space of the existing data source, the second new logic write subspace is reassigned to the physical storage space of the new data source, and the mapping relationship between the logic write subspace and the physical storage is updated.

[0016] Optionally, storing the data in the physical storage corresponding to the connected data source includes: If the original data format of the accessed data source is a file format, obtain at least one rational number plane data to be stored. Each rational number plane data includes a first target value in the Y-axis dimension and a second target value in the X-axis dimension. For each rational number plane data, perform the following steps: determine the target first-dimensional partition directory corresponding to the rational number plane data based solely on the first target value or simultaneously based on the first target value and the first partition range corresponding to the first-dimensional partition directory existing on the disk; determine the target second-dimensional partition directory corresponding to the rational number plane data based solely on the second target value or simultaneously based on the second target value and the second partition range corresponding to the second-dimensional partition directory in the target first-dimensional partition directory; store the rational number plane data in the file corresponding to the path constructed based on the second target value under the target second-dimensional partition directory. If the original data format of the connected data source is a nested data structure or a relational data format, then the target logical subspace is determined based on the X-axis and Y-axis coordinates of the rational number plane data to be written, and the database that has a mapping relationship with the target logical subspace is taken as the database to be written; the target logical subspace is a partition in the rational number data space; the first table of the database to be written is determined by adding a preset suffix to the first part of the X-axis coordinates of the rational number plane data to be written; the first column of the first table is determined based on the second part of the X-axis coordinates of the rational number plane data to be written; the first part is the part of the X-axis coordinate excluding the second part; the value of the rational number plane data to be written is stored in the first column of the first table of the database to be written.

[0017] Optionally, the method further includes: If the data source to be accessed does not contain data, then the read-only mode of the data source to be accessed becomes invalid, and the read-write mode of the data source to be accessed degenerates into a pure write mode.

[0018] In a second aspect, the present invention provides a data source integration device, the device comprising: The receiving module is used to receive data sources to be connected. A processing module is used to determine the access mode of the data source to be accessed; The processing module is further configured to, if the access mode of the data source to be accessed is read-only mode, map the data source to be accessed to at least one logical read-only subspace on the two-dimensional rational number data plane based on the original data format of the data source to be accessed. The processing module is further configured to, if the access mode of the data source to be accessed is read-write mode, map the read permission of the data source to be accessed to at least one logical read-only subspace on the two-dimensional rational number data plane based on the original data format of the data source to be accessed, and map the write permission of the data source to be accessed to at least one logical write subspace on the two-dimensional rational number data plane. The positions of data in the logical read-only subspace and the logical write subspace are represented by two-dimensional coordinate points.

[0019] Optionally, the processing module is further configured to, if a write operation is performed on an already connected data source, add an isolation identifier to the data to be written and store it in the logical write subspace and physical storage corresponding to the already connected data source; the isolation identifier is used to distinguish between newly written data and original data.

[0020] Optionally, the metadata of the logical read-only subspace of the data source to be accessed includes the boundary and access mode of the logical read-only subspace on the two-dimensional rational number data plane; The metadata of the logical write subspace of the data source to be accessed includes the boundary of the logical write subspace on the two-dimensional rational number data plane, the distribution of the data source to be accessed in the physical storage space, and the access mode.

[0021] Optionally, the processing module is further configured to parse the query statement into a reading operation of the corresponding values ​​of the two-dimensional coordinate points of the data to be queried in the logical read-only subspace and / or logical write subspace corresponding to the connected data source; Based on the reading operation of the corresponding values ​​of the two-dimensional coordinate points of the data to be queried, the original data of the data to be queried is obtained through the abstract storage interface; the abstract storage interface encapsulates atomic operations oriented towards the two-dimensional rational number data plane; Based on a standardized data flow component, the raw data of the data to be queried is converted into two-dimensional coordinate points and returned.

[0022] Optionally, the processing module is further configured to add isolation markers to the two-dimensional coordinate points of the data to be written based on the standardized data stream component to form the original data format of the data that has been connected to the data source. Write the data with the added isolation identifier into the physical storage of the connected data source.

[0023] Optionally, the processing module is further used for If the original data format of the data source to be accessed is a file format, then for each file in the file directory, the file path of the file is converted to obtain the X-axis coordinate; the file content of the file is converted to obtain the Y-axis coordinate; based on the X-axis coordinate, the Y-axis coordinate and the corresponding file sub-content, a set of data point triples for the file is generated; based on the set of data point triples for all the files, the point set of the file directory on the data plane is determined. If the original data format of the data to be accessed is a nested data structure, then for any first element in the nested data structure, the path dimension encoding of the first element is determined based on all the key names in the complete path from the root node to the first element; the position dimension encoding of the first element is determined based on the order in which the first element appears in the same path; and each first element in the nested data structure is mapped to a two-dimensional coordinate form on a two-dimensional rational number plane. If the original data format of the data source to be accessed is a relational data format, then the primary key or row number of the relational data is used as the X-axis coordinate, and the Y-axis coordinate of the relational data is determined according to the database, table name and column name where the relational data is stored, so as to convert the relational data into the form of two-dimensional coordinate points on a two-dimensional rational number data plane.

[0024] Optionally, the processing module is further configured to select at least one target subspace from the existing logical write subspaces when a new data source with a read-write access mode is accessed. The at least one target subspace is divided to generate at least one first new logical write subspace containing all existing data within the target subspace, and at least one second new logical write subspace that does not contain any existing data at the time of division. The first new logic write subspace is reassigned to the physical storage space of the existing data source, the second new logic write subspace is reassigned to the physical storage space of the new data source, and the mapping relationship between the logic write subspace and the physical storage is updated.

[0025] Optionally, the processing module is further used for If the original data format of the accessed data source is a file format, obtain at least one rational number plane data to be stored. Each rational number plane data includes a first target value in the Y-axis dimension and a second target value in the X-axis dimension. For each rational number plane data, perform the following steps: determine the target first-dimensional partition directory corresponding to the rational number plane data based solely on the first target value or simultaneously based on the first target value and the first partition range corresponding to the first-dimensional partition directory existing on the disk; determine the target second-dimensional partition directory corresponding to the rational number plane data based solely on the second target value or simultaneously based on the second target value and the second partition range corresponding to the second-dimensional partition directory in the target first-dimensional partition directory; store the rational number plane data in the file corresponding to the path constructed based on the second target value under the target second-dimensional partition directory. If the original data format of the connected data source is a nested data structure or a relational data format, then the target logical subspace is determined based on the X-axis and Y-axis coordinates of the rational number plane data to be written, and the database that has a mapping relationship with the target logical subspace is taken as the database to be written; the target logical subspace is a partition in the rational number data space; the first table of the database to be written is determined by adding a preset suffix to the first part of the X-axis coordinates of the rational number plane data to be written; the first column of the first table is determined based on the second part of the X-axis coordinates of the rational number plane data to be written; the first part is the part of the X-axis coordinate excluding the second part; the value of the rational number plane data to be written is stored in the first column of the first table of the database to be written.

[0026] Optionally, if the data source to be accessed does not contain data, then the read-only mode of the data source to be accessed becomes invalid, and the read-write mode of the data source to be accessed degenerates into a pure write mode.

[0027] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the data source integration methods described above.

[0028] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the data source integration methods described above.

[0029] Fifthly, the present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the data source integration methods described above.

[0030] The present invention provides a data source integration method that introduces a unified abstract data space (two-dimensional rational number data plane) and automatically and non-intrusively maps it to mutually isolated read-only logical subspaces and write logical subspaces on the two-dimensional rational number data plane based on the access mode of the data source. This fundamentally achieves the separation of read and write paths, and can effectively isolate the original data from the integrated data while ensuring the flexibility of data access. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0032] Figure 1 A flowchart illustrating a data source integration method provided in an embodiment of the present invention; Figure 2 A schematic diagram of the overall system architecture of a data source integration method provided in an embodiment of the present invention; Figure 3 A schematic diagram illustrating a read / write subspace separation mechanism provided in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating automatic mapping based on data source type, provided as an embodiment of the present invention. Figure 5 This is a schematic diagram illustrating a process for adding an isolation flag when writing data, as provided in an embodiment of the present invention. Figure 6 A schematic diagram illustrating metadata comparison provided in an embodiment of the present invention; Figure 7 This is a schematic diagram illustrating a unified operation interface and data stream conversion provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of the structure of a data source integration device provided in an embodiment of the present invention; Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0034] All actions involving the acquisition of signal information or data in this invention are carried out in compliance with the relevant data protection laws and policies of the country where the device is located, and with the authorization granted by the owner of the device.

[0035] In embodiments of the present invention, "multiple" refers to two or more. Terms such as "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or order.

[0036] Direct data source integration refers to not only directly reading existing data from a data source, but also writing relevant data to the data source as needed. Existing direct data source integration methods mainly include: PG's Foreign Data Wrapper (FDW) and Data Fabric technology.

[0037] PostgreSQL's external data wrapper is a standardized mechanism provided by the PostgreSQL database for accessing data stored in external data sources. FDW is implemented based on the SQL / MED (Management of External Data) standard, allowing external data sources to be mapped to foreign tables in the PostgreSQL database. However, this approach has the following limitations: 1. High mapping complexity: The external data packaging of PG requires manual definition of the mapping relationship between the integrated data source and the central PG database schema. Database administrators need to have a detailed understanding of the source data structure and target schema, and manually create the corresponding DDL statements, which results in high maintenance costs and high expenses.

[0038] 2. Poor data isolation: When the data source allows read and write operations, write operations performed through FDW may directly overwrite existing data in the original data source, failing to efficiently achieve complete isolation between the original data and the integrated data. This poses a significant risk in scenarios where maintaining the independence and integrity of the data source is required.

[0039] 3. Limited scalability: Each new data source requires a series of operations, such as installing the corresponding FDW extension, configuring connection parameters, defining external server and user mappings, and creating external tables. The process is cumbersome and prone to errors.

[0040] Data weaving is a more advanced enterprise-level architecture and product concept designed to create a unified, intelligent data management layer. It automatically discovers, understands, and connects data assets distributed across different locations through a metadata-driven approach. Key features of data weaving technology include: 1. Query-oriented design: The data weaving architecture is biased towards read operation optimization. Its main goal is to integrate existing enterprise data and support complex queries across data sources. Although it also supports updating multiple data sources through a unified interface, its write functionality is relatively weak.

[0041] 2. Mapping consistency issue: Since reading and writing use the same mapping method, the original data may be overwritten when performing a write operation, which will destroy the original state of the data and make it impossible to efficiently achieve complete isolation between the original data and the integrated data.

[0042] 3. High architectural complexity: Data weaving requires building complex metadata management systems, semantic layers, data catalogs and other components, which is costly to implement and may be too burdensome for small and medium-sized data integration needs.

[0043] In-depth analysis of existing data source integration technologies reveals the following key shortcomings in current mainstream multi-data source integration technologies. These shortcomings severely restrict the effectiveness and widespread adoption of data integration technologies in practical applications: 1. Data source integration and mapping are highly complex, making real-time integration difficult. In existing technologies, data source integration requires users to pre-declare detailed mapping relationships. This requirement is particularly prominent when dealing with heterogeneous data sources, especially non-relational data sources. Specifically, this manifests in: Mapping definitions are cumbersome: Users need to manually define mapping rules from the source schema to the target schema for each data source, including table structure mapping, field correspondence, data type conversion rules, and constraint conversion. For enterprise-level data sources containing hundreds of tables and thousands of fields, this task is extremely arduous.

[0044] Adaptation to non-relational data sources is challenging: For non-relational data sources such as JSON, XML, key-value stores, graph databases, and time-series databases, schema extraction and structure transformation are required before establishing mapping relationships. This transformation is often lossy and may result in the loss of certain characteristics or semantic information of the original data.

[0045] Dynamic data sources cannot be handled: For data sources with frequently changing patterns (such as log files, dynamic JSON returned by API interfaces, etc.), the predefined mapping relationship will soon become invalid and needs to be constantly updated and maintained, making real-time integration impossible.

[0046] Long integration cycle: From discovering a new data source to completing the integration configuration, it usually takes several days or even weeks, including multiple stages such as requirements analysis, pattern understanding, mapping design, and testing verification, which seriously affects the agility of the business.

[0047] root cause: The root cause of this deficiency lies in the fact that existing technologies rely too heavily on predefined static mapping patterns, lack intelligent pattern discovery and automatic mapping capabilities, and are unable to adapt to the diversity and dynamism of data sources.

[0048] (2) Metadata management is complex and the system has poor scalability. Because existing technologies commonly use SQL query language and are built on relational schemas to facilitate cross-source queries, they require detailed recording of relational schema metadata, which leads to a series of problems: Explosive growth of metadata: Each data source requires recording a large amount of metadata, including table structure definitions, field attributes, index information, constraints, statistics, etc. When the number of data sources increases from a few to hundreds, the amount of metadata may increase thousands of times, facing a problem of rapid expansion.

[0049] Metadata synchronization is costly: It requires establishing a complex metadata synchronization mechanism to ensure that the metadata in the integrated system is consistent with that of various data sources. This synchronization not only consumes a lot of computing resources, but may also lead to metadata inconsistencies due to network latency or data source unavailability.

[0050] Query optimization is challenging: the massive metadata database makes it difficult for the query optimizer to quickly select the optimal execution plan. When performing cross-source join queries, the number of execution paths that need to be evaluated grows exponentially, severely impacting query performance.

[0051] Storage costs are skyrocketing: Metadata itself requires a significant amount of storage space, especially when historical versions and change records need to be preserved. In some scenarios, the storage cost of metadata even exceeds that of the actual data.

[0052] High maintenance complexity: Metadata management requires specialized tools and processes, including metadata modeling, version control, conflict resolution, lifecycle management, etc., which increases the system's operational burden.

[0053] Deeper impact: While this relational schema-based design concept unifies the data access interface conceptually, it introduces serious scalability problems in practice. As the number of data sources and the volume of data increase, the complexity and management costs of the system grow non-linearly, potentially leading to a system that is difficult to maintain and expand.

[0054] This invention provides a data source integration method and apparatus to ensure data access flexibility while effectively isolating raw data from integrated data, reducing metadata management complexity, and providing a more efficient and reliable data integration solution.

[0055] Figure 1 This is a flowchart illustrating a data source integration method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the method includes the following steps: Step 110: Receive the data source to be connected.

[0056] This invention connects to various heterogeneous data sources through standard methods such as file system interfaces or APIs. The data sources to be connected can be data in file formats, nested data structures, relational data formats, etc.

[0057] Figure 2 A schematic diagram of the overall system architecture of a data source integration method provided in an embodiment of the present invention is shown below. Figure 2 As shown, heterogeneous data sources are connected to the system through access adapters. Specifically, relational databases (such as MySQL, PostgreSQL, etc.), file systems (such as CSV, JSON, XML, etc.), and NoSQL databases (such as MongoDB, Redis, etc.) are connected to the system's data source access layer through the IStorage interface.

[0058] Step 120: Determine the access mode of the data source to be connected.

[0059] When integrating a data source, the system allows users or administrators to declare its access mode for each data source, including read-only and read-write modes. This data source declaration differs from existing technologies; it is extremely simplified, requiring only the determination of access permissions without involving any complex schema mapping definitions. This step is the starting point of the data source integration process, providing the foundational strategy for subsequent read / write space separation.

[0060] Specifically, when a data source is connected but does not contain any data, its read-only mode automatically becomes invalid, and the read-write mode automatically degrades to pure write mode. This adaptive mode adjustment enhances the system's flexibility and intelligence.

[0061] Step 130: If the access mode of the data source to be accessed is read-only mode, then based on the original data format of the data source to be accessed, map the data source to be accessed to at least one logical read-only subspace on the two-dimensional rational number data plane.

[0062] Based on the access mode of the data source to be accessed, the logical subspace of the data source to be accessed is determined on the two-dimensional rational number plane. Specifically, if the access mode of the data source to be accessed is read-only, the read permission of the data source to be accessed is mapped to the logical read-only subspace; if the access mode of the data source to be accessed is read-write, the read permission of the data source to be accessed is mapped to the logical read-only subspace, and the write permission is mapped to the logical write subspace.

[0063] In one possible implementation, after mapping the data source to be accessed to at least one logical read-only subspace on the two-dimensional rational number data plane, the metadata of the logical read-only subspace is recorded. The metadata of the logical read-only subspace includes the boundary and access mode of the logical read-only subspace on the two-dimensional rational number data plane.

[0064] Figure 3 This is a schematic diagram of a read / write subspace separation mechanism provided in an embodiment of the present invention, as shown below. Figure 3 As shown, this invention introduces a unified abstract data space, namely a two-dimensional rational number data plane, for data sources with different data formats. If the access mode of the data source to be accessed is read-only, then based on the original data format of the data source to be accessed, the data source to be accessed is mapped to at least one logical read-only subspace on the two-dimensional rational number data plane. The position of the data in the logical read-only subspace is represented by two-dimensional coordinate points. After determining the two-dimensional coordinate points, the value corresponding to the two-dimensional coordinate points can be accessed. The number of logical read-only subspaces mapped to the two-dimensional rational number data plane can be determined according to the actual data volume of the data source. The logical read-only subspace is a logical data space, and no physical data migration occurs during the mapping process. The coordinate range (x1, y1) - (x2, y2) of the logical read-only subspace is determined according to the range of all data mapped from the original data to the two-dimensional rational number plane.

[0065] Figure 4 This is a schematic diagram illustrating automatic mapping based on data source type, provided as an embodiment of the present invention. Figure 4 As shown, if the original data format of the data source to be accessed is a file format, the file path is converted into a hierarchical string, and this string is used as the X-axis coordinate (PATH). The file is divided into blocks, and the block number (or line number) where the file content is located is used as the Y-axis coordinate (KEY). The content of each block in the file is converted into the form of two-dimensional coordinate points on a two-dimensional rational number data plane. The two-dimensional coordinate points include (X-axis coordinate, Y-axis coordinate), and the coordinate values ​​are the corresponding block content.

[0066] If the original data format of the data source to be accessed is a file format, then for each file in the file directory, the file path of the file is converted to obtain the X-axis coordinate; the file content of the file is converted to obtain the Y-axis coordinate; based on the X-axis coordinate, the Y-axis coordinate, and the corresponding file sub-content, a set of data point triples for the file is generated; based on the set of data point triples for all the files, the point set of the file directory on the data plane is determined. Here, the X-axis of the two-dimensional rational number coordinate system is used to represent the file path dimension, and the Y-axis of the two-dimensional rational number coordinate system is used to represent the file content dimension. Each file corresponds to multiple data point triples, and all data point triples of all files constitute the data plane for storage. This realizes the conversion from file directory to data plane, and the integration of file content from different files, facilitating cross-file content operations based on the data plane.

[0067] If the original data format of the data to be accessed is a nested data structure, then for any first element in the nested data structure, the path dimension encoding of the first element is determined based on all key names in the complete path from the root node to the first element; the position dimension encoding of the first element is determined based on the order in which the first element appears in the same path; and each first element in the nested data structure is mapped to a two-dimensional coordinate form on a two-dimensional rational number plane. After converting each element in the nested data structure into a two-dimensional coordinate form on the data plane and storing it, when querying a certain element, only the necessary columns are read, which can avoid loading the entire nested data structure, reduce I / O and memory overhead, and improve query performance, especially when the data volume is huge. After storage, the data types in the same column are consistent, which makes it easier to compress, improves data compression efficiency, and each column can be stored and processed independently, which is suitable for distributed execution. In addition, the converted nested data structure can be queried in a unified manner on the two-dimensional rational number plane with other types of data.

[0068] If the original data format of the data source to be accessed is a relational data format, then the primary key or row number of the relational data is used as the X-axis coordinate, and the Y-axis coordinate of the relational data is determined according to the database, table name and column name where the relational data is stored, so as to convert the relational data into the form of two-dimensional coordinate points on a two-dimensional rational number data plane.

[0069] The mapping process described above is based on the inherent type and structure of the data source, requiring no manual intervention or the writing of mapping rules. This solves the problem of high manual configuration costs in existing technologies and achieves "instant integration." Because this mapping process defines a set of coordinate conversion formulas from the original data to a unified data plane, without moving, copying, or modifying any original data, it is a non-intrusive reading method when reading data from the data source, ensuring absolute read-only security of the original data.

[0070] Step 140: If the access mode of the data source to be accessed is read-write mode, then based on the original data format of the data source to be accessed, the dataset with read permission of the data source to be accessed is mapped to at least one logical read-only subspace on the two-dimensional rational number data plane, and the storage space with write permission of the data source to be accessed is mapped to at least one logical write subspace on the two-dimensional rational number data plane.

[0071] If the access mode of the data source to be accessed is read-write mode, then the dataset with read permissions of the data source to be accessed is mapped to at least one logical read-only subspace on the two-dimensional rational number data plane, and the storage space with write permissions of the data source to be accessed is mapped to at least one logical write subspace on the two-dimensional rational number data plane. The location of data in the logical write subspace is represented by two-dimensional coordinate points. After determining the two-dimensional coordinate points, the value corresponding to the two-dimensional coordinate points can be accessed. The number of logical write subspaces mapped to the two-dimensional rational number data plane can be determined according to the actual data volume of the data source. The write logical subspace is also part of the two-dimensional rational number data plane and is distributed and "virtually hosted" on the physical data source.

[0072] In one possible implementation, if a write operation is performed on an already connected data source, the data to be written is added to an isolation identifier and then stored in the logical write subspace and physical storage corresponding to the data source. The isolation identifier is used to distinguish between the newly written data and the original data.

[0073] Figure 5 This is a schematic diagram illustrating a process for adding an isolation flag when writing data, as provided in an embodiment of the present invention. All data written through the integrated system will automatically have an isolation flag added to ensure physical isolation from the original data.

[0074] In this embodiment of the invention, the isolation identifier is a system-generated unique prefix or suffix that ensures no naming conflict with existing data in the data source. For example, the isolation identifier can be a UUID generated using a hash algorithm. All data written using this method will be automatically appended with a system-specific, globally unique isolation identifier when materialized into a physical data source, used to strictly distinguish the newly written data from the original data in the data source. In the file system, the isolation identifier can be applied to the written filename or directory name, for example, using the isolation identifier as a prefix or suffix for the filename, or as a prefix or suffix for the directory name. In the database, the isolation identifier can be applied to table names, schema names, or key fields of data records, for example, using the isolation identifier as a prefix or suffix for the table name, or as a prefix or suffix for the schema name, or as a prefix or suffix for the key field.

[0075] This isolation and identification mechanism ensures that newly written data in the integrated system forms a physically or logically independent namespace, clearly distinguishing it from the original data. This fundamentally solves the problem in existing technologies where write operations may overwrite or contaminate the original data, thus guaranteeing the security and stability of the source business system.

[0076] Specifically, if the original data format of the accessed data source is a file format, at least one rational number plane data to be stored is obtained. Each rational number plane data includes a first target value in the Y-axis dimension and a second target value in the X-axis dimension. For each rational number plane data, the following steps are performed: The target first-dimensional partition directory corresponding to the rational number plane data is determined based solely on the first target value, or simultaneously based on the first target value and the first partition range corresponding to the first-dimensional partition directory existing on the disk; the target second-dimensional partition directory corresponding to the rational number plane data is determined solely on the second target value, or simultaneously based on the second target value and the second partition range corresponding to the second-dimensional partition directory in the target first-dimensional partition directory; the rational number plane data is stored in the file corresponding to the path constructed based on the second target value under the target second-dimensional partition directory. By utilizing the mapping relationship between the first-dimensional partition directory and the second-dimensional partition directory and the first target value in the Y-axis dimension and the second target value in the X-axis dimension included in the rational number plane data, the unique two-dimensional structure of the rational number plane data can be directly converted into a physical directory, thereby achieving efficient storage of the rational number plane data.

[0077] If the original data format of the connected data source is a nested data structure or a relational data format, then the target logical subspace is determined based on the X-axis and Y-axis coordinates of the rational number plane data to be written, and the databases that have a mapping relationship with the target logical subspace are designated as the databases to be written to; the target logical subspace is a partition in the rational number data space; the first table of the database to be written is determined by adding a preset suffix to the first part of the X-axis coordinates of the rational number plane data to be written; the first column of the first table is determined based on the second part of the X-axis coordinates of the rational number plane data to be written; the first part is the portion of the X-axis coordinate excluding the second part; the value of the rational number plane data to be written is stored in the first column of the first table of the database to be written to. Through a dynamic, hierarchical, and adaptive mapping mechanism, the infinite logical data space (rational number plane) is cleverly and divide-and-conquerly mapped to a finite, discrete set of relational database physical resources (databases, tables, columns). This achieves a transformation from dense to sparse dimensions, ensuring the system's high performance, high scalability, and ease of management.

[0078] In one possible implementation, after mapping the data source to be accessed to at least one logical write subspace on the two-dimensional rational number data plane, the metadata of the logical write subspace is recorded. The metadata of the logical write subspace includes the boundary of the logical write subspace on the two-dimensional rational number data plane, the distribution of the data source to be accessed in the physical storage space, and the access mode.

[0079] In this embodiment of the invention, the recorded metadata is mainly limited to: the subspace boundary definition (i.e., a set of coordinate ranges on the two-dimensional rational number plane) corresponding to each data source and its access mode (read-only / read-write). The system no longer needs to record detailed schema information such as tables, columns, and field types within each data source. No matter how complex the internal structure of a data source is, it corresponds to only one or a few extremely concise subspace metadata entries in the integrated system. Therefore, the total amount of metadata is no longer linearly or even exponentially related to the number of connected data sources, completely solving the "metadata explosion" problem of existing technologies and ensuring that the system remains lightweight, efficient, and stable when integrating tens of thousands of data sources. Figure 6 This is a schematic diagram of metadata comparison provided in an embodiment of the present invention, such as... Figure 6 The comparison demonstrates the differences between traditional solutions and this invention in metadata management. Traditional solutions require storing detailed table structure information, and the amount of metadata increases linearly with the complexity of the data source; this invention only records subspace boundaries, and the amount of metadata is independent of the internal complexity of the data source.

[0080] Existing technologies generally lack native, mandatory physical isolation mechanisms. This invention creatively proposes and implements a simple yet extremely effective physical isolation method. Furthermore, the security levels of existing technologies and this invention differ. Existing technologies rely on careful user operations and application-layer logic control for write security, while the security of this invention is system-level, automated, and mandatory, fundamentally eliminating the risk of original data corruption due to misoperation or improper design. Its security is far superior to existing technologies.

[0081] Existing technologies (FDW and data weaving techniques) are based on a direct mapping from source schema to target schema. This invention, however, is based on a mapping from source data to subspaces within an abstract space, completely eliminating schema-to-schema dependency. Existing technologies share the same mapping for read and write operations, causing write operations to directly affect the original data. This invention, at the logical level, fundamentally separates read and write paths by mapping read and write behaviors to independent logical read-only and logical write subspaces. Existing technologies lack effective isolation mechanisms. This invention, by forcibly attaching an isolation flag during physical writes, ensures absolute physical or logical isolation between the integrated write data and the original data, improving data security.

[0082] In one possible implementation, the present invention provides a standardized abstract storage interface (IStorage) and a standardized data stream conversion component (RowStream), through which various types of data sources are accessed. This interface encapsulates atomic operations for two-dimensional rational number data spaces, such as reading, writing, and deleting by path / coordinate. The standardized data stream conversion component acts as a bidirectional translator between the "physical source format" and the "unified logical coordinate format."

[0083] When reading data, the query statement is parsed into reading operations of the corresponding values ​​of the two-dimensional coordinate points of the data to be queried in the logical read-only subspace and / or logical write subspace corresponding to the connected data source; based on the reading operations of the corresponding values ​​of the two-dimensional coordinate points of the data to be queried, the original data of the data to be queried is obtained through the abstract storage interface; the abstract storage interface encapsulates atomic operations for the two-dimensional rational number data plane; based on the standardized data flow component, the original data of the data to be queried is converted into the form of two-dimensional coordinate points and returned therein, the standardized data flow component is used for the conversion between the original data format of the data source and the form of two-dimensional coordinate points.

[0084] When writing data, the standardized data flow component adds an isolation marker to the two-dimensional coordinate points of the data to be written, forming the original data format of the connected data source; the data with the isolation marker is then written into the physical storage of the connected data source; the standardized data flow component is used for the conversion between the original data format of the data source and the two-dimensional coordinate point format.

[0085] Figure 7 This is a schematic diagram illustrating a unified operation interface and data stream conversion provided in an embodiment of the present invention, as shown below. Figure 7 As shown, when executing a cross-source SQL query, the query is first parsed and transformed into operations on relevant subspaces on a unified data plane. The IStorage implementation class calls the API of the underlying data source to retrieve data, and then the RowStream component converts the returned raw data (whether database rows, JSON objects, or text lines) into a unified (KEY, PATH)->VALUE coordinate point or (X-axis coordinate, Y-axis coordinate)->VALUE format on an immediate, row-by-row basis. When performing insert or delete operations, the RowStream component reverse-engineers the logical coordinate points (KEY, PATH) and VALUE, according to the mapping rules corresponding to their target subspaces, into a physical storage format that the specific data source can understand (e.g., constructing a prefixed SQL INSERT statement or creating a prefixed file).

[0086] Existing query engines require maintaining conversion logic for various SQL dialects or APIs. The execution engine of this invention unifies all operations to operations on abstract coordinate points, completely shielding heterogeneity at the lowest level and greatly simplifying upper-level logic. The RowStream component of this invention is the core of achieving "coordinate-based everything." It unifies all types of data in streaming processing into the (KEY, PATH) -> VALUE paradigm, a unique intermediate expression for achieving ultimate decoupling not found in existing technologies. When a new data source is connected, only an adapter conforming to the IStorage interface and RowStream conversion logic needs to be implemented for seamless integration into the entire system, exhibiting excellent horizontal scalability.

[0087] This invention provides a data source integration method aimed at systematically overcoming the inherent deficiencies of existing technologies in terms of integration efficiency, data security, and system scalability. Specifically, this invention addresses the technical challenge of low integration efficiency and the inability to achieve truly "instant" access caused by the mandatory requirement for users to pre-declare schema mapping relationships (especially for non-relational data sources), thereby achieving efficient and instant integration of heterogeneous data sources with "zero configuration." Simultaneously, this invention aims to solve the serious security risks of original data being tampered with, overwritten, or even endangering the stability of the source business system due to the lack of effective read / write isolation measures. By constructing a native isolation mechanism, it ensures efficient separation between the data generated by the integration system and the original data, while supporting cross-source queries and writes. Furthermore, this invention aims to solve the "metadata explosion" and system scalability bottleneck problems caused by the prevalence of relational models in existing technologies. By proposing an innovative metadata management strategy, it eliminates the linear growth relationship between the volume of metadata and the number of integrated data sources, thus providing a solid and scalable technical foundation for the unified and efficient integration and management of large-scale, massive data sources.

[0088] Figure 8 This is a schematic diagram of the structure of a data source integration device provided in an embodiment of the present invention, as shown below. Figure 8 As shown, the device 800 includes: Receiver module 810 is used to receive the data source to be accessed; Processing module 820 is used to determine the access mode of the data source to be accessed; The processing module 820 is further configured to, if the access mode of the data source to be accessed is read-only mode, map the data source to be accessed to at least one logical read-only subspace on the two-dimensional rational number data plane based on the original data format of the data source to be accessed. The processing module 820 is further configured to, if the access mode of the data source to be accessed is read-write mode, map the read permission of the data source to be accessed to at least one logical read-only subspace on the two-dimensional rational number data plane based on the original data format of the data source to be accessed, and map the write permission of the data source to be accessed to at least one logical write subspace on the two-dimensional rational number data plane. The positions of data in the logical read-only subspace and the logical write subspace are represented by two-dimensional coordinate points.

[0089] It should be noted that the data source integration device provided in this embodiment of the invention can implement all the method steps implemented in the above-mentioned data source integration method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0090] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9 As shown, the electronic device may include a processor 910, a communications interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communications interface 920, and the memory 930 communicate with each other through the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute the data source integration method.

[0091] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0092] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the data source integration method provided by the above methods.

[0093] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the data source integration methods provided by the above methods.

[0094] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0095] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data source integration method, characterized in that, The method includes: Receive data sources to be connected; Determine the access mode of the data source to be accessed; If the access mode of the data source to be accessed is read-only mode, then based on the original data format of the data source to be accessed, the data source to be accessed is mapped to at least one logical read-only subspace on the two-dimensional rational number data plane. If the access mode of the data source to be accessed is read-write mode, then based on the original data format of the data source to be accessed, the dataset with read permission of the data source to be accessed is mapped to at least one logical read-only subspace on the two-dimensional rational number data plane, and the storage space with write permission of the data source to be accessed is mapped to at least one logical write subspace on the two-dimensional rational number data plane. The positions of data in the logical read-only subspace and the logical write subspace are represented by two-dimensional coordinate points.

2. The data source integration method according to claim 1, characterized in that, The method further includes: If a write operation is performed on an already connected data source, the data to be written is added to an isolation identifier and then stored in the logical write subspace and physical storage corresponding to the already connected data source; the isolation identifier is used to distinguish between newly written data and original data.

3. The data source integration method according to claim 1, characterized in that, The method further includes: The metadata of the logical read-only subspace of the data source to be accessed includes the boundary and access mode of the logical read-only subspace on the two-dimensional rational number data plane. The metadata of the logical write subspace of the data source to be accessed includes the boundary of the logical write subspace on the two-dimensional rational number data plane, the distribution of the data source to be accessed in the physical storage space, and the access mode.

4. The data source integration method according to claim 1, characterized in that, The method further includes: The query statement is parsed into a read operation of the corresponding value of the two-dimensional coordinate point of the data to be queried in the logical read-only subspace and / or logical write subspace corresponding to the data source that has been connected; Based on the reading operation of the corresponding values ​​of the two-dimensional coordinate points of the data to be queried, the original data of the data to be queried is obtained through the abstract storage interface; the abstract storage interface encapsulates atomic operations oriented towards the two-dimensional rational number data plane; Based on a standardized data flow component, the raw data of the data to be queried is converted into two-dimensional coordinate points and returned.

5. The data source integration method according to claim 2, characterized in that, If a write operation is performed on an already connected data source, the data to be written is added to an isolation identifier and then stored in the physical storage corresponding to the data source, including: Based on standardized data flow components, the two-dimensional coordinates of the data to be written are added with isolation markers to form the original data format of the data source that has been connected. Write the data with the added isolation identifier into the physical storage of the connected data source.

6. The data source integration method according to claim 1, characterized in that, Based on the original data format of the data source to be accessed, mapping the data source to be accessed to at least one logical read-only subspace or logical write subspace on the two-dimensional rational number data plane includes: If the original data format of the data source to be accessed is a file format, then for each file in the file directory, the file path of the file is converted to obtain the X-axis coordinate; the file content of the file is converted to obtain the Y-axis coordinate; based on the X-axis coordinate, the Y-axis coordinate and the corresponding file sub-content, a set of data point triples for the file is generated; based on the set of data point triples for all the files, the point set of the file directory on the data plane is determined. If the original data format of the data to be accessed is a nested data structure, then for any first element in the nested data structure, the path dimension encoding of the first element is determined based on all the key names in the complete path from the root node to the first element; the position dimension encoding of the first element is determined based on the order in which the first element appears in the same path; and each first element in the nested data structure is mapped to a two-dimensional coordinate form on a two-dimensional rational number plane. If the original data format of the data source to be accessed is a relational data format, then the primary key or row number of the relational data is used as the X-axis coordinate, and the Y-axis coordinate of the relational data is determined according to the database, table name and column name where the relational data is stored, so as to convert the relational data into the form of two-dimensional coordinate points on a two-dimensional rational number data plane.

7. The data source integration method according to claim 1, characterized in that, The method further includes: When a new data source with a read-write access mode is accessed, at least one target subspace is selected from the existing logical write subspaces. The at least one target subspace is divided to generate at least one first new logical write subspace containing all existing data within the target subspace, and at least one second new logical write subspace that does not contain any existing data at the time of division. The first new logic write subspace is reassigned to the physical storage space of the existing data source, the second new logic write subspace is reassigned to the physical storage space of the new data source, and the mapping relationship between the logic write subspace and the physical storage is updated.

8. The data source integration method according to claim 2, characterized in that, Stored in the physical storage corresponding to the connected data source, including: If the original data format of the accessed data source is a file format, obtain at least one rational number plane data to be stored. Each rational number plane data includes a first target value in the Y-axis dimension and a second target value in the X-axis dimension. For each rational number plane data, perform the following steps: determine the target first-dimensional partition directory corresponding to the rational number plane data based solely on the first target value or simultaneously based on the first target value and the first partition range corresponding to the first-dimensional partition directory existing on the disk; determine the target second-dimensional partition directory corresponding to the rational number plane data based solely on the second target value or simultaneously based on the second target value and the second partition range corresponding to the second-dimensional partition directory in the target first-dimensional partition directory; store the rational number plane data in the file corresponding to the path constructed based on the second target value under the target second-dimensional partition directory. If the original data format of the connected data source is a nested data structure or a relational data format, then the target logical subspace is determined based on the X-axis and Y-axis coordinates of the rational number plane data to be written, and the database that has a mapping relationship with the target logical subspace is taken as the database to be written; the target logical subspace is a partition in the rational number data space; the first table of the database to be written is determined by adding a preset suffix to the first part of the X-axis coordinates of the rational number plane data to be written; the first column of the first table is determined based on the second part of the X-axis coordinates of the rational number plane data to be written; the first part is the part of the X-axis coordinate excluding the second part; the value of the rational number plane data to be written is stored in the first column of the first table of the database to be written.

9. The data source integration method according to claim 1, characterized in that, The method further includes: If the data source to be accessed does not contain data, then the read-only mode of the data source to be accessed becomes invalid, and the read-write mode of the data source to be accessed degenerates into a pure write mode.

10. A data source integration device, characterized in that, The device includes: The receiving module is used to receive data sources to be connected. A processing module is used to determine the access mode of the data source to be accessed; The processing module is further configured to, if the access mode of the data source to be accessed is read-only mode, map the data source to be accessed to at least one logical read-only subspace on the two-dimensional rational number data plane based on the original data format of the data source to be accessed. The processing module is further configured to, if the access mode of the data source to be accessed is read-write mode, map the read permission of the data source to be accessed to at least one logical read-only subspace on the two-dimensional rational number data plane based on the original data format of the data source to be accessed, and map the write permission of the data source to be accessed to at least one logical write subspace on the two-dimensional rational number data plane. The positions of data in the logical read-only subspace and the logical write subspace are represented by two-dimensional coordinate points.

Images