Method and device for constructing unified data space of multiple data sources

Abstract

The application provides a multi-data-source unified data space construction method and device, and relates to computer technology.The method comprises the following steps: mapping data of at least one data source to a multi-dimensional space with dense and countable characteristics; for each data source, determining a geometric figure corresponding to the data source based on the mapping points of the data of the data source after being mapped to the multi-dimensional space, wherein the geometric figure contains all the mapping points corresponding to the data source; generating an index coordinate of the data source based on the identification information of the data source and the boundary information of the geometric figure; and establishing an association between the index coordinate and the metadata of the data source, so as to construct a unified data space of the multi-data-source, wherein the metadata of the data source comprises the identification information of the data source and the boundary information. In the process of constructing the unified data space of the multi-data-source, the application can effectively take into account the data real-time performance of the multi-data-source, the dynamic scalability when the data source is accessed and the construction automation.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for constructing a unified data space from multiple data sources. Background Technology

[0002] With the acceleration of digital transformation, enterprises and organizations face highly heterogeneous and distributed data environments. This complex data ecosystem, composed of different technology stacks, storage paradigms, and physical locations, has led to a severe "data silo" phenomenon. To gain holistic business insights from this scattered and heterogeneous data, conduct unified data governance, and develop upper-level intelligent applications, it is essential to first address how to overcome these silos and achieve unified access and collaborative operation across multiple data sources. Therefore, building a "unified data space" that can shield underlying differences and provide a unified view is crucial.

[0003] Existing technologies typically address these challenges using two main approaches: The first is a centralized integration approach, which extracts, transforms, and loads data from various data sources into a centralized data platform through a unified loading and storage mechanism, thereby achieving physical data integration and unified management. The second is a federated access approach, which does not centrally store existing data but instead uses data virtualization or federated query technology to enable on-demand access to data from various sources.

[0004] However, the first approach mentioned above, relying on physical data migration, incurs significant storage, computing, and operational costs. Furthermore, the batch processing synchronization method introduces substantial data latency, making it unsuitable for business scenarios requiring immediate responses. The second approach, on the other hand, heavily relies on manually defined schema mappings and access rules for each data source, resulting in low automation and high maintenance costs and a high risk of errors when data source structures change or scales dynamically. Therefore, constructing a data space that effectively balances the real-time performance of multiple data sources, dynamic scalability during data source access, and automated data construction for unified management of multiple data sources is a pressing technical challenge. Summary of the Invention

[0005] This invention provides a method and apparatus for constructing a unified data space from multiple data sources, which addresses the shortcomings of existing technologies in constructing data spaces from multiple data sources, such as poor data real-time performance, insufficient scalability, and lack of automation. It can effectively balance the real-time performance of data from multiple data sources, the dynamic scalability during data source access, and the automation of construction in the process of constructing a unified data space from multiple data sources.

[0006] This invention provides a method for constructing a unified data space from multiple data sources, comprising: Map data from at least one data source to a multidimensional space that is dense and countable; For each of the aforementioned data sources, based on the mapping points of the data from the data source mapped onto the multidimensional space, the geometric figure corresponding to the data source is determined, and the geometric figure contains all the mapping points corresponding to the data source. Based on the identification information of the data source and the boundary information of the geometric figure, the index coordinates of the data source are generated; Establish the association between the index coordinates and the metadata of the data source to construct a unified data space for multiple data sources. The metadata of the data source includes the identification information of the data source and the boundary information.

[0007] According to the present invention, a method for constructing a unified data space from multiple data sources is provided, wherein the multidimensional space includes a two-dimensional rational number plane.

[0008] According to a method for constructing a unified data space from multiple data sources provided by the present invention, generating the index coordinates of the data sources based on the identification information of the data sources and the boundary information of the geometric figures includes: A bijective function is used to map the identification information of the data source and the boundary information to the index coordinates.

[0009] According to a method for constructing a unified data space from multiple data sources provided by the present invention, determining the geometric shape corresponding to the data source based on the mapping points of the data source mapped onto the multidimensional space includes: Based on the distribution of all the mapping points in the multidimensional space, determine the maximum x-coordinate, minimum x-coordinate, maximum y-coordinate, and minimum y-coordinate of all the mapping points in the multidimensional space. Based on the maximum x-coordinate, the minimum x-coordinate, the maximum y-coordinate, and the minimum y-coordinate, the geometric shape corresponding to the data source is determined.

[0010] According to a method for constructing a unified data space from multiple data sources provided by the present invention, the method further includes: In response to a data query request, determine the query coordinates corresponding to the data to be queried; The query region is constructed with the query coordinate point as the center point and based on a preset neighborhood range; Based on the query region and the unified data space, the target metadata is determined; The data query request is distributed to the target data source corresponding to the identification information in the target metadata to perform the data query operation.

[0011] According to a method for constructing a unified data space from multiple data sources provided by the present invention, the step of determining target metadata based on the query region and the unified data space includes: A spatial indexing algorithm is used to determine the target index coordinates corresponding to at least one target geometric figure that intersects with the query region; Obtain the target metadata corresponding to the target index coordinates from the unified data space.

[0012] According to a method for constructing a unified data space from multiple data sources provided by the present invention, the method further includes: Receive a data source deletion request, the data source deletion request including the identification information of the data source to be deleted and the boundary information of the geometric figure corresponding to the data source to be deleted; Based on the identification information of the data source to be deleted and the boundary information of the geometric shape corresponding to the data source to be deleted, the index coordinates of the data source to be deleted are generated. Remove the index coordinates and associated metadata of the data source to be deleted from the unified data space.

[0013] The present invention also provides a unified data space construction device with multiple data sources, comprising: The mapping module is used to map data from at least one data source to a multidimensional space that has dense and countable properties. The determination module is used to determine the geometric figure corresponding to each data source based on the mapping points of the data source after mapping on the multidimensional space, wherein the geometric figure contains all the mapping points corresponding to the data source. The generation module is used to generate the index coordinates of the data source based on the identification information of the data source and the boundary information of the geometric figure; A module is established to establish the association between the index coordinates and the metadata of the data source, so as to construct a unified data space for multiple data sources. The metadata of the data source includes the identification information of the data source and the boundary information.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a unified data space construction method for multiple data sources as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a unified data space construction method for multiple data sources as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a unified data space construction method for multiple data sources as described above.

[0017] The present invention provides a method and apparatus for constructing a unified data space for multiple data sources. This method maps data from at least one data source to a dense and countable multidimensional space. For each data source, based on the mapping points after mapping the data in the multidimensional space, a corresponding geometric shape is determined. This geometric shape contains all the mapping points corresponding to the data source. Based on the data source's identification information and the boundary information of the geometric shape, index coordinates of the data source are generated. A relationship is established between the index coordinates and the data source's metadata to construct a unified data space for multiple data sources. The data source's metadata includes its identification information and boundary information. Because a logically unified data space can be constructed, all operations are completed at the metadata index level. Therefore, zero-copy data access can be achieved, fundamentally eliminating data latency caused by the ETL process and ensuring real-time data access. Furthermore, by using a dense and countable multidimensional space as the mapping basis and generating unique index coordinates for each data source based on its identification information and the boundary information of its geometric shape, the metadata size of the data source is linearly related only to the number of data sources. Therefore, it can overcome the metadata management bottleneck of traditional discrete grid models and improve the dynamic scalability when accessing data sources. In addition, unique index coordinates can be automatically generated based on the data source's identification and geometric boundary information, and the association between the index coordinates and metadata can be established. This process does not require manual definition of mapping rules. Therefore, the entire process of data source access, registration and indexing can be automated, which significantly improves the automation level and efficiency of data space construction. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts illustrating the method for constructing a unified data space from multiple data sources provided in an embodiment of the present invention.

[0020] Figure 2 This is a schematic diagram illustrating the generation of index coordinates provided in an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram illustrating data querying as provided in an embodiment of the present invention.

[0022] Figure 4 This is the second flowchart illustrating the method for constructing a unified data space from multiple data sources provided in this embodiment of the invention.

[0023] Figure 5 This is a schematic diagram of the structure of a unified data space construction device for multiple data sources provided in an embodiment of the present invention.

[0024] Figure 6 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] With the acceleration of digital transformation, industries across all sectors are generating and accumulating massive amounts of data at an unprecedented pace. This data exhibits highly diverse characteristics; it may be stored in traditional relational databases (such as MySQL and Oracle), or distributed across non-relational databases (such as MongoDB and Redis), big data file systems (such as the Hadoop Distributed File System (HDFS)), cloud object storage (such as S3), and even real-time streaming data platforms (such as Kafka) and third-party application programming interfaces (APIs). This complex data ecosystem, composed of different technology stacks, storage paradigms, and physical locations, has led to the widespread problem of "data silos." If enterprises or organizations want to extract comprehensive business insights, conduct unified data governance, or develop upper-level intelligent applications from this scattered and heterogeneous data, they must first solve the problem of how to overcome these silos and achieve unified access and collaborative operation across multiple data sources. Therefore, building a "unified data space" that can shield underlying differences and provide a unified view has become a key technical challenge in modern data architecture.

[0027] To address this challenge, existing technologies have mainly developed two main implementation paths. The first type of solution is the "centralized integration" path, which involves extracting, transforming, and loading data from various data sources into a centralized data platform through a unified loading and storage mechanism. Examples include classic data warehouses, data lakes, or hybrid lake-warehouse systems.

[0028] However, due to the physical migration and transformation of data, the Extract-Transform-Load (ETL) process itself is a resource-intensive task, involving complex script development, significant computational resource consumption, and massive storage space usage. Both the initial full data loading and subsequent incremental synchronization constitute a huge development, maintenance, and hardware cost. When the data source structure changes, the ETL pipeline often needs to be redesigned and adjusted, further exacerbating the complexity and cost of maintenance.

[0029] Furthermore, because ETL processes are typically executed periodically in batch mode (e.g., daily, hourly), there is an inherent time delay between the data in the unified data space and the original data source. This "data obsolescence" or "data lag" problem means that analysis and decision-making based on this data space are built on outdated information, resulting in serious data timeliness issues. This makes real-time data access insufficient and unable to support business scenarios requiring immediate responses, such as real-time risk control, real-time recommendations, or real-time monitoring of production operations.

[0030] The second type of solution is the "federated access" path, which does not centrally store the original data but uses data virtualization or federated query technology to achieve on-demand access to data from various sources. For example, PostgreSQL's Foreign Data Wrapper (FDW), Apache Calcite's multi-data source interface layer, and Spark's data virtualization interface all map different types of data sources to a unified relational query space through access bridging layers, enabling users to query distributed data as if it were a single data table.

[0031] This method typically requires users or administrators to explicitly define the access methods, schema mappings, and field correspondences for each data source. In other words, the system lacks the intelligent capabilities for automatic identification, semantic fusion, and spatial expansion of multi-source data. Adding a new data source means adding a new set of manually defined and adapted rules, which is not only labor-intensive but also prone to errors, resulting in insufficient automation. Furthermore, this method usually maps different data sources to a unified discrete table model (i.e., an integer grid-based data plane). While this logically achieves structural uniformity, when faced with complex semi-structured data (such as logs, graph data, and document data), the overly rigid discrete model cannot fully express its hierarchical and semantic features.

[0032] Furthermore, discrete table models require maintaining a large amount of metadata for each field and each mapping relationship. As the scale of data sources connected to the system expands from millions to hundreds of millions or even higher, the volume of metadata grows exponentially, making unified management and retrieval a performance bottleneck. When performing global optimization or query planning, the system must traverse massive amounts of metadata table entries, significantly increasing query latency. In addition, systems that heavily rely on manual configuration of mappings are prone to incompatibility issues when versions are updated or the source structure changes, further limiting the system's dynamic scalability.

[0033] Therefore, neither centralized data integration methods, characterized by heavy processing and low real-time performance, nor federated virtualization methods, characterized by high fragmentation and low automation, can effectively balance the real-time performance of data from multiple data sources, the dynamic scalability during data source access, and construction efficiency. Thus, how to construct a data space that can effectively balance the real-time performance of data from multiple data sources, the dynamic scalability during data source access, and automated construction for unified management of multiple data sources is a pressing technical problem that needs to be solved.

[0034] To address the aforementioned issues, the unified data space construction method for multiple data sources in this embodiment of the invention provides an efficient architecture mechanism that maintains the autonomy of data sources while achieving unified semantics and spatial mapping. It eliminates the need for centralized physical integration of the original data sources and enables unified data description, semantic mapping, and access operations globally. In the process of constructing a unified data space for multiple data sources, it effectively balances the real-time performance of data from multiple data sources, the dynamic scalability during data source access, and the automation of construction.

[0035] The following is combined Figures 1 to 4 This invention describes a method for constructing a unified data space from multiple data sources, as provided in embodiments of the present invention. These embodiments are applicable to scenarios requiring unified access, governance, and analysis of heterogeneous data from multiple sources, such as real-time data analysis across business systems and global querying of massive amounts of IoT sensor data.

[0036] The subject executing this method can be an electronic device such as a terminal device, computer, server, server cluster, or specially designed unified data space construction device for multiple data sources. It can also be a unified data space construction device for multiple data sources installed in the electronic device. The unified data space construction device for multiple data sources can be implemented by software, hardware, or a combination of both.

[0037] Figure 1 This is one of the flowcharts illustrating the unified data space construction method for multiple data sources provided in this embodiment of the invention, such as... Figure 1 As shown, the method includes: Step 101: Map data from at least one data source to a multidimensional space that is dense and countable.

[0038] In this step, the multidimensional space possesses the properties of being dense and countable, including a two-dimensional rational number plane. This refers to the set of all points (x, y) whose x and y coordinates are rational numbers. In this embodiment of the invention, the multidimensional space is always represented by a two-dimensional rational number plane. Let's take an example to illustrate.

[0039] For any data source to be connected, the data structure or content within that data source can be logically mapped to a two-dimensional rational number plane. The above method converts recorded data or data entities such as files into a two-dimensional rational number plane. Points or regions on the map. The data source to be accessed can be, for example, a database, a file directory, or an API. The x and y coordinates of each mapped point in the two-dimensional rational number plane can represent the data identifier and storage path, respectively.

[0040] By mapping data from at least one data source to a two-dimensional rational number plane 2 This allows for the standardization and abstraction of diverse and heterogeneous data sources into a unified two-dimensional rational number plane. 2 Geometric figures or point sets in a coordinate system are used to complete the "geometry" preprocessing of the data source.

[0041] Among them, the two-dimensional rational number plane As the basis for data mapping, it can overcome the limitations of traditional integer lattice planes. Its limitations. Compared to traditional integer lattice planes. Two-dimensional rational number plane 2It possesses the following key characteristics: 1) Denseness: Between any two different rational numbers, there must exist infinitely many other rational numbers. This ensures the infinite insertion capability of data mapping. Regardless of the distribution of existing data points, the system can always find an unoccupied new rational number coordinate to map new data items or data sources, thus completely avoiding address conflicts and space exhaustion problems that may occur in discrete grid models, laying the foundation for the scalability of the system. 2) No isolated points: Under the standard topology, the neighborhood of any point in the rational number plane contains infinitely many other points. This contrasts sharply with the discrete and isolated structure of integer grid points, making the logical space constructed by this invention essentially "continuous," providing the possibility for more complex spatial partitioning and relational expression, and breaking through the limitation of treating data sources as isolated units in traditional schemes. 3) Countability: Despite its density, the set of rational numbers is still a countably infinite set. This ensures the feasibility of this invention in computer systems. All coordinate points can be enumerated, stored, and computed by the algorithm, ensuring that all mapping and indexing methods proposed in this invention have a solid theoretical foundation and engineering feasibility. 4) Total Order: Rational numbers in each dimension possess a natural, comparable order. This supports efficient searching, sorting, and range queries, enabling efficient operation of index structures based on this plane (such as spatial index trees). Therefore, utilizing the two-dimensional rational number plane... These characteristics allow for flexible construction of data spaces with an unlimited address space.

[0042] Step 102: For each data source, based on the mapping points of the data source in multidimensional space, determine the geometric figure corresponding to the data source. The geometric figure contains all the mapping points corresponding to the data source.

[0043] In this step, for each data source, the boundary of the area occupied by the data source in the multidimensional space can be determined based on the distribution of the mapping points after the data of the data source is mapped to the multidimensional space. This forms a geometric figure that can completely surround all its mapping points. The geometric figure can be the smallest rectangle that can completely surround all its mapping points, or it can be other shapes, such as ellipse or square.

[0044] By defining the geometry corresponding to each data source, complex data sources can be abstracted into a simple object defined by geometric boundaries, rather than managing countless tables or entities within it. This achieves non-intrusive, automated, and highly scalable unified management of massive heterogeneous data sources.

[0045] Step 103: Generate the index coordinates of the data source based on the identification information of the data source and the boundary information of the geometric figure.

[0046] In this step, when the geometric shape is a rectangle or square, the boundary information can be defined by the minimum and maximum x and y coordinates (x_min, x_max, y_min, y_max), where x_min represents the minimum x-coordinate, x_max represents the maximum x-coordinate, y_min represents the minimum y-coordinate, and y_max represents the maximum y-coordinate. When the geometric shape is an ellipse or any other irregular shape, the boundary information is the smallest bounding rectangle of that shape, that is, the smallest rectangle that can completely enclose the ellipse and whose sides are parallel to the coordinate axes, also defined by (x_min, x_max, y_min, y_max).

[0047] Furthermore, the name information n or other unique identifiers of the data source S can be associated with the boundary information of the corresponding geometric shape to form the metadata of the data source S. In this way, a complex data source can be abstracted into a simple, unified metadata object S = {n, (x_min, x_max, y_min, y_max)}. This high level of abstraction encapsulates the identity of the data source and its "placement" in the logical space, which is key to achieving efficient indexing later.

[0048] Furthermore, for the i-th data source in a unified data space that requires the addition of multiple data sources, after obtaining its metadata descriptor S_i = {n_i, (x1_i, x2_i, y1_i, y2_i)}, a unique coordinate z_i will be assigned to the data source S in the third dimension using one or more bijective functions based on the content of S_i. This coordinate is used to index the data source itself, rather than the data content. The third dimension can be, for example, the z-axis.

[0049] Specifically, based on the identification information and geometric boundary information of each data source, a unique and reproducible address can be automatically assigned to each data source, thus avoiding manual configuration and improving the automation and intelligence of data source configuration. Furthermore, metadata that might otherwise require storing massive amounts of discrete entries can be organized into a structured three-dimensional space. The z-axis becomes an efficient data source directory, avoiding metadata explosion and enabling the construction of a scalable metadata structure. Moreover, bijective functions are used to generate the index coordinates of each data source. These bijective functions ensure that no two different data sources are assigned the same z-coordinate, and theoretically, the corresponding data source information can be deduced from the z-coordinate, thus providing a guarantee for subsequent data queries or other precise operations.

[0050] Step 104: Establish the association between index coordinates and data source metadata to build a unified data space for multiple data sources. The data source metadata includes the data source's identification information and boundary information.

[0051] In this step, at the index coordinate z_i of the data source S, the complete metadata descriptor S_i of the data source S is recorded. This metadata contains the unique identification information n_i of the data source and its boundary information (x1_i, x2_i, y1_i, y2_i) on the two-dimensional rational number plane.

[0052] For each data source, a relationship is established between its corresponding index coordinates and metadata, thereby obtaining a unified data space for multiple data sources. In this way, a precise mapping from spatial location to data source entity can be achieved, thus transforming the management problem of massive and heterogeneous data sources into operations on a logical space with a clear structure and efficient indexability.

[0053] By abstracting a complete data source (no matter how complex its internals) into a point in a three-dimensional index space, the scale of metadata is linearly related to the number of data sources, rather than proportional to the number of tables or entities within those data sources. This highly aggregated approach to metadata management fundamentally solves the metadata explosion problem, enabling the system to manage and index hundreds of millions of heterogeneous data sources, thus improving macroscopic scalability.

[0054] The unified data space construction method for multiple data sources provided in this invention maps data from at least one data source to a dense and countable multidimensional space. For each data source, based on the mapping points after the data is mapped in the multidimensional space, the corresponding geometric shape is determined. The geometric shape contains all the mapping points corresponding to the data source. Based on the identification information of the data source and the boundary information of the geometric shape, index coordinates of the data source are generated. The association between the index coordinates and the metadata of the data source is established to construct a unified data space for multiple data sources. The metadata of the data source includes the identification information and boundary information of the data source. Since a logically unified data space can be constructed, all operations are completed at the metadata index level. Therefore, zero-copy access to data can be achieved, fundamentally eliminating the data latency caused by the ETL process and ensuring the real-time performance of data access. Moreover, by using a dense and countable multidimensional space as the mapping basis and generating unique index coordinates for each data source based on its identification information and the boundary information of the geometric shape, the metadata scale of the data source is linearly related only to the number of data sources. Therefore, it can overcome the metadata management bottleneck of the traditional discrete grid model and improve the dynamic scalability when accessing data sources. In addition, unique index coordinates can be automatically generated based on the data source's identification and geometric boundary information, and the association between the index coordinates and metadata can be established. This process does not require manual definition of mapping rules. Therefore, the entire process of data source access, registration and indexing can be automated, which significantly improves the automation level and efficiency of data space construction.

[0055] For example, based on the above embodiments, when generating the index coordinates of the data source based on the identification information of the data source and the boundary information of the geometric figure, it can be achieved in the following way: A bijective function is used to map the identification and boundary information of the data source into index coordinates.

[0056] Specifically, a bijective function can be understood as an invertible function with a one-to-one correspondence between input and output. That is, the bijection property of a bijective function ensures that every combination of inputs is uniquely mapped to an output, and this process is reversible. This is crucial for achieving automated, conflict-free indexing and accurate query and removal operations.

[0057] Figure 2 This is a schematic diagram of the generation of index coordinates provided in an embodiment of the present invention, as shown below. Figure 2As shown, the identifier information n_i of the data source can be mapped conflict-free to a unique first value v using a name encoding function, also known as a bijective function g. This first value v can be a natural number. In the specific implementation, a reversible string encoding method can be used, for example, treating it as a representation in a large number base. This is achieved by determining a character set that contains all characters that might appear in the identifier information of the data source. For example, it can be assumed to be the ASCII character set, with 256 characters. Therefore, a base B = 256 can be chosen.

[0058] Treat the identifier information n_i of the data source as a base-B number. If the string of n_i is c_k c_{k-1}... c_1 c_0, where c_0 is the least significant character, and each character c_j corresponds to an ASCII code value val(c_j) from 0 to 255, then the bijective function can be defined as: g(n_i) = v = Σ (j=0 to k) [ val(c_j) B^j ] That is, g(n_i) = val(c_0) B^0 + val(c_1) B^1 + ... + val(c_k) B^k For example, suppose the data source identifier is n_i = "DB1", and the radix B = 256. The ASCII values ​​are: val('1') = 49, val('B') = 66, and val('D') = 68.

[0059] Then g("DB1") = v = 49 256^0 + 66 256^1 + 68 256^2, thus we can get v equal to 4,473,393. Therefore, g("DB1") uniquely maps to the first value 4,473,393.

[0060] Based on the fundamental principles of number system representation, any positive integer has a unique representation in a base greater than 1. Therefore, the identifier information n_i from different data sources will inevitably yield different values. Thus, using a bijective function, a unique first value v can be assigned to the identifier information of each data source. Furthermore, the bijective function is invertible; therefore, based on the first value v, the original character c_j can be decoded bit by bit by repeatedly performing modulo and integer division operations on the base B, thereby recovering the original string n_i.

[0061] Alternatively, a boundary encoding function, namely a bijective function f, can be used to map the boundary information (x1, x2, y1, y2) represented by four rational numbers to a unique second value u without conflict, where the second value u is a natural number.

[0062] Specifically, an iterative pairing function approach can be used to gradually compress multiple values ​​into a single value. First, each rational number can be represented in its simplest fractional form p / q, where q > 0 and gcd(|p|, q) = 1. In this way, four rational numbers (x1, x2, y1, y2) become eight integers (p1, q1, p2, q2, p3, q3, p4, q4).

[0063] Since pairing functions are usually defined in the natural number field The integer p is on the range (0, 1, 2, ...), and p can be negative. Therefore, it is necessary to first convert the integer p to p. Mapped to Specifically, a standard bijective function `map_Z_to_N(z)` can be used for mapping: if z ≥ 0, then `map_Z_to_N(z) = 2z`, and if z < 0, then `map_Z_to_N(z) = -2z – 1`. This bijective function maps 0 to 0, 1 to 2, -1 to 1, 2 to 4, -2 to 3, and so on, thus achieving... arrive One-to-one correspondence.

[0064] Furthermore, the Cantor pairing function π(k1, k2) can be used to map two natural numbers (k1, k2) to a unique natural number. The Cantor pairing function is defined as: π(k1, k2) = (1 / 2) (k1+ k2) (k1 + k2 + 1) + k2, the function is x arrive A double shot.

[0065] Based on the above, during iterative pairing, the eight integers (p1, q1, p2, q2, p3, q3, p4, q4) can be converted into eight natural numbers (k_p1, ..., k_q4) using the bijective function map_Z_to_N.

[0066] In the first round of pairing, the eight natural numbers can be paired in pairs to obtain four new natural numbers: a1 = π(k_p1, k_q1), a2 = π(k_p2, k_q2), a3 = π(k_p3, k_q3) and a4 = π(k_p4, k_q4).

[0067] In the second round of pairing, the four natural numbers obtained above are paired again to obtain two new natural numbers: b1 = π(a1, a2) and b2 = π(a3, a4).

[0068] In the third round of pairing, the last two natural numbers are paired to obtain the final result u: u = f(x1, x2, y1, y2) = π(b1, b2).

[0069] Using the above method, a unique second value u can be assigned to each set of boundary information for each data source. Since the mapping from integers to natural numbers and the Cantor pairing function are both bijective functions, the composite bijective function f must also be a bijective. Moreover, based on the second value u, b1, b2 can be decoded step by step using the inverse function of the Cantor pairing function, then a1,..., a4, until the eight original natural numbers are finally restored. Then, the inverse function of map_Z_to_N is used to restore them to eight integers, finally obtaining four rational numbers, thus reversibly obtaining the boundary information.

[0070] Furthermore, a combination function, namely a bijective function h, can be used to combine the first value v and the second value u into a final and unique index coordinate z_i.

[0071] In the specific implementation, the Cantor pairing function can be used to pair the first value v and the second value u: z_i = h(u, v) = π(u, v) = (1 / 2) (u + v) (u + v + 1) + v, thus obtaining the index coordinate z_i.

[0072] By using bijective functions, the spatial identity of the data source, represented by boundary information, and the named identity, represented by identification information, can be combined into an absolutely unique index coordinate z_i that can be used as a third-dimensional index.

[0073] It should be noted that if the identification information of the two data sources is different, or the boundary information is different (even if there is only a very small difference in one coordinate), the calculated first value u or the second value v will be different. Since the Cantor pairing function is a bijective function, as long as the (u, v) pairs are different, the output index coordinate z_i will necessarily be different, thus ensuring the uniqueness of the (zi, Si) record and solving the potential address conflict and metadata management problems in the existing technology.

[0074] In the above embodiments, the bijective function allows the heterogeneous information of a data source—its identifier and boundary information—to be deterministically mapped to a unique numerical value z_i through a series of reversible mathematical transformations. This z_i becomes the "address" of the data source in the logical three-dimensional index space. This method requires no manual intervention, avoiding human error and ensuring rapid removal and management of the data source. Furthermore, the entire process of generating index coordinates is fully automated and deterministic, significantly improving data management efficiency and reducing operational costs. Moreover, due to the bijective nature, it guarantees the conflict-free, scalable, and highly operable metadata index even at the scale of hundreds of millions of data sources.

[0075] For example, based on the above embodiments, when determining the geometric figure corresponding to the data source based on the mapping points after the data from the data source is mapped in a multidimensional space, the maximum x-coordinate, minimum x-coordinate, maximum y-coordinate, and minimum y-coordinate of all mapping points in the multidimensional space can be determined based on the distribution of all mapping points in the multidimensional space, and the geometric figure corresponding to the data source can be determined based on the maximum x-coordinate, minimum x-coordinate, maximum y-coordinate, and minimum y-coordinate.

[0076] Specifically, regardless of the complexity, sparseness, or density of the data point distribution within the data source, by calculating its minimum bounding rectangle, a complex set that may contain a massive number of discrete points can be uniformly abstracted into a simple geometric shape that can be fully described by only four coordinate values ​​(x_min, x_max, y_min, y_max). This approach can greatly simplify the calculation of spatial relationships, providing a foundation for subsequent efficient data indexing.

[0077] For example, based on the above embodiments, when mapping data from at least one data source to a multidimensional space with dense and countable properties, it may be possible to map all or part of the data from each data source to the multidimensional space.

[0078] Specifically, data sources can be logically partitioned and managed with fine granularity based on actual business needs and security policies. When it is necessary to map the entire subset of data from a data source to a multi-dimensional space, the data source can be treated as a complete, indivisible logical unit for overall access. The minimum bounding rectangle generated after mapping represents the overall distribution range of all data from the data source.

[0079] Due to business needs, it is also possible to map only specific business tables in a database, a portion of files in a file directory, or streaming data that meets specific filtering conditions.

[0080] When mapping a subset of data from a data source onto a multidimensional space, the geometric shape corresponding to the data source can be determined based on the mapping points after the data from the data source is mapped onto the multidimensional space. This can be done by determining the geometric shape corresponding to the subset of data based on the mapping points after the subset of data is mapped onto the multidimensional space, and then determining the geometric shape corresponding to the subset of data as the geometric shape corresponding to the data source.

[0081] Specifically, after mapping a subset of data, a more accurate and compact minimum bounding rectangle can be independently calculated for this specific data, and a new, independent index coordinate can be generated accordingly.

[0082] By employing the above methods, different subsets of data from the same data source can be accessed as multiple independent logical units into a unified data space. This improves the granularity and flexibility of data source access, provides strong native support for data views, on-demand exposure, incremental access, and more granular access control, meets complex business needs, and greatly enhances the system's practicality and flexibility.

[0083] Furthermore, based on the above embodiments, data queries can also be performed from a unified data space of multiple data sources. For example, in response to a data query request, the query coordinates corresponding to the data to be queried are determined; a query region is constructed based on the query coordinates; target metadata is determined based on the query region and the unified data space; and the data query request is distributed to the target data source corresponding to the identification information in the target metadata to perform the data query operation.

[0084] In determining target metadata based on the query region and the unified data space, a spatial indexing algorithm can be used. After determining the target index coordinates corresponding to at least one target geometry that intersects with the query region, the target metadata corresponding to the target index coordinates can be obtained from the unified data space.

[0085] Specifically, Figure 3 This is a schematic diagram of data query provided in an embodiment of the present invention, such as... Figure 3As shown, after receiving a data query request, the query coordinate point (x, y) corresponding to the data to be queried can be generated based on the identifier and storage path of the data to be queried contained in the data query request. The query region Q is determined based on the query coordinate point (x, y), and the query region Q represents the logical space range to be covered by this query.

[0086] In one possible implementation, a query region can be constructed using the query coordinate point as the center point and based on a preset neighborhood range. Assuming the preset neighborhood range is delta, a constructed rectangular query region Q would be (x-delta, x+delta, y-delta, y+delta), where x-delta represents the minimum value of query region Q on the horizontal axis, x+delta represents the maximum value of query region Q on the horizontal axis, y-delta represents the minimum value of query region Q on the vertical axis, and y+delta represents the maximum value of query region Q on the horizontal axis. Constructing the query region using the query coordinate point and the preset neighborhood range not only effectively controls the coverage of each query, avoiding the introduction of too many irrelevant data sources due to an excessively large query range, thus significantly improving query accuracy and execution efficiency, but also allows for flexible adjustment of query granularity according to business needs, improving the flexibility of data querying.

[0087] After constructing the query region, it's not necessary to traverse all data sources. Instead, the existing 3D data space is utilized, and efficient spatial indexing algorithms, such as R-trees or z-axis indexes, are employed to quickly perform geometric intersection checks. Taking R-trees as an example, this index manages the geometry of all data sources as leaf nodes. By comparing the spatial relationship between the query region and these geometry elements, all intersecting geometry elements can be quickly identified, and their corresponding unique index coordinates (z-coordinates) can be obtained. This enables efficient query pruning, avoiding the need to query data from all data sources. For z-axis indexes, multidimensional spatial coordinates can be continuously mapped onto the one-dimensional z-axis using space-filling curves (such as Z-order curves or Hilbert curves). After constructing the query region, its coverage area in the two-dimensional rational number plane is also mapped onto the z-axis, transforming it into one or more continuous z-coordinate intervals. By efficiently scanning the ordered z-coordinate indexes within these intervals, the target index coordinates corresponding to at least one target geometry intersecting the query region can be quickly identified.

[0088] Furthermore, based on these target index coordinates, a precise search is performed within the unified data space to determine the target metadata corresponding to the target index coordinates. This allows the received data query requests to be distributed to the target data source corresponding to the identification information included in the target metadata, enabling the execution of data query operations. The results are then aggregated and returned. For example... Figure 3 As shown, if the geometry DS1' of data source 1 is found to intersect with the query region Q, the data query request can be distributed only to the identified data source 1 (DS1) to perform the data query operation, and other data sources DS2 and DS3 that are not related to the query region Q will be ignored.

[0089] In this embodiment, the global cross-source data query problem can be transformed into an efficient local geometric intersection problem. Based on the query coordinates of the query request, it can quickly filter out the few relevant data sources that intersect with its boundary rectangle, thereby achieving accurate query routing. Rapid index pruning avoids sending data query requests to a large number of irrelevant data sources, reducing the complexity of the global query to local location. This significantly reduces unnecessary network I / O and the computational load on downstream data sources, improving query efficiency and performance, and achieving the goal of efficiently constructing and manipulating a unified data space.

[0090] Furthermore, based on the above embodiments, data source removal operations can also be flexibly performed from the unified data space. For example, by receiving a data source deletion request, which includes the identification information of the data source to be deleted and the boundary information of the geometric shape corresponding to the data source to be deleted, the index coordinates of the data source to be deleted are generated based on the identification information of the data source to be deleted and the boundary information of the geometric shape corresponding to the data source to be deleted, and the index coordinates and associated metadata of the data source to be deleted are deleted from the unified data space.

[0091] Specifically, when a data source needs to be removed, a data source deletion request can be triggered, including the identifier information n of the data source to be deleted and the boundary information (x1, x2, y1, y2) of the geometry corresponding to the data source to be deleted. Based on the method described in the foregoing embodiments, using the same bijective functions f, g, and h, the unique index coordinate z of the data source to be deleted is recalculated based on the identifier information and the boundary information of the geometry corresponding to the data source to be deleted, and the index coordinate z and the metadata associated with the index coordinate z are directly deleted in the unified data space.

[0092] In this embodiment, when deleting a data source, it is only necessary to recalculate the index coordinates of the data source to be deleted and delete the index coordinates and associated metadata of the data source to be deleted from the unified data space. This achieves fast and accurate removal with O(1) or O(log N) complexity. The whole process does not affect the index and status of any other data source, ensuring the high dynamism and stability of the system.

[0093] Figure 4 This is the second flowchart illustrating the method for constructing a unified data space from multiple data sources provided in this embodiment of the invention. Figure 4As shown, when multiple heterogeneous data sources exist, such as a database (DS1), a file system (DS2), and an API interface (DS3), this embodiment of the invention does not perform physical data migration. Instead, it uses a logical mapping step to logically map the content of each data source (DS1, DS2, DS3) to a two-dimensional rational number plane. The algorithm calculates a minimum bounding rectangle (B1, B2, B3) for each mapped data source, which is a geometric bounding box. This bounding rectangle represents the "occupancy" of the data source on the logical plane.

[0094] Furthermore, in the dimension-upgrading indexing operation, a unique index coordinate (z1, z2, z3) can be assigned to the boundary rectangle of each data source on an additional third logical dimension (z-axis), ultimately forming a logical three-dimensional metadata index space, where each point on the z-axis precisely indexes a data source boundary rectangle on the XY plane. This model differs from the discrete grid model of existing technologies, possessing both density and infinite scalability.

[0095] This invention constructs a logically unified data space where all operations are completed at the metadata index level, eliminating the need for any physical extraction, transformation, and loading (ETL) of the source data. This significantly reduces the high storage costs and computational resource consumption associated with data replication. Furthermore, because queries are ultimately directed to the real-time data source itself, users always access the most up-to-date and accurate data, achieving "zero-copy" data integration. This fundamentally reduces costs and ensures maximum data timeliness, completely eliminating the inherent data latency issues in traditional data warehouse and data lake solutions, and powerfully supporting business scenarios with extremely high real-time requirements.

[0096] The unified data space construction apparatus for multiple data sources provided by the present invention will be described below. The unified data space construction apparatus for multiple data sources described below can be referred to in correspondence with the unified data space construction method for multiple data sources described above.

[0097] Figure 5 This is a schematic diagram of the structure of the unified data space construction device for multiple data sources provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the unified data space construction device 500 for multiple data sources includes: Mapping module 11 is used to map data from at least one data source to a multidimensional space with dense and countable properties; The determining module 12 is used to determine the geometric figure corresponding to each data source based on the mapping points of the data source after mapping on the multidimensional space, wherein the geometric figure contains all the mapping points corresponding to the data source. The generation module 13 is used to generate the index coordinates of the data source based on the identification information of the data source and the boundary information of the geometric figure; The module 14 is used to establish the association between the index coordinates and the metadata of the data source, so as to construct a unified data space for multiple data sources. The metadata of the data source includes the identification information of the data source and the boundary information.

[0098] In one example embodiment, the multidimensional space includes a two-dimensional rational number plane.

[0099] In one example embodiment, the generation module is specifically used for: A bijective function is used to map the identification information of the data source and the boundary information to the index coordinates.

[0100] In one example embodiment, the determining module is specifically used for: Based on the distribution of all the mapping points in the multidimensional space, determine the maximum x-coordinate, minimum x-coordinate, maximum y-coordinate, and minimum y-coordinate of all the mapping points in the multidimensional space. Based on the maximum x-coordinate, the minimum x-coordinate, the maximum y-coordinate, and the minimum y-coordinate, the geometric shape corresponding to the data source is determined.

[0101] In one example embodiment, the apparatus further includes a construction module, an acquisition module, and a sending module, wherein: The determination module 12 is also used to determine the query coordinate point corresponding to the data to be queried in response to the data query request; The construction module is used to construct the query region based on a preset neighborhood range, with the query coordinate point as the center point; The determining module 12 is further configured to determine target metadata based on the query region and the unified data space; The sending module is used to distribute the data query request to the target data source corresponding to the identification information in the target metadata in order to perform the data query operation.

[0102] In one example embodiment, the apparatus further includes an acquisition module, wherein: The determining module 12 is further configured to use a spatial indexing algorithm to determine the target index coordinates corresponding to at least one target geometry that intersects with the query region; The acquisition module is used to acquire the target metadata corresponding to the target index coordinates from the unified data space.

[0103] In one example embodiment, the apparatus further includes a receiving module and a deleting module, wherein: A receiving module is used to receive a data source deletion request, wherein the data source deletion request includes the identification information of the data source to be deleted and the boundary information of the geometric shape corresponding to the data source to be deleted; The generation module 13 is also used to generate the index coordinates of the data source to be deleted based on the identification information of the data source to be deleted and the boundary information of the geometric figure corresponding to the data source to be deleted; The deletion module is used to delete the index coordinates and associated metadata of the data source to be deleted from the unified data space.

[0104] The apparatus of this embodiment can be used in any of the methods in the side embodiment of the method for constructing a unified data space from multiple data sources. Its specific implementation process and technical effects are similar to those in the side embodiment of the method for constructing a unified data space from multiple data sources. For details, please refer to the detailed description in the side embodiment of the method for constructing a unified data space from multiple data sources, which will not be repeated here.

[0105] Figure 6 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 6 As shown, the electronic device may include a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, communications interface 620, and memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a method for constructing a unified data space from multiple data sources. This method includes: mapping data from at least one data source to a multi-dimensional space with dense and countable properties; for each data source, determining a geometric figure corresponding to the data source based on the mapping points of the data source mapped onto the multi-dimensional space, wherein the geometric figure contains all the mapping points corresponding to the data source; generating index coordinates of the data source based on the identification information of the data source and the boundary information of the geometric figure; and establishing an association between the index coordinates and the metadata of the data source to construct a unified data space from multiple data sources, wherein the metadata of the data source includes the identification information of the data source and the boundary information.

[0106] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the unified data space construction method for multiple data sources provided by the above methods. The method includes: mapping data from at least one data source to a multi-dimensional space with dense and countable properties; for each data source, determining a geometric figure corresponding to the data source based on the mapping points of the data source mapped in the multi-dimensional space, wherein the geometric figure contains all the mapping points corresponding to the data source; generating index coordinates of the data source based on the identification information of the data source and the boundary information of the geometric figure; and establishing an association between the index coordinates and the metadata of the data source to construct a unified data space for multiple data sources, wherein the metadata of the data source includes the identification information of the data source and the boundary information.

[0108] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a method for constructing a unified data space for multiple data sources provided by the methods described above. This method includes: mapping data from at least one data source to a multi-dimensional space possessing dense and countable properties; for each data source, determining a geometric figure corresponding to the data source based on the mapping points of the data source mapped onto the multi-dimensional space, the geometric figure containing all mapping points corresponding to the data source; generating index coordinates of the data source based on the identification information of the data source and the boundary information of the geometric figure; and establishing an association between the index coordinates and the metadata of the data source to construct a unified data space for multiple data sources, the metadata of the data source including the identification information of the data source and the boundary information.

[0109] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0110] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a unified data space from multiple data sources, characterized in that, include: Map data from at least one data source to a multidimensional space that is dense and countable; For each of the aforementioned data sources, based on the mapping points of the data from the data source mapped onto the multidimensional space, the geometric figure corresponding to the data source is determined, and the geometric figure contains all the mapping points corresponding to the data source. Based on the identification information of the data source and the boundary information of the geometric figure, the index coordinates of the data source are generated; Establish the association between the index coordinates and the metadata of the data source to construct a unified data space for multiple data sources. The metadata of the data source includes the identification information of the data source and the boundary information.

2. The method for constructing a unified data space from multiple data sources according to claim 1, characterized in that, The multidimensional space includes a two-dimensional rational number plane.

3. The method for constructing a unified data space from multiple data sources according to claim 1, characterized in that, The step of generating the index coordinates of the data source based on the identification information of the data source and the boundary information of the geometric figure includes: A bijective function is used to map the identification information of the data source and the boundary information to the index coordinates.

4. The method for constructing a unified data space from multiple data sources according to claim 1, characterized in that, The mapping points obtained by mapping the data from the data source onto the multidimensional space, and the determination of the geometric shape corresponding to the data source, include: Based on the distribution of all the mapping points in the multidimensional space, determine the maximum x-coordinate, minimum x-coordinate, maximum y-coordinate, and minimum y-coordinate of all the mapping points in the multidimensional space. Based on the maximum x-coordinate, the minimum x-coordinate, the maximum y-coordinate, and the minimum y-coordinate, the geometric shape corresponding to the data source is determined.

5. The method for constructing a unified data space from multiple data sources according to any one of claims 1-4, characterized in that, The method further includes: In response to a data query request, determine the query coordinates corresponding to the data to be queried; A query region is constructed based on a preset neighborhood range, with the query coordinate point as the center point; Based on the query region and the unified data space, the target metadata is determined; The data query request is distributed to the target data source corresponding to the identification information in the target metadata to perform the data query operation.

6. The method for constructing a unified data space from multiple data sources according to claim 5, characterized in that, The determination of target metadata based on the query region and the unified data space includes: A spatial indexing algorithm is used to determine the target index coordinates corresponding to at least one target geometric figure that intersects with the query region; Obtain the target metadata corresponding to the target index coordinates from the unified data space.

7. The method for constructing a unified data space from multiple data sources according to any one of claims 1-4, characterized in that, The method further includes: Receive a data source deletion request, the data source deletion request including the identification information of the data source to be deleted and the boundary information of the geometric figure corresponding to the data source to be deleted; Based on the identification information of the data source to be deleted and the boundary information of the geometric shape corresponding to the data source to be deleted, the index coordinates of the data source to be deleted are generated. Remove the index coordinates and associated metadata of the data source to be deleted from the unified data space.

8. A unified data space construction device with multiple data sources, characterized in that, include: The mapping module is used to map data from at least one data source to a multidimensional space that has dense and countable properties. The determination module is used to determine the geometric figure corresponding to each data source based on the mapping points of the data source after mapping on the multidimensional space, wherein the geometric figure contains all the mapping points corresponding to the data source. The generation module is used to generate the index coordinates of the data source based on the identification information of the data source and the boundary information of the geometric figure; A module is established to establish the association between the index coordinates and the metadata of the data source, so as to construct a unified data space for multiple data sources. The metadata of the data source includes the identification information of the data source and the boundary information.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the unified data space construction method for multiple data sources as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the unified data space construction method for multiple data sources as described in any one of claims 1 to 7.

Images