Distributed data virtualization system and method, computer equipment and storage medium
By adopting peer-to-peer network structure and distributed hashing algorithms in the distributed data virtualization system, the problem of low data usage efficiency under the centralized management architecture is solved, and more efficient data processing and access is achieved.
Patent Information
- Application Number
- CN202411481240.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-23
- Publication Date
- 2024-11-22
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
When processing multivariate heterogeneous data sources, existing distributed data virtualization systems adopt a system architecture of centralized management and centralized data processing, which affects users' efficiency of data use.
A group is composed of peer-to-peer network structure, each group corresponds to a data table, including data service processing module, peer-to-peer network communication module, virtual view SQL engine, metadata management module and data cache module, through distributed hashing and Consistent hashing for data storage and communication, realizing distributed processing and caching of data.
The efficiency of the use of distributed data virtualization systems has been improved, and the speed and efficiency of data access have been improved through distributed processing and caching technology.
Smart Images

Figure CN119003479A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing, and in particular to a distributed data virtualization system, method, computer equipment and storage medium. Background Art
[0002] Data virtualization adds a data virtualization layer to the original system or software framework, abstracts or maps all data sources, and becomes a unified field-oriented data access layer, shielding the diversity and complexity of multiple heterogeneous data sources such as traditional databases, distributed clouds, and big data, allowing users to access all data as if they were accessing a data table from a single data source. It allows real-time viewing and analysis without migrating data, copying data, processing data through ETL, or requiring additional storage. This provides convenience in data processing, can bring real-time and insightful results to enterprises, and enables faster response and decision-making.
[0003] There are already multi-source data virtualization methods that are used to uniformly access data from various data sources, accelerating data by using cache instead of loading all the data. For distributed data sources, these methods all use a system architecture of centralized management and centralized data processing, which to some extent affects the efficiency of users' use of data. Summary of the invention
[0004] Embodiments of the present invention provide a distributed data virtualization system, method, computer device and storage medium to improve the efficiency of using distributed data virtualization.
[0005] In order to solve the above technical problems, the embodiment of the present application provides a distributed data virtualization system, wherein the distributed data virtualization system adopts a peer-to-peer network structure to form groups, each group corresponds to a data table, and the system includes n peer-to-peer network nodes, wherein the set of peer-to-peer network nodes is represented as {node i |i=1,…,n}, each peer-to-peer network node includes a data service processing module, a peer-to-peer network communication module, a virtual view SQL engine, a metadata management module, a data cache module and a data ETL module; wherein, The data service processing module is used to receive data query and statistical requests, and is also used to combine the data obtained after processing by the virtual view SQL engine module into a virtual view; The peer-to-peer network communication module is used to communicate with other peer network nodes, complete data storage through distributed hashing and consistent hashing, and store the node status of other peer network nodes at the same time; The virtual view SQL engine is used to parse SQL statements and construct sub-query SQL of each data source according to metadata relationships; The metadata management module is used to implement metadata configuration of the data virtual view according to user settings, and manage and maintain metadata relationships; The data cache module caches the query results. The data ETL module extracts data according to the sub-query SQL.
[0006] Optionally, the data service processing module includes: A request receiving unit, used to receive a user virtual view query request, parse the request parameters, and forward the request to a module corresponding to the request type for processing according to the request type; The data collection unit is used to collect the data of the node group fed back by the task of the virtual view SQL engine and the data of other node groups fed back by the peer network communication module to form a virtual view and return it to the user.
[0007] Optionally, the virtual view SQL engine includes: A query SQL parsing unit, used to convert the virtual view SQL query statement into an SQL query statement executed on each physical source; The task management unit is used to manage data extraction tasks.
[0008] Optionally, the metadata management module includes: A metadata storage unit, which realizes the storage of metadata information; A metadata relationship configuration unit, which forms a metadata relationship tree according to a user metadata relationship graph configuration request; The metadata relationship graph query unit is used to respond to metadata relationship queries of the virtual view SQL engine.
[0009] Optionally, the data cache module includes: A persistent cache unit, which caches the data request results of the data source through a relational database; A cache update unit, used to update cache data according to the returned data results obtained by the data ETL module; The cache query unit is used to respond to the data reading request of the virtual view SQL engine and return the data in the cache table.
[0010] In order to solve the above technical problems, the embodiment of the present application further provides a distributed data virtualization method, including: Receive a virtual view query request sent by a user terminal, and parse the virtual view query request to obtain a virtual view SQL query statement; Forward user virtual view query requests to all groups in the network; In each node of the group, compare the SQL query statement with the cached SQL to obtain the comparison result; If the comparison result is a miss, the virtual view SQL query statement is converted into an SQL query statement executed on each physical source according to the virtual view query request, data is requested on the data source, and the cache is updated; If the comparison result is a hit, the cache is returned, and the query result is returned to the requesting node. The requesting node aggregates all returned data to form a virtual view and returns it to the user.
[0011] Optionally, if the comparison result is a miss, converting the virtual view SQL query statement into an SQL query statement executed on each physical source according to the virtual view query request, requesting data on the data source, and updating the cache includes: According to the metadata information, the tables and fields in the virtual table are converted accordingly to form SQL statements executed by the data source; Execute the parsed SQL statement on the data source and obtain the query results; Traverse the query results, calculate the hash value of each result, and calculate the storage location of each result based on the consistent hashing algorithm. If it is not stored on this node, transfer the cached data to the corresponding peer network node; After receiving the data cache entry on a peer network node basis, the data is stored in the local node; If it is stored on this node, the metadata information is obtained, and the cache entry is converted into a virtual table entry according to the metadata information, and then the cache is updated.
[0012] Optionally, if the comparison result is a hit, returning the cache, returning the query result to the requesting node, and the requesting node collecting all returned data to form a virtual view to return to the user include: Return cached data and send cached data requests to other nodes in the group; Receive cache data fed back by each node in the group; The cached data is aggregated according to the metadata information to form a virtual view and return it to the user.
[0013] Optionally, before forwarding the user virtual view query request to all groups in the network, the distributed data virtualization method further includes: Get the number of nodes K and K different hash functions contained in each pre-set group {F i |i=1,2,...k}; For the data table collection {T j |j=1,2,...l}, using an improved consistent hashing algorithm to perform group division to obtain the group, wherein the data table set includes a unique identification code UUID of each data table; The data table set {T j |j=1,2,...l}, an improved consistent hashing algorithm is used to divide the groups, and the obtained groups include: Map the range of the hash function into a m The closed loop space of the buckets is 0 to 2 m-1 The digital space of , where m is a natural number; Use hash function F1 to calculate the hash value of each peer node based on the unique identification code UUID of the peer node , where 0≤h≤J, J is the number of peer-to-peer network nodes, is an m-bit binary number falling into 2 m Bucket space.
[0014] Each node is based on the hash value of the peer network node Sort them to form a peer-to-peer network node sequence queue H; Use hash function F1 for each data table T j Calculating the hash value , is a binary number of m digits; Check the elements in queue H one by one. If the data table T j Hash value Comply with the relationship , then node h+1 Divide into T j For the corresponding group, if the node h+1 Already in T j In the corresponding group, the subsequent nodes in queue H are checked in sequence until the nodes that are not in the group are divided into T j Corresponding group; Use hash function F in turn i The above steps are executed to obtain the group.
[0015] In order to solve the above technical problems, an embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above distributed data virtualization method when executing the computer program.
[0016] In order to solve the above technical problems, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above distributed data virtualization method are implemented.
[0017] The distributed data virtualization system, method, computer device and storage medium provided by the embodiment of the present invention receive a virtual view query request sent by a user terminal, parse the virtual view query request, obtain a virtual view SQL query statement, forward the user virtual view query request to all groups in the network, compare the SQL query statement with the cached SQL in each node of the group, obtain a comparison result, if the comparison result is a miss, then according to the virtual view query request, convert the virtual view SQL query statement into an SQL query statement executed on each physical source, request data on the data source, and update the cache, if the comparison result is a hit, return the cache, return the query result to the requesting node, the requesting node aggregates all returned data, and forms a virtual view to return to the user. The use efficiency of distributed data is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0019] Figure 1 It is a schematic diagram of metadata information of this application; Figure 2 is a flow chart of an embodiment of the distributed data virtualization method of the present application; Figure 3 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0021] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0023] In order to solve the above technical problems, the embodiment of the present application also provides a distributed data virtualization system, which adopts a peer-to-peer network structure to form groups, each group corresponds to a data table, and the system includes n peer-to-peer network nodes, and the set of peer-to-peer network nodes is represented by {node i |i=1,…,n}, each peer-to-peer network node includes a data service processing module, a peer-to-peer network communication module, a virtual view SQL engine, a metadata management module, a data cache module and a data ETL module; wherein, The data service processing module is used to receive data query and statistical requests, and is also used to combine the data obtained after processing by the virtual view SQL engine module into a virtual view; The peer-to-peer network communication module is used to communicate with other peer network nodes, complete data storage through distributed hashing and consistent hashing, and store the node status of other peer network nodes at the same time; The virtual view SQL engine is used to parse SQL statements and construct sub-query SQL of each data source and / or other data reading parameters according to metadata relationships; The metadata management module is used to implement metadata configuration of the data virtual view according to user settings, and manage and maintain metadata relationships; The data cache module caches the query results. The data ETL module extracts data according to sub-query SQL or other data reading parameters.
[0024] Optionally, the data service processing module includes: A request receiving unit, used to receive a user virtual view query request, parse the request parameters, and forward the request to different modules for processing according to the request type; The data collection unit is used to collect the data of the node group fed back by the task of the virtual view SQL engine and the data of other node groups fed back by the peer network communication module to form a virtual view and return it to the user.
[0025] The metadata management module organizes metadata relationships in a tree-like manner. Each peer-to-peer network node contains the same metadata information. The virtual data table is the root of the tree, and its attributes include the data table name, etc. The next layer is the data fields contained in the virtual data table, and its attributes include field name, field type, etc. The leaves of the tree are the data table fields corresponding to the data fields, and their attributes include field name, field type, field table name, field database name, etc. Due to the use of a distributed structure, the metadata held by the peer-to-peer network node only contains the metadata of the data table corresponding to the group where the node is located.
[0026] See also Figure 1 , Figure 1 This is a specific schematic diagram of the metadata information of this application, such as Figure 1 As shown, Figure 1 They include metadata logic information and metadata information held by each node, among which: Figure 1 (a) is a schematic diagram of metadata logic information. Figure 1 In (a), VT is the table name of the data table, id, a, b, c, ..., p are the attribute fields contained in the data table, each attribute field corresponds to at least one data field, and the attributes include the field name, field type, the table name where the field is located, and the database name where the field is located. For example, the id attribute field includes three data fields, the first data field includes the data source table S1, table T1, and id, the second data field includes the data source table S1, table T2, and id, and the third data field includes the data source table S2, table T1, and id. Similarly, the a attribute field includes two data fields. To avoid repetition, they are not described here one by one; Figure 1 (b) is the source data information held by the data source S1 table T1 group, Figure 1 (c) is the source data information held by the data source S1 table T2 group, Figure 1 (d) is the source data information held by the data source S2 table T1 group.
[0027] Optionally, the virtual view SQL engine includes: A query SQL parsing unit, used to convert the virtual view SQL query statement into an SQL query statement executed on each physical source; The task management unit is used to manage data extraction tasks.
[0028] Optionally, the metadata management module includes: A metadata storage unit, which realizes the storage of metadata information; A metadata relationship configuration unit, which forms a metadata relationship tree according to a user metadata relationship graph configuration request; The metadata relationship graph query unit is used to respond to metadata relationship queries of the virtual view SQL engine.
[0029] Optionally, the data cache module includes: A persistent cache unit, which caches the data request results of the data source through a relational database; A cache update unit, used to update cache data according to the returned data results obtained by the data ETL module; The cache query unit is used to respond to the data reading request of the virtual view SQL engine and return the data in the cache table.
[0030] See also Figure 2 , Figure 2 A distributed data virtualization method provided by an embodiment of the present invention is shown. Figure 1 The server in is used as an example to illustrate the details as follows: S201: receiving a virtual view query request sent by a user terminal, and parsing the virtual view query request to obtain a virtual view SQL query statement; S202: forwarding the user virtual view query request to all groups in the network; Optionally, before forwarding the user virtual view query request to all groups in the network, the distributed data virtualization method further includes: Get the number of nodes K and K different hash functions contained in each pre-set group {F i |i=1,2,...k}; For the data table collection {T j |j=1,2,...l}, using an improved consistent hashing algorithm to perform group division to obtain the group, wherein the data table set includes a unique identification code UUID of each data table; The data table set {T j |j=1,2,...l}, an improved consistent hashing algorithm is used to divide the groups, and the obtained groups include: Map the range of the hash function into a m The closed loop space of the buckets is 0 to 2 m-1 The digital space of , where m is a natural number; Use hash function F1 to calculate the hash value of each peer node based on the unique identification code UUID of the peer node , where 0≤h≤J, J is the number of peer-to-peer network nodes, is an m-bit binary number falling into 2m Bucket space.
[0031] Each node is based on the hash value of the peer network node Sort them to form a peer-to-peer network node sequence queue H; Use hash function F1 for each data table T j Calculating the hash value , is a binary number of m digits; Check the elements in queue H one by one. If the data table T j Hash value Comply with the relationship , then node h+1 Divide into T j For the corresponding group, if the node h+1 Already in T j In the corresponding group, the subsequent nodes in queue H are checked in sequence until the nodes that are not in the group are divided into T j Corresponding group.
[0032] Among them, the closed loop space is 0 to 2 m-1 The numbers in the digital space are connected end to end to form a closed ring.
[0033] S203: In each node of the group, compare the SQL query statement with the cached SQL to obtain a comparison result; S204: If the comparison result is a miss, the virtual view SQL query statement is converted into an SQL query statement executed on each physical source according to the virtual view query request, data is requested on the data source, and the cache is updated; Optionally, if the comparison result is a miss, converting the virtual view SQL query statement into an SQL query statement executed on each physical source according to the virtual view query request, requesting data on the data source, and updating the cache includes: According to the metadata information, the tables and fields in the virtual table are converted accordingly to form SQL statements executed by the data source; Execute the parsed SQL statement on the data source and obtain the query results; Traverse the query results, calculate the hash value of each result, and calculate the storage location of each result based on the consistent hashing algorithm. If it is not stored on this node, transfer the cached data to the corresponding peer network node; After receiving the data cache entry on a peer network node basis, the data is stored in the local node; If it is stored on this node, the metadata information is obtained, and the cache entry is converted into a virtual table entry according to the metadata information, and then the cache is updated.
[0034] S205: If the comparison result is a hit, the cache is returned, and the query result is returned to the requesting node. The requesting node aggregates all returned data to form a virtual view and returns it to the user.
[0035] Optionally, if the comparison result is a hit, returning the cache, returning the query result to the requesting node, and the requesting node collecting all returned data to form a virtual view to return to the user include: Return cached data and send cached data requests to other nodes in the group; Receive cache data fed back by each node in the group; The cached data is aggregated according to the metadata information to form a virtual view and return it to the user.
[0036] In this embodiment, a virtual view query request sent by a user terminal is received, and the virtual view query request is parsed to obtain a virtual view SQL query statement, and the user virtual view query request is forwarded to all groups in the network. In each node of the group, the SQL query statement is compared with the cached SQL to obtain a comparison result. If the comparison result is a miss, the virtual view SQL query statement is converted into an SQL query statement executed on each physical source according to the virtual view query request, data is requested on the data source, and the cache is updated. If the comparison result is a hit, the cache is returned, and the query result is returned to the requesting node. The requesting node aggregates all returned data to form a virtual view and return it to the user. The use efficiency of distributed data is improved.
[0037] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.
[0038] For the specific definition of the distributed data virtualization device, please refer to the definition of the distributed data virtualization method above, which will not be repeated here. Each module in the above-mentioned distributed data virtualization device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.
[0039] To solve the above technical problems, the present application also provides a computer device. Figure 3 , Figure 3 This is a basic structural block diagram of the computer device in this embodiment.
[0040] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components connected to the memory 41, the processor 42, and the network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.
[0041] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.
[0042] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or D interface display memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as the program code for data repair based on machine learning in a distributed environment. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.
[0043] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the program code stored in the memory 41 or process data, such as running the program code for data repair based on machine learning in a distributed environment.
[0044] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.
[0045] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores an interface display program, and the interface display program can be executed by at least one processor so that the at least one processor performs the steps of the distributed data virtualization method as described above.
[0046] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0047] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to replace some of the technical features therein with equivalents. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. A distributed data virtualization system, characterized in that: The distributed data virtualization system adopts a peer-to-peer network structure to form groups, each group corresponds to a data table, and the system includes n peer-to-peer network nodes, and the set of peer-to-peer network nodes is represented by {node i |i=1,…,n}, each peer-to-peer network node includes a data service processing module, a peer-to-peer network communication module, a virtual view SQL engine, a metadata management module, a data cache module and a data ETL module; wherein, The data service processing module is used to receive data query and statistical requests, and is also used to combine the data obtained after processing by the virtual view SQL engine module into a virtual view; The peer-to-peer network communication module is used to communicate with other peer network nodes, complete data storage through distributed hashing and consistent hashing, and store the node status of other peer network nodes at the same time; The virtual view SQL engine is used to parse SQL statements and construct sub-query SQL of each data source according to metadata relationships; The metadata management module is used to implement metadata configuration of the data virtual view according to user settings, and manage and maintain metadata relationships; The data cache module caches the data of each query result; The data ETL module extracts data according to the sub-query SQL.
2. The distributed data virtualization system according to claim 1, characterized in that: The data service processing module includes: A request receiving unit, used to receive a user virtual view query request, parse the request parameters, and forward the request to a module corresponding to the request type for processing according to the request type; The data collection unit is used to collect the data of the node group fed back by the task of the virtual view SQL engine and the data of other node groups fed back by the peer network communication module to form a virtual view and return it to the user.
3. The distributed data virtualization system according to claim 1, characterized in that: The virtual view SQL engine includes: A query SQL parsing unit, used to convert the virtual view SQL query statement into an SQL query statement executed on each physical source; The task management unit is used to manage data extraction tasks.
4. The distributed data virtualization system according to claim 1, characterized in that: The metadata management module includes: A metadata storage unit, which realizes the storage of metadata information; A metadata relationship configuration unit, which forms a metadata relationship tree according to a user metadata relationship graph configuration request; The metadata relationship graph query unit is used to respond to metadata relationship queries of the virtual view SQL engine.
5. The distributed data virtualization system according to claim 1, characterized in that: The data cache module comprises: A persistent cache unit, which caches the data request results of the data source through a relational database; A cache update unit, used to update cache data according to the returned data results obtained by the data ETL module; The cache query unit is used to respond to the data reading request of the virtual view SQL engine and return the data in the cache table.
6. A distributed data virtualization method, characterized in that: include: Receive a virtual view query request sent by a user terminal, and parse the virtual view query request to obtain a virtual view SQL query statement; Forward user virtual view query requests to all groups in the network; In each node of the group, compare the SQL query statement with the cached SQL to obtain the comparison result; If the comparison result is a miss, the virtual view SQL query statement is converted into an SQL query statement executed on each physical source according to the virtual view query request, data is requested on the data source, and the cache is updated; If the comparison result is a hit, the cache is returned, and the query result is returned to the requesting node. The requesting node aggregates all returned data to form a virtual view and returns it to the user.
7. The distributed data virtualization method according to claim 6, characterized in that: If the comparison result is a miss, converting the virtual view SQL query statement into an SQL query statement executed on each physical source according to the virtual view query request, requesting data on the data source, and updating the cache includes: According to the metadata information, the tables and fields in the virtual table are converted accordingly to form SQL statements executed by the data source; Execute the parsed SQL statement on the data source and obtain the query results; Traverse the query results, calculate the hash value of each result, and calculate the storage location of each result based on the consistent hashing algorithm. If it is not stored on this node, transfer the cached data to the corresponding peer network node; After receiving the data cache entry on a peer network node basis, the data is stored in the local node; If it is stored on this node, the metadata information is obtained, and the cache entry is converted into a virtual table entry according to the metadata information, and then the cache is updated.
8. The distributed data virtualization method according to claim 6, characterized in that: Before forwarding the user virtual view query request to all groups in the network, the distributed data virtualization method further includes: Get the number of nodes K and K different hash functions contained in each pre-set group {F i |i=1,2,...k}; For the data table collection {T j |j=1,2,...l}, using an improved consistent hashing algorithm to perform group division to obtain the group, wherein the data table set includes a unique identification code UUID of each data table; The data table set {T j |j=1,2,...l}, an improved consistent hashing algorithm is used to divide the groups, and the obtained groups include: Map the range of the hash function into a m The closed loop space of the buckets is 0 to 2 m-1 The digital space of , where m is a natural number; Use hash function F1 to calculate the hash value of each peer node based on the unique identification code UUID of the peer node , where 0≤h≤J, J is the number of peer-to-peer network nodes, is an m-bit binary number falling into 2 m In the bucket space; Each node is based on the hash value of the peer network node Sort them to form a peer-to-peer network node sequence queue H; Use hash function F1 for each data table T j Calculating the hash value , is a binary number of m digits; Check the elements in queue H one by one. If the data table T j Hash value Comply with the relationship , then node h+1 Divide into T j For the corresponding group, if the node h+1 Already in T j In the corresponding group, the subsequent nodes in queue H are checked in sequence until the nodes that are not in the group are divided into T j Corresponding group; Use hash function F in turn i By executing the above steps, the group consisting of a peer-to-peer network structure is obtained.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the distributed data virtualization method according to any one of claims 6 to 8 is implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the distributed data virtualization method according to any one of claims 6 to 8 is implemented.
Citation Information
Patent Citations
Data virtualization system, method, device and equipment based on graph structure and medium
CN116383238A
Center-free distributed storage method and system
CN117440003A
