A distributed data query method and system

Abstract

The application discloses a kind of distributed data query method and system, data distributed is stored in multiple computing nodes, the method is suitable for computing node, comprising the following steps: receiving the query request initiated by user;Parsing query request;When judging that the data to be queried in parsed query request is not on this computing node, determine the other computing node according to the data primary key to be queried;Query request is sent to other computing node, to receive the data returned after other computing node executes query request, and present to user.The application has greatly improved performance in data loading or data query, and with the increase of data volume, only need to increase computing node to ensure that data loading or query performance is not affected, without modifying system software, system has good scalability, which is not possessed by traditional centralized system.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a distributed data query method and system. Background Technology

[0002] With increasingly heavy business pressures, digital transformation has become a top priority for all types of enterprises, especially industrial enterprises. This brings up the question of how industrial enterprises can effectively manage and utilize their data assets to achieve cost reduction and efficiency improvement. Real-time databases are the best choice for industrial enterprises' digital transformation and are currently the most suitable product for this type of data. In some very large industrial enterprises, the amount of data generated daily by various sensors and equipment can easily reach tens of millions or even hundreds of millions of records, with the number of data monitoring points in the entire system easily exceeding ten thousand. The pressure of data loading, storage, and querying brought about by such a massive amount of data places high demands on the system's capabilities, but currently, there is a lack of such an efficient data processing system. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to address at least one deficiency of the related technologies mentioned in the background: in the face of the pressure of data loading, storage and querying brought about by the huge amount of data, there is a lack of an efficient data processing system, and a distributed data query method and system is provided.

[0004] The technical solution adopted by this invention to solve its technical problem is: to construct a distributed data query method, wherein the data is distributed and stored in multiple computing nodes, and the method is applicable to computing nodes, including the following steps:

[0005] Receive query requests initiated by users;

[0006] Parse the query request;

[0007] If it is determined that the data to be queried in the parsed query request is not on this computing node, the other computing node where it is located is determined based on the primary key of the data to be queried;

[0008] The query request is sent to other computing nodes to receive the data returned by the other computing nodes after executing the query request, and then presented to the user.

[0009] Preferably, in the distributed data query method of the present invention, the method further includes:

[0010] When it is determined that the data to be queried in the parsed query request is on this computing node, the data is retrieved from this computing node and presented to the user.

[0011] Preferably, in the distributed data query method of the present invention, multiple computing nodes can communicate and exchange data with each other;

[0012] Multiple computing nodes include multiple physical computing nodes; or, include one physical computing node and multiple simulated computing nodes running simultaneously on the physical computing node.

[0013] Preferably, in the distributed data query method of the present invention, when there are N computing nodes, each computing node generates at least one communication listening thread for listening to the communication connections of other computing nodes, one client sending thread for sending the query request when acting as a client, (N-1) server data processing and providing threads for providing data when acting as a server, (N-1) client data receiving threads for receiving data when acting as a client, and one management and scheduling main thread; wherein, N is greater than or equal to 2.

[0014] Preferably, in the distributed data query method of the present invention, the data is distributed and stored across multiple computing nodes, including:

[0015] The data of each compute node's corresponding shards is stored on disk;

[0016] Each computing node loads the data from the disk into a preset data structure in memory.

[0017] Preferably, in the distributed data query method of the present invention, determining whether the data to be queried in the parsed query request is present or not in this computing node includes:

[0018] The shard to which the data is divided is determined based on the primary key of the data to be queried, and it is determined whether the shard is on this computing node or not.

[0019] Preferably, in the distributed data query method of the present invention, the preset data structure is a B+ tree structure, and the data is stored at the leaf nodes according to the data primary key and the data value.

[0020] Preferably, in the distributed data query method of the present invention, the method further includes:

[0021] When reading data into memory and parsing it, determine whether the primary key of the data is being retrieved for the first time. If so, use the data value corresponding to the primary key as part of the leaf node value, generate a new leaf node of the B+ tree, and use it as the first leaf node.

[0022] If not, the data value corresponding to the primary key will be used as the overflow page node of the existing leaf node and directly linked after the first leaf node.

[0023] Preferably, in the distributed data query method of the present invention, the method further includes:

[0024] Based on the data sharding category of each computing node, at least one type of B+ tree is generated and cached in the memory.

[0025] The present invention also constructs a distributed data query system, including multiple computing nodes, in which data is distributed and stored in multiple computing nodes, and the computing nodes implement the distributed data query method described in any of the above claims.

[0026] By implementing this invention, the following beneficial effects are achieved:

[0027] Analysis comparing centralized and distributed deployment methods reveals that in traditional centralized systems, all business operations are concentrated on a single device, resulting in heavy load and susceptibility to single points of failure. If this device fails, all business operations cease. In contrast, distributed systems distribute business operations across multiple inexpensive devices (computing nodes). If one device fails, only the business on that device is affected, while the business on other devices remains unaffected. Compared to traditional centralized systems, the distributed data query method and system of this invention significantly improve performance in both data loading and querying. Furthermore, as data volume increases, simply adding computing nodes ensures unaffected data loading or query performance without modifying the system software. The system exhibits excellent scalability, a feature lacking in traditional centralized systems. Attached Figure Description

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0029] Figure 1 This is a flowchart of the distributed data query method of the present invention;

[0030] Figure 2 This is a flowchart of the method for distributed data storage across multiple computing nodes according to the present invention;

[0031] Figure 3 This is a flowchart of the method for resolving duplicate primary keys in a B+ tree according to the present invention;

[0032] Figure 4 This is a schematic diagram of the overflow page node of the present invention. Detailed Implementation

[0033] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0034] It should be noted that the flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0035] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0036] like Figure 1 As shown, one embodiment of the present invention discloses a distributed data query method, wherein the data is distributed and stored in multiple computing nodes. The method is applicable to computing nodes and includes the following steps:

[0037] Receive query requests initiated by users;

[0038] Parse the query request;

[0039] If it is determined that the data to be queried in the parsed query request is not on this compute node, the other compute node where it is located is determined based on the primary key of the data to be queried;

[0040] It sends query requests to other compute nodes to receive the data returned by those nodes after executing the queries, and then presents it to the user. For example, the primary key of the data is the data ID.

[0041] In this embodiment, as Figure 1 As shown, the method also includes:

[0042] If it is determined that the data to be queried in the parsed query request is on this computing node, the data is retrieved from this computing node and presented to the user.

[0043] In this embodiment, multiple computing nodes can communicate and exchange data with each other, for example, through connection-oriented Socket communication based on the TCP / IP network communication protocol.

[0044] Multiple computing nodes can include multiple physical computing nodes. Alternatively, multiple computing nodes can include one physical computing node and multiple analog computing nodes running simultaneously on that physical computing node. For example, the physical computing node can be a general-purpose or embedded computing device, and the analog computing nodes can be processes.

[0045] Specifically, this distributed system consists of one or more physical computing nodes participating in the business operations. When there is only one physical computing node in the distributed system, that physical computing node can simultaneously run several identical simulated computing nodes, i.e., processes. Each process constitutes a logical computing node and simulates a physical computing node to complete various business operations. The processes still exchange data through connection-oriented socket communication based on the TCP / IP network communication protocol, but each process listens on a different port. The entire system is a scalable distributed architecture. Whether one or more physical nodes participate in the distributed business operations, the system and software architecture remain unchanged, and no program code modification is required.

[0046] When multiple physical computing nodes exist in a distributed system, each node must be configured with a different listening IP address, while the listening port number can be the same or different. Understandably, physical devices must have different IP addresses to distinguish them; otherwise, IP address conflict alerts will occur. Devices with different IP addresses can choose the same port to listen, just as different cities can have street names with the same name or different street names.

[0047] When a distributed system has only one physical computing node, that node can simultaneously launch multiple identical processes. Each process must be configured with the same listening IP address, but each must have a different listening port number. This is understandable, as it involves launching multiple processes on the same physical device, with each process simulating a real physical device. To distinguish between different processes, similar to distinguishing between different physical devices, the only way is through different port numbers.

[0048] The system assigns consecutive numbers to each physical computing node. For example, computing node 1 is numbered 0, computing node 2 is numbered 1, ..., computing node N is numbered (N-1). When there is only one physical computing node in the system, the numbering method for multiple physical computing nodes is used to assign consecutive numbers to the one physical computing node and the multiple simulation computing nodes running within it.

[0049] In this embodiment, when there are N computing nodes, each computing node generates at least one communication listening thread for listening to the communication connections of other computing nodes, one client sending thread for sending query requests when acting as a client, (N-1) server data processing and providing threads for providing data when acting as a server, (N-1) client data receiving threads for receiving data when acting as a client, and one management and scheduling main thread; wherein, N is greater than or equal to 2.

[0050] Specifically, each computing node acts as both a TCP listening server and a client initiating TCP connections. The system features customized configurations for each party in the Socket communication to achieve better and more robust communication services. This includes setting the number and size of TCP data transmission and reception buffers. The buffers are used to cache sent or received data to prevent the data sender or receiver from being unable to process the data in time, thus avoiding later data overwriting earlier data.

[0051] Since connection-oriented TCP communication byte streams lack clear boundaries, large amounts of data transmission and reception can easily cause TCP packets to become contiguous. To address this issue, this distributed system uses system-defined packet structures as the basic unit of data exchange. Each packet consists of a header field (24 bytes long, configurable before system startup) and a payload field, where the actual data is stored. To reduce the number of small packets and improve communication efficiency, the maximum allowed packet length is 8KB. Therefore, the maximum payload field length = 8KB – header length.

[0052] When each compute node starts up, it creates a TCP listening thread to listen for TCP connections from other compute nodes. Each time the TCP listening thread accepts a TCP connection from a client, it generates a separate server-side data processing and providing thread to provide data. When there are N compute nodes in the distributed system, any one compute node will generate at most (N-1) server-side data processing and providing threads. At the same time, any one compute node that initiates a connection request will also generate at most (N-1) client-side data receiving threads to receive data. Since the amount of data sent in the request is small, much smaller than the amount of data sent in the response, each client node that initiates a TCP connection only needs to generate one client-side sending thread to send query requests. This sending thread sends request data to each TCP server node in a round-robin manner.

[0053] Each computing node needs to maintain a set of TCP connections established with other computing nodes. When there are N computing nodes in the system, each computing node can establish TCP connections with at most (N-1) other nodes. These connections form a TCP connection pool. The algorithm steps for each computing node to create a TCP connection pool are as follows:

[0054] Step S1: First, initialize the relevant parameters. Set the TCP connection pool array to conPool to store (N-1) TCP connections. Set the initial index of the TCP connection pool array to 0. Set the variable iMachine to the compute node number (which can be configured in the system configuration file and is determined when the system starts). Set the listening base port number BASEPORT to a positive integer greater than 1024 and less than 60000 (which can be configured in the system configuration file and is determined when the system starts). Set the listening IP address array of N compute nodes to IPAddrARR (used to store the listening IP addresses of N compute nodes, with each array element storing the listening IP address of one compute node). Set the loop variable i to a positive integer with an initial value of 0.

[0055] Step S2: Determine if i is less than N. If yes, increment i by 1 and go to S3; otherwise, go to S7.

[0056] Step S3: Determine if i is not equal to iMachine. If yes, go to S4; otherwise, go to S2.

[0057] Step S4: Calculate the value of the peer node's listening port number iListenPort: iListenPort = BASEPORT + i * 10, then go to S5;

[0058] Step S5: Call the TCP connect function to connect to the i-th compute node (the connection IP address is stored in the IPAddrARR[i] array element), the port number is iListenPort calculated in the previous step, and after successful connection, it is used as the index-th connection in the TCP connection pool array conPool; go to S6;

[0059] Step S6: Increment the index value by 1 and then go to S2;

[0060] Step S7: End this process.

[0061] In addition, the algorithm steps for determining which other computing node corresponds to a long connection of any computing node in the TCP long connection pool are as follows:

[0062] Step S1: Determine if the compute node number iMachine is equal to 0. If yes, then the connection number index value iConIndex in the TCP connection pool is equal to the fragment number iSlice minus 1, and then go to S5; otherwise, go to S2.

[0063] Step S2: Determine if iMachine is equal to (N-1). If yes, then iConIndex is equal to iSlice, and then go to S5; otherwise, go to S3.

[0064] Step S3: Determine if iSlice is less than iMachine. If so, iConIndex equals iSlice, then proceed to S5; otherwise, proceed to S4.

[0065] Step S4: iConIndex equals iSlice minus 1; go to S5;

[0066] Step S5: End this process.

[0067] A complete TCP connection session requires a seven-way handshake. Short connections incur too much overhead, so this system supports persistent TCP connections. According to the TCP / IP protocol, even persistent TCP connections will be forcibly closed if there is no data exchange for two consecutive hours. Therefore, this system has a keep-alive mechanism for persistent TCP connections. When there is no data exchange between the two computing nodes' TCP connections, the keep-alive mechanism is triggered, and the TCP protocol stack in the operating system kernel of one computing node sends a TCP keep-alive probe message. If the other computing node is functioning normally, it will respond normally when the TCP keep-alive probe message is sent. Therefore, when a response is received after sending the TCP keep-alive probe message to the other computing node, the TCP keep-alive timeout is reset, and the system waits for the next TCP keep-alive timeout. If the peer computing node crashes or the message becomes unreachable for other reasons, and it is determined that there is no response after the TCP keep-alive probe message is sent to the peer computing node and the maximum number of keep-alive probes has been reached, then TCP reports that the TCP connection has been broken, re-establishes the TCP long connection, and replaces the corresponding failed TCP connection in the TCP connection pool.

[0068] A complete TCP communication session involves two main steps: connection establishment and connection termination. These two steps involve three and four handshake exchanges, respectively, totaling seven handshake operations.

[0069] 1. Establish connection protocol (three-way handshake)

[0070] (1) The client sends a TCP packet with the SYN flag to the server. This is packet 1 in the three-way handshake process.

[0071] (2) The server responds to the client with the second message in the three-way handshake. This message carries both the ACK and SYN flags. Therefore, it indicates a response to the client's previous SYN message; at the same time, it sends a SYN flag to the client, asking if the client is ready to communicate.

[0072] (3) The client must respond to the server with an ACK message again, which is segment 3.

[0073] 2. Connection Termination Protocol (Four-Way Handshake)

[0074] TCP connections are full-duplex (allowing simultaneous sending and receiving), therefore, when closing a connection, both the sending and receiving directions must be closed. The client sends a TCP segment with FIN set to 1 to the server. The server then returns an ACK segment to the client and sends a FIN segment. Once the client replies with an ACK segment (the four-way handshake), the connection is terminated.

[0075] (1) The TCP client sends a FIN to close the data transmission from the client to the server (segment 4).

[0076] (2) When the server receives this FIN, it sends back an ACK, with the sequence number being the received sequence number plus 1 (segment 5). Like SYN, a FIN will occupy one sequence number.

[0077] (3) The server closes the client's connection and sends a FIN to the client (segment 6).

[0078] (4) The client sends back an ACK message to confirm the receipt, and sets the confirmation sequence number to the received sequence number plus 1 (segment 7).

[0079] In this embodiment, as Figure 2 As shown, the data is distributed and stored across multiple computing nodes, including:

[0080] The data of each compute node's corresponding shards is stored on disk;

[0081] Each compute node maps and loads data from the disk into a pre-defined data structure in memory.

[0082] The step of storing the shard data corresponding to each computing node on the disk includes:

[0083] Based on the number of computing nodes and the amount of data, the data file to be loaded is divided into fragments and the fragment size is determined.

[0084] Each compute node's disk stores one data shard.

[0085] Accordingly, whether the data to be queried in the query request parsed above is present or not in this computing node includes:

[0086] The system determines the shard to which the data is partitioned based on the primary key of the data being queried, and then determines whether the shard is on this compute node. Specifically, it determines the shard number to which the data is partitioned. The formula for determining the shard number iSlice is as follows: iSlice = (iID - 1) / PERSLICE; where iID is the primary key (ID) of the data, and PERSLICE is a fixed value calculated by the system.

[0087] Specifically, data is generally permanently stored on the disk and loaded into memory only when needed. For example, a Word file, after shutting down, is definitely saved on the disk (such as a hard drive or USB flash drive). It is only loaded into memory when viewed or edited. If the file content is deleted, modified, or added, it also needs to be saved to the disk; otherwise, the changes will not be reflected when the Word file is reopened after shutdown. Traditional file I / O operations first need to read the disk file content into memory, then modify it, and finally write it back to the disk. The first step, reading the disk file, requires a system call, which first copies the file content from the disk to a buffer in kernel space, and then copies this data to user space—actually two data copies. The third step, writing back, also requires two data copies, so there are essentially four data copy operations.

[0088] Memory-mapped files reduce unnecessary data copying, thus improving efficiency. They directly map disk data to user space without actual data copying. The real data copying occurs during page fault handling. A page fault is an interrupt issued by the CPU's memory management unit when software attempts to access a page mapped to the virtual address space but not yet loaded into memory. Because disk data is directly mapped to user space, the interrupt handler copies the data from disk to user space in a single operation, twice as efficient as the two data copies performed by a read call. Therefore, memory mapping is significantly more efficient than read / write, especially for large file operations.

[0089] This invention uses memory-mapped files to load large amounts of data. The specific loading method is as follows:

[0090] (1) Based on the number of computing nodes and the amount of data, the data file to be loaded is partitioned and the partition size is determined. For example, when the primary key (e.g., the ID number field of the data can be set as the primary key) takes the value 1–MAX (MAX is the maximum value of ID, a positive integer), and there are N computing nodes, the data file to be loaded is partitioned according to the range of data IDs, then the size of each partition is PERSLICE = MAX / N.

[0091] (2) Each compute node stores one data shard on its disk. For example, the range of data IDs that each compute node can load is {i*PERSLICE+1} to {(i+1)*PERSLICE}, where PERSLICE is the size of a single shard, i is the node number, and 0≤i≤(N-1). The maximum data ID that each compute node can load is (i+1)*PERSLICE, and the minimum is i*PERSLICE+1. Maximum - Minimum = PERSLICE, which is exactly one shard size. Obviously, the data loaded by the last node may be less than one shard size.

[0092] (3) Each compute node loads the data from the disk into a preset data structure in memory. That is, each compute node loads data through a memory-mapped file, and only loads data within the data ID range corresponding to that compute node into the preset data structure in the memory of that compute node. Data outside the data ID range corresponding to that compute node will be discarded without any processing after being detected.

[0093] The advantages of this sharding and partitioning approach are: each computing node has a moderate data load and is easily scalable. When querying all data with any ID, the system can intelligently determine which computing node to retrieve the data from based on the data ID, eliminating the need for all computing nodes to participate simultaneously. This reduces network communication bandwidth consumption, minimizes unnecessary repeated query time, and improves the business processing efficiency of the distributed system.

[0094] In this embodiment, the preset data structure is a B+ tree structure, and the data is stored at the leaf nodes according to the primary key and the data value.

[0095] like Figure 3 As shown, the method further includes:

[0096] When reading data from memory and parsing it, determine whether the primary key is being retrieved for the first time. If so, use the data value corresponding to the primary key as part of the leaf node value, generate a new leaf node in the B+ tree, and use it as the first leaf node.

[0097] If not, the data value corresponding to the primary key will be used as the overflow page node of the existing leaf node and directly linked after the first leaf node.

[0098] Specifically, after massive amounts of data are loaded from disk into memory, different storage strategies directly lead to different query efficiencies. Efficient and inefficient storage strategies can result in query efficiency differences of thousands of times. For example, if a large library holds millions of books, and instead of properly indexing and storing them, they are stored haphazardly, finding a single book might take up to 10 days. However, with a well-organized index, finding a book takes only seconds.

[0099] B+ trees are used to store data in memory. B+ trees are multi-branch balanced search trees designed for efficient access. Following the principle of "key-value" pairs for B+ tree leaf nodes, where the "value" is not limited to simple numbers or strings but can be a complete record structure, the primary key (e.g., ID) is selected as the key of the B+ tree. Since data exists from multiple points in time, each data ID may be duplicated extensively in the B+ tree. The following strategy is used to solve the problem of ID duplication:

[0100] (1) When parsing data in memory, when the data ID is retrieved for the first time, the data value corresponding to the data ID is used as a component of the leaf node value, and a new leaf node of the B+ tree is generated and used as the first leaf node.

[0101] (2) Otherwise, the data value corresponding to the data ID is used as the overflow page node of the existing leaf node and directly linked after the first leaf node, instead of linking it after the last overflow page node. This can save the retrieval time of the linked list (since there may be a large number of duplicate IDs, the overflow page linked list may be very long) and greatly improve the storage speed. Obviously, on average, each first leaf node may be linked to multiple overflow page nodes. Figure 4 This is a diagram illustrating the overflow page node when there are multiple duplicate keys for data ID 57.

[0102] In some embodiments, the method further includes:

[0103] Assign the currently read value to the corresponding field value of the first leaf node value, and assign the count field value to 1;

[0104] The currently read value is incremented in the corresponding field value of the first leaf node value, and the count is updated in the counter field value;

[0105] Calculate the average of the corresponding field values ​​in the first leaf node.

[0106] Specifically, the first leaf node value contains multiple field values, such as temperature or other scalar data. Each first leaf node can cache one or more field values. The currently read value is assigned to the corresponding field value of the first leaf node value, and the count field is assigned a value of 1. As mentioned earlier, each type of data can have data from multiple time points. For example, a sensor with ID number 1 may have monitoring data at points 1, 2, 3, etc. When the first monitoring data with ID number 1 is read, the currently read value is assigned to the corresponding field value of the first leaf node value, and the count field is assigned a value of 1. When monitoring data from other time points with ID number 1 is read subsequently, the count field value simply needs to be incremented by 1 (because it has already been initialized to 1).

[0107] The system accumulates the currently read value in the corresponding field of the first leaf node and updates the count in the counter field (count field value +1). Through this caching operation, when querying the average monitored temperature of a certain sensor over a period of time, it only needs to read the first leaf node once to immediately calculate data such as the average temperature. For example, if the accumulated temperature data value of the first leaf node is read as 500, and the node has 50 monitoring data, then the average temperature = 500 / 50 = 10 degrees. It is not necessary to search all the overflow page nodes under the first leaf node, which greatly reduces the query time.

[0108] In this embodiment, the method further includes:

[0109] Based on the data sharding category of each computing node, at least one type of B+ tree is generated and cached in memory.

[0110] Specifically, in practice, it is often necessary to classify and manage data. For example, Y types of data can be divided into M categories, with each category containing approximately Y / M types of data on average. In order to quickly retrieve one or more types of data from a certain category, all data IDs contained in each category can also be stored in a B+ tree. For example, if there are 25 categories of data, 25 B+ trees can be generated and cached in memory. Each B+ tree only stores the data IDs contained in that category. This caching indexing mechanism can effectively improve the efficiency of querying large amounts of data, which will not be elaborated further.

[0111] The order of a B+ tree affects both tree creation and read / write time, requiring a balance between tree creation and node retrieval. Generally, any leaf data node on any tree should be retrieved within three read / write operations. In this system, each compute node can create various types of B+ trees as needed, such as:

[0112] (1) Monitor unclassified B+ trees. If there are 500,000 data points on each computing node, when logm500000<=3, the order m=80 can be calculated. The order represents how many child nodes each node in a B+ tree can have at most. For example, an order of 80 means that a node can have at most 80 child nodes.

[0113] (2) Monitor the classified B+ trees. If there are 500,000 data points on each computing node, divided into 25 categories, then each classified B+ tree has an average of about 20,000 data points. logm20000<=3, and the order m=28 is calculated.

[0114] One embodiment of the present invention discloses a distributed data query system, including multiple computing nodes, in which data is distributed and stored in multiple computing nodes. The computing nodes implement the distributed data query method described in the above embodiment, which will not be repeated here.

[0115] Using TPC-H (Transaction Processing Performance Council) as the benchmark, the TPC-H public dataset contains 8 tables and 22 queries, following SQL92. TPC-H simulates database operations in a decision support system, testing the response time of complex queries in the database system. The number of queries executed per hour is used as the metric, demonstrating strong versatility and guidance. The TPC-H standard specifies that when the scaling factor is 10, the generated test database size is approximately 10GB. This implementation selects two tables: the parts table and the order table, as the test dataset (the combined size of the two tables is approximately 7.5GB). The parts table stores 2 million different types of parts records, and the order table stores approximately 60 million order records, equivalent to an average of about 30 orders for each different part. This is well-suited for simulating scenarios in industrial enterprise databases where each monitored object has several monitoring records from different time periods. To simplify the test, the test environment consisted of a single ordinary PC, i.e., only one physical computing node, with the following configuration: CPU information: Intel(R) Core(TM) i5-8500 CPU@3.00GHz, clock speed: 3000MHz, number of cores (i.e., number of CPUs): 6 cores, memory 8GB. Data loading, storage, and query response times were tested under both centralized and distributed deployment methods.

[0116] Centralized deployment: When only one program instance is launched on a single PC, it is equivalent to a centralized system. This instance loads 2 million part records and approximately 60 million order records and stores them in an in-memory B+ tree. The total loading and storage time is approximately 170 seconds, and the average time to query all order data for any single part is approximately 0.1 seconds.

[0117] Distributed Deployment: Four distributed program instances are launched simultaneously on a single PC to simulate four different physical computing nodes in a distributed system. Each program instance loads 500,000 part records and approximately 15 million order records into an in-memory B+ tree. The test results are as follows:

[0118] (1) Four instances completed loading and storing data, with an average time of approximately 62 seconds.

[0119] (2) Distributed query of all order data for any single part takes an average of about 0.01 seconds.

[0120] It should be noted that if the test device has 6 CPUs, it is appropriate to start fewer than 6 program instances (i.e., processes) at the same time. More than 6 will affect performance because too many processes need to queue up and take turns using the CPU, and too much context switching by the operating system will cause delays.

[0121] Analysis comparing centralized and distributed deployment methods reveals that in traditional centralized systems, all business operations are concentrated on a single device, resulting in heavy load and susceptibility to single points of failure. If this device fails, all business operations cease. In contrast, distributed systems distribute business operations across multiple (up to thousands or tens of thousands) inexpensive devices (computing nodes). If one device fails, only the business on that device is affected, while the business on other devices remains unaffected. Compared to traditional centralized systems, the distributed data query method and system of this invention significantly improve performance in both data loading (including storage in specific memory data structures) and data querying. Furthermore, as the data volume increases, simply adding computing nodes ensures that data loading or query performance remains unaffected without modifying the system software. The system possesses excellent scalability, a feature lacking in traditional centralized systems.

[0122] It is understood that the above embodiments only illustrate some implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that for those skilled in the art, without departing from the concept of the present invention, the above embodiments or technical features can be freely combined, and several modifications and improvements can be made. These all fall within the protection scope of the present invention. That is, the embodiments described "in some embodiments" can be freely combined with any of the embodiments above and below. Therefore, all equivalent transformations and modifications made within the scope of the claims of the present invention should fall within the scope of the claims of the present invention.

Claims

1. A distributed data query method, characterized in that, Data is distributed and stored across multiple computing nodes. These nodes exchange data via connection-oriented Socket communication based on the TCP / IP network communication protocol. Each party uses a system-defined message data packet as the basic unit for data exchange. The message data packet consists of a header field and a payload data field, with the payload data field used to store the actual data. Multiple computing nodes include multiple physical computing nodes; or, include one physical computing node and multiple simulated computing nodes running simultaneously on the physical computing node. When there are N computing nodes, each computing node generates at least one communication listening thread for listening to the communication connections of other computing nodes, one client sending thread for sending the query request when acting as a client, N-1 server data processing and providing threads for providing data when acting as a server, N-1 client data receiving threads for receiving data when acting as a client, and one management and scheduling main thread; wherein, N is greater than or equal to 2; The method is applicable to computing nodes and includes the following steps: Receive a query request initiated by a user; parse the query request; When it is determined that the data to be queried in the parsed query request is on this computing node, the data is retrieved from this computing node and presented to the user; If it is determined that the data to be queried in the parsed query request is not on this computing node, the other computing nodes where the data to be queried is located are determined based on the primary key of the data to be queried; the query request is sent to the other computing nodes, the data returned by the other computing nodes after executing the query request is received, and presented to the user; The data is distributed and stored across multiple computing nodes, including: Each computing node stores its corresponding shard data on a disk; each computing node maps and loads the data from the disk into a preset data structure in memory; the preset data structure is a B+ tree structure, and the leaf nodes are stored according to the primary key and the data value. Determining whether the data to be queried in the parsed query request is on or not on this computing node includes: determining the shard to which the data is divided based on the primary key of the data to be queried, and determining whether the shard is on or not on this computing node; When reading data from memory and parsing it, determine whether the primary key is being retrieved for the first time. If so, use the data value corresponding to the primary key as part of the leaf node value to generate a new leaf node in the B+ tree and use it as the first leaf node. If not, use the data value corresponding to the primary key as the overflow page node of the existing leaf node and link it directly after the first leaf node. The method further includes assigning the currently read value to the corresponding field value of the first leaf node value, assigning a count field value of 1, accumulating the currently read value to the corresponding field value of the first leaf node value, and updating the count in the count field value.

2. The distributed data query method according to claim 1, characterized in that, The method further includes: Based on the data sharding category of each computing node, at least one type of B+ tree is generated and cached in the memory.

3. A distributed data query system, characterized in that, It includes multiple computing nodes, and the data is distributed and stored on the multiple computing nodes. The computing nodes implement the distributed data query method according to any one of claims 1-2. The system also sets the number and size of TCP data transmission and reception buffers. The buffer is used to cache the data being sent or received to prevent the data sender or receiver from being unable to process the data in time, causing later data to overwrite the previous data.

Images