Distributed query method, device and system and storage medium

Abstract

The invention discloses a distributed query method, device and system and a storage medium. The method comprises the steps of obtaining a query operation request of a user; according to a query operation request of a user, key value pair data is obtained, and meanwhile, a processed correct multicast sent data packet is output; according to the key value pair data, obtaining data stream operation requests after batch processing; remote memory access processing is carried out on the data stream operation requests after batch processing, and correct multicast sent data packets are obtained; processing a correct multicast sent data packet to obtain a rate-limited data packet; sending the data packet with the limited rate to a remote memory; and tracking the transmission state of the data packet. By adopting the technical scheme of the invention, data transmission and query processing between nodes in a distributed architecture are optimized.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of network cards, and particularly relates to a distributed query method, device, system and storage medium. BACKGROUND

[0002] Traditional network cards (NICs) are mainly used for receiving and sending data packets, and their functions are fixedly implemented by hardware, lacking flexibility. For example, traditional network cards usually support the basic functions of Ethernet protocols and TCP / IP protocol stacks, but cannot be customized according to specific needs. With the rapid growth of network bandwidth (such as from 10 Gbps to 100 Gbps or even higher), traditional network cards face performance bottlenecks in processing high-throughput, low-latency network traffic. In particular, in data center and cloud computing environments, the hardware architecture of traditional network cards is difficult to meet the needs of diversified applications. Although modern network cards support hardware offloading of part of the protocol stack functions (such as TCP segmentation, checksum calculation, encryption, etc.), their offloading capabilities are limited and cannot handle complex network functions (such as deep packet inspection, dynamic load balancing, custom protocol processing, etc.).

[0003] Programmable network cards allow users to customize data packet processing logic according to specific needs by introducing programmable hardware (such as FPGAs, ASICs) and software-defined networking (SDN) technology. For example, users can implement custom protocols, advanced traffic scheduling, multi-tenant isolation, etc. through programmable network cards. Through hardware acceleration technology, complex network functions such as encryption, compression, neural network inference, etc. are offloaded to hardware for processing, significantly improving performance and reducing CPU load. In cloud computing and data center environments, programmable network cards can provide isolated network resources for different tenants and dynamically allocate bandwidth, priority, and traffic scheduling strategies according to needs. However, existing programmable network cards have insufficient embedded CPU kernel performance, low data packet processing efficiency, and arbitrary task division.

[0004] In traditional distributed query systems, the processing of query requests usually relies on general-purpose processors (CPUs), which can cause performance bottlenecks when processing large-scale data. The CPU needs to handle a large amount of data transmission and query logic, resulting in long query response time and low system throughput. At the same time, data transmission mainly depends on the network, and network latency and bandwidth limitations can severely affect query performance. The utilization rate of CPU and network resources is also often low, and network bandwidth is often not fully utilized due to low data transmission efficiency. The scalability and flexibility of existing distributed query systems are relatively poor, and expansion requires constant hardware resources. When faced with new query patterns or data types, the system needs to be modified and redeployed complexly. Finally, the current distributed query system has security and reliability problems in data transmission and storage processes. Data is easy to be stolen or tampered with, and the system's recovery ability is weak when facing network attacks and hardware failures. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a distributed query method, device, system and storage medium, which optimizes data transmission and query processing between nodes in a distributed architecture, can directly perform data filtering, aggregation and sorting operations at the network card level, reduces the overhead of CPU and network, and can preprocess network traffic before data reaches the host, thereby reducing data transmission volume and reducing delay.

[0006] To achieve the above object, the present application adopts the following technical solution: A distributed query method, comprising: Step S1, obtaining a user's query operation request; Step S2, obtaining key-value pair data according to the user's query operation request; and processing hash conflicts of incorrect multicast sent data packets, and outputting processed correct multicast sent data packets; Step S3, obtaining a batch processed data stream operation request according to the key-value pair data; Step S4, performing remote memory access processing on the batch processed data stream operation request, and obtaining correct multicast sent data packets; Step S5, obtaining rate limited data packets according to the correct multicast sent data packets; Step S6, sending the rate limited data packets to a remote memory; Step S2 comprises: Receiving the user's query operation request as input, parsing and executing read, write or delete operations, and outputting operation results or key-value pair data to be stored; Receiving incorrect data packets in the multicast sent data packets, calculating storage locations and processing hash conflicts through a hash table, and outputting processed correct multicast sent data packets; Step S4 comprises: Forwarding the batch processed data stream operation request to the next network node according to predetermined routing information, and outputting the forwarded data packets; Copying and sending the forwarded data packets to multiple target nodes, and outputting multicast sent data packets, wherein incorrect data packets are processed for hash conflict.

[0007] The present application also provides a distributed query device, comprising: a network card hardware acceleration module integrated on a programmable network card, a KV Store integrated module, a query optimization module, a configurable matching table module, a traffic aggregation module, a sending module and a tracking module; wherein, The network card hardware acceleration module is configured to obtain a user's query operation request; The KV Store integration module is configured to obtain key-value pair data according to a query operation request of a user, and to process hash conflicts of a data packet that is incorrectly multicast, and output a correctly multicast data packet after processing. The query optimization module is configured to obtain a batch-processed data stream operation request according to the key-value pair data. The configurable match table module is configured to perform remote memory access processing on the batch-processed data stream operation request, and obtain a correctly multicast data packet. The traffic aggregation module is configured to process the correctly multicast data packet, and obtain a rate-limited data packet. The sending module is configured to send the rate-limited data packet to a remote memory. The tracking module is configured to track a transmission state of the data packet.

[0008] Preferably, the KV Store module includes a RISC-V Processor component and a Hash Into Memory component. The RISC-V Processor component is configured to receive a query operation request of a user as input, parse and execute a read, write or delete operation, and output an operation result or key-value pair data that needs to be stored. The Hash Into Memory component is configured to receive a data packet that has a problem in a multicast data packet, calculate a storage location through a hash table and process hash conflicts, and output a correctly multicast data packet after processing.

[0009] Preferably, the data query optimization module includes a Rate Limiter component and a Batching component. The Rate Limiter component is configured to perform rate limiting and conversion processing on the key-value pair data, and output a plurality of data streams after rate adjustment. The Batching component is configured to combine a plurality of data stream operation requests into batch processing, and output a batch-processed data stream operation request.

[0010] Preferably, the configurable match table module includes a Forwarder and a Multicaster. The Forwarder is configured to forward the batch-processed data stream operation request to a next network node according to predetermined routing information, and output a forwarded data packet. The Multicaster is configured to copy and send the forwarded data packet to a plurality of target nodes, and output a multicast data packet. Correct data packets are sent to the traffic aggregation module, and incorrect data packets are sent to the KV Store module.

[0011] As preferred, the traffic aggregation module comprises: a Traffic Manager, an Aggregator and an RDMA Limiter; wherein the Traffic Manager is configured to process the data packets of the correct multicast transmission according to priority, and output the scheduled data packets; the Aggregator is configured to aggregate the scheduled data packets; and the RDMA Limiter is configured to control the rate of remote direct memory access operation of the aggregated data packets, and output the rate-restricted data packets.

[0012] As preferred, the sending module comprises: a RoCEv2 Crafter component and an RDMA Emitter component; wherein the RoCEv2 Crafter component is configured to encapsulate the rate-restricted data packets into a format conforming to the RoCEv2 protocol, and output the encapsulated RoCEv2 data packets; and the RDMA Emitter component is configured to directly send the RoCEv2 data packets to the remote memory, and output the successfully sent data packets.

[0013] The application further provides a distributed query system comprising: a memory and a processor, wherein the memory stores a computer program which is run by the processor, and the computer program performs the distributed query method when run by the processor.

[0014] The application further provides a storage medium which stores a computer program, and the computer program performs the distributed query method when run.

[0015] The application has the following technical effects: 1. The programmable network card adopts a configurable matching table module, which can effectively improve the performance and efficiency of the system through optimization in terms of low latency, high throughput, load balancing, resource reuse, reliability guarantee, scalability, programmability and modular design, and provide strong technical support and solutions for modern distributed systems. This technology is suitable for various application scenarios such as distributed storage systems, high-performance computing clusters and real-time data processing, and can effectively solve the limitations of traditional RMT technology in terms of performance, flexibility and scalability, and provide more efficient, reliable and flexible data transmission and processing services for users.

[0016] 2、In the programmable network card, the Hash Into Memory component is introduced to handle hash collisions, which can bring significant benefits in terms of efficient handling of complex hash collisions, low latency and high throughput, high resource utilization efficiency, strong reliability and security, and easy development and maintenance. This design not only improves the performance and efficiency of the system, but also provides more reliable, secure, and flexible data processing and management services for users. It is suitable for various application scenarios such as distributed storage systems, high-performance computing clusters, and real-time data processing, and can effectively solve the limitations of traditional hash collision handling techniques in terms of performance, flexibility, and reliability, providing strong technical support and solutions for modern distributed systems.

[0017] 3、In the distributed storage architecture, programmable network cards are introduced, which significantly improve the performance, flexibility, resource utilization efficiency, and reliability of the system through high-performance hardware acceleration, flexible programmability, efficient resource utilization, reliability and security, easy development and maintenance, and wide application scenarios. This design not only meets the requirements of modern distributed systems for high performance and low latency, but also provides more reliable, secure, and flexible data processing and management services for users.

[0018] 4、The full-link architecture optimization scheme of the traffic aggregation technology is designed, which not only improves the performance and efficiency of the system, but also provides more reliable, secure, and flexible data processing and management services for users. It is suitable for various application scenarios such as distributed storage systems, high-performance computing clusters, and real-time data processing, and can effectively solve the limitations of traditional techniques in terms of performance, flexibility, and reliability, providing strong technical support and solutions for modern distributed systems. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0020] Figure 1 Flowchart of distributed query method; Figure 2 Structure diagram of distributed query device; Figure 3 Data processing diagram of distributed query device; Figure 4 Data packet diagram of distributed query device; Figure 5 Related function diagram of hash table on RISCV. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] Example 1: like Figure 1 As shown, this embodiment of the invention provides a distributed query method, including: Step S1: Obtain the user's query operation request; Step S2: Obtain key-value pair data based on the user's query operation request; at the same time, handle hash collisions of erroneous multicast data packets and output the correct multicast data packets after processing. Step S3: Based on the key-value pair data, obtain the batch-processed data stream operation request; Step S4: Perform remote memory access processing on the batch-processed data stream operation requests to obtain the correct multicast data packets; Step S5: Obtain the rate-limited data packets based on the correctly sent multicast data packets; Step S6: Send the rate-limited data packets to remote memory; Step S7: Track the transmission status of data packets; Step S2 includes: It receives user query requests as input, parses and executes read, write or delete operations, and outputs the operation results or key-value pairs that need to be stored. Receives problematic data packets from multicast packets, calculates their storage location using a hash table and handles hash collisions, and outputs the correct multicast packets after processing. Step S4 includes: Based on the predetermined routing information, the batch-processed data stream operation requests are forwarded to the next network node, and the forwarded data packets are output. The forwarded data packets are copied and sent to multiple target nodes, and the multicast data packets are output. In this process, erroneous data packets are handled by hash collision processing.

[0024] Example 2: like Figure 2As shown, the embodiment of the present application provides a distributed query device, comprising: a network card hardware acceleration module integrated on a programmable network card, a KV Store integrated module, a query optimization module, a configurable matching table module, a traffic aggregation module, a sending module and a tracking module; wherein, The network card hardware acceleration module is used for obtaining a query operation request of a user. The KV Store integrated module is used for obtaining key-value pair data according to the query operation request of the user; and processing hash conflicts of incorrect multicast sent data packets, and outputting correct multicast sent data packets after processing. The query optimization module is used for obtaining a data stream operation request after batch processing according to the key-value pair data. The configurable matching table module is used for performing remote memory access processing on the data stream operation request after batch processing, and obtaining correct multicast sent data packets. The traffic aggregation module is used for processing the correct multicast sent data packets, and obtaining rate limited data packets. The sending module is used for sending the rate limited data packets to a remote memory. The tracking module is used for tracking a transmission state of the data packets.

[0025] As an embodiment of the present application, the network card hardware acceleration module comprises: an RIAT component, which is used for converting the query operation request of the user into executable instructions.

[0026] As an embodiment of the present application, the KV Store module comprises: a RISC-V Processor component and a Hash Into Memory component; the RISC-V Processor module is used for receiving the query operation request of the user as input, analyzing and executing reading, writing or deleting operations, and outputting operation results or key-value pair data needed to be stored; and the Hash Into Memory module is used for receiving incorrect data packets in the multicast sent data packets, calculating storage positions and processing hash conflicts through a hash table, and outputting correct multicast sent data packets after processing.

[0027] As an embodiment of the present application, the data query optimization module comprises: a Rate Limiter component and a Batching component; wherein, the Rate Limiter component is used for rate limiting and converting the output key-value pair data into ordered data packets and collecting multiple data streams, and outputting data streams after rate adjustment; and the Batching component is used for merging multiple data streams for batch processing, and outputting data stream operation requests needed during batch processing.

[0028] As an embodiment of the present application, the configurable match table module includes: a Forwarder and a Multicaster; wherein the Forwarder is configured to forward the batch-processed data stream operation request to the next network node according to the predetermined routing information, and output the forwarded data packet; the Multicaster is configured to copy and send the forwarded data packet to multiple target nodes, and output the multicast-sent data packet, wherein the correct data packet is sent to the traffic aggregation module, and the incorrect data packet is sent to the KV Store module.

[0029] As an embodiment of the present application, the traffic aggregation module includes: a Traffic Manager, an Aggregator, and an RDMA Limiter; wherein the Traffic Manager is configured to manage the correct multicast-sent data packet, and output the scheduled data packet; the Aggregator is configured to aggregate the scheduled data packet; and the RDMA Limiter is configured to control the rate of the remote direct memory access operation of the aggregated data packet, and output the rate-limited data packet.

[0030] As an embodiment of the present application, the sending module includes: a RoCEv2 Crafter component and an RDMA Emitter component; wherein the RoCEv2 Crafter component is configured to encapsulate the rate-limited data packet into a format conforming to the RoCEv2 protocol, and output the encapsulated RoCEv2 data packet; and the RDMA Emitter component is configured to directly send the rate-limited data packet to the remote memory, and output the successfully sent data packet.

[0031] As an embodiment of the present application, the tracking module includes: a PSN Tracker component configured to track the transmission status of the data packet, record the sequence number thereof, and output the tracking information containing the sequence number and the transmission status.

[0032] As shown in Figure 3 , the above-mentioned embodiments are further refined Figure 1The flow within different modules shows the specific data packet flow direction and processing details. The execution flow of the distributed query architecture of the application is roughly as follows: the query operation request initiated by the user, including reading key-value pairs, writing key-value pairs, checking data, and deleting key-value pairs, is connected to the network interface card (NIC) through the switch and enters the subsequent processing flow. These requests are classified as "User Traffic" and accessed to the system through the switch. The user request first reaches the RIAT (Reconfigurable Instruction Address Translation) component, which is part of the network card hardware acceleration module, responsible for converting the request into executable instructions and performing address translation to ensure that the request can be correctly parsed and processed. Then, the RISC-V Processor component, as part of the KV Store integrated module, uses a processor with RISC-V architecture to execute the instructions converted by RIAT, performs preliminary data processing and logical operations. The Rate Limiter, which belongs to the query optimization module, controls the speed of request processing to prevent overload and congestion, ensuring the stability and responsiveness of the system. The main task of this stage is to convert user requests into executable instructions and perform preliminary processing and rate control to prepare for subsequent batch processing and forwarding. Then, the requests processed initially enter the Batching component, which is part of the query optimization module, for batch processing to reduce the processing overhead of individual requests and improve overall processing efficiency. The Forwarder and Multicaster, as part of the configurable matching table module, forward the processed requests or data packets to the next processing node or copy and send them to multiple target nodes, supporting distributed processing and load balancing. This stage realizes efficient data transmission and processing through batch processing, negative acknowledgement mechanism, and multicasting technology. The Hash Into Memory component, as part of the KV Store integrated module, hashes data into memory to realize fast data lookup and access, improving data processing speed. The Traffic Manager, which belongs to the traffic aggregation module, is responsible for managing and scheduling network traffic to ensure the orderly transmission and processing of data packets, avoiding network congestion and resource competition. This stage ensures fast data access and reliable transmission through memory hashing, traffic management, and checksum calculation. The Aggregator, also part of the traffic aggregation module, combines and processes multiple data packets or requests to reduce network transmission times and overhead, improving transmission efficiency. The RDMA Limiter, as part of the traffic aggregation module, controls the rate of RDMA operations to avoid network congestion and resource competition, ensuring the stability and performance of the system.The RoCEv2 Crafter component constructs data packets conforming to the RoCEv2 protocol. The RDMA Emitter component sends the constructed data packets to the target node using RDMA technology. The Egress output sends the processed data packets to an external network or storage device. The PSN Tracker, a part of the tracking module, records and manages the sequence numbers of data packets, supporting reliable transmission and retransmission mechanisms. This stage, through the RoCEv2 protocol, RDMA technology, and packet sequence number tracking, achieves efficient and reliable data transmission and delivery, completing the entire processing flow.

[0033] Combination Figure 2 and Figure 3 The query optimization module optimizes distributed queries through the Rate Limiter and Batching modules, improving query execution efficiency. By dynamically adjusting execution strategies, flow control, and scheduling strategies, as well as merging and processing multiple data packets or requests, the module significantly improves query efficiency and reduces query latency. The network card hardware acceleration module, through the RIAT module, converts user requests into executable instructions and performs address translation, ensuring that requests are correctly parsed and processed, thereby improving the efficiency and accuracy of request processing. The configurable matching table module, through the Forwarder and Multicaster, implements remote memory access, supporting distributed processing and load balancing; the sending module, through the RoCEv2 Crafter and RDMA Emitter components, achieves efficient data transmission; and the tracking module, through the PSN tracker, ensures reliable data transmission. These three modules work together to ensure data transmission reliability and system stability. The KV Store (key-value store) integration module, through the RISC-V Processor and Hash IntoMemory components, enables rapid data processing and logical operations, as well as fast data lookup and access, improving data processing speed and system performance. The traffic aggregation module preprocesses, aggregates, and manages network traffic through Traffic Manager, Aggregator, and RDMA Limiter, reducing data transmission volume and network bandwidth consumption, and improving transmission efficiency and system performance. By dynamically adjusting execution strategies, traffic control, and scheduling strategies, as well as merging and processing multiple packets or requests, it significantly improves query efficiency and reduces query latency.

[0034] like Figure 4The diagram illustrates the structure of a data packet, which consists of multiple fields, each with a specific bit width and function. These fields include: an opcode specifying the operation type (e.g., read, write, delete, or conditional); a key identifying the data item; a value for storing or transmitting the data; a result indicating the existence of the data; a status indicating success; a memory address specifying the data storage location; and a checksum verifying data integrity. The diagram also includes request and response formats, with the request portion containing the opcode, key, and value, and the response portion containing the result, status, memory address, and checksum. This structure is typically used to ensure the accuracy and reliability of data during network transmission or storage operations.

[0035] like Figure 5 The diagram illustrates a series of operation functions for a HashTable object on RISC-V, each responsible for a specific task: `handleMergeMessage` handles merge messages involving merging multiple hash tables; `insertHashTable` inserts new data into the hash table; `searchHashTable` searches for specific data within the hash table; `deleteHashTable` removes data from the hash table; `freeHashTable` releases the resources occupied by the hash table; `processData` performs some form of processing or transformation on data from the hash table and other sources; and `resizeHashTable` resizes the hash table to optimize performance or adapt to changes in data volume. These functions collectively form the core of hash table operations, ensuring efficient data management and manipulation.

[0036] The present application can directly process data and perform calculations on the network card, greatly reducing the delay and overhead of data transmission to the host CPU. For example, in the process of packet parsing, hash calculation, aggregation operation, etc., it can be completed directly on the network card, significantly improving the processing speed. Through the Batching module, multiple requests are processed in batches, reducing the processing overhead of individual requests and improving overall throughput and processing efficiency. At the same time, the RDMA technology supports remote direct memory access, further reducing network transmission delay and achieving efficient data transmission. The programmable network card allows users to customize and configure data processing procedures according to specific needs, flexibly responding to different application scenarios and business requirements. For example, the RIAT module can be used for instruction address translation. The entire architecture adopts a modular design, with each module working and optimizing independently, facilitating functional expansion and upgrading. For example, new processing modules can be added or existing modules can be optimized to adapt to changing needs and technological developments. Through the Multicaster multicaster and Traffic Manager traffic manager, distributed processing and load balancing are achieved, avoiding single-point bottlenecks and resource waste. At the same time, the Rate Limiter rate limiter controls the speed of request processing, preventing overload and congestion and ensuring system stability and responsiveness. The programmable network card can implement multiple functions and applications on the same hardware platform, reducing the number and cost of hardware devices. For example, the same network card can support data processing, storage management, and network communication, among other functions, improving resource utilization. Through PSN Tracker packet sequence number tracking and other technologies, the integrity and correctness of data transmission are ensured, improving system reliability and stability. At the same time, it supports retransmission and error recovery mechanisms, enhancing system fault tolerance. Finally, the architecture supports distributed deployment and horizontal expansion, allowing dynamic adjustment of node quantity and processing capacity according to demand. For example, by adding more programmable network cards and processing nodes, the system can be linearly expanded and its performance improved to meet the needs of large-scale distributed systems.

[0037] A distributed query system based on programmable network cards (Programmable NIC) can effectively solve multiple shortcomings in existing distributed query technologies by offloading some query processing tasks to the network card hardware level.

[0038] 1. CPU bottleneck problem In traditional distributed query systems, query processing tasks are mainly handled by CPUs. However, with the increase in network bandwidth and data size, CPUs become a performance bottleneck, especially in high-concurrency query scenarios, where the computing power and memory bandwidth of CPUs cannot meet the demand. Programmable network cards use hardware acceleration technology to offload some query processing tasks (such as data filtering, aggregation, sorting, etc.) to the network card, reducing the load on CPUs and alleviating CPU bottlenecks, thereby improving query processing efficiency.

[0039] 2. Network transmission overhead In distributed queries, data needs to be frequently transmitted between nodes, resulting in high network bandwidth occupation and increased query delay. Programmable network cards can preprocess data (such as data filtering or compression) before it reaches the host, reducing the amount of data that needs to be transmitted, thereby reducing network bandwidth occupation and transmission delay and improving overall query performance.

[0040] 3. Query delay In traditional distributed query systems, query tasks need to pass through multiple software stacks (such as operating system kernels and network protocol stacks), resulting in high processing delay. Programmable network cards use hardware acceleration and kernel bypass mechanisms to directly process query tasks at the network card level, reducing software stack overhead and significantly reducing query delay, especially in low-latency query scenarios.

[0041] 4. Multi-tenant resource isolation In a multi-tenant environment, query tasks from different tenants may interfere with each other, leading to unfair resource allocation or performance degradation. Programmable network cards use hardware-level resource isolation mechanisms to ensure that each tenant's query task can run independently, avoiding resource contention issues, and use fair scheduling algorithms to ensure fair resource allocation.

[0042] 5. Flexibility and scalability Traditional distributed query systems usually rely on fixed hardware and software designs, making it difficult to adapt to different query requirements or extend new functions. Programmable network cards support custom query processing logic through programming, allowing flexible adaptation to different query requirements, and supporting function extension through modular design, improving system flexibility and scalability.

[0043] 6. Energy efficiency In traditional distributed query systems, the high power consumption of CPUs and network devices leads to low energy efficiency, especially in large-scale data centers, where energy costs become a major issue. Programmable network cards use hardware acceleration and efficient resource utilization to reduce the load on CPUs and networks, thereby reducing the overall power consumption of the system and improving energy efficiency.

[0044] 7. Complex query optimization In traditional architectures, complex queries (such as multi-table joins, nested queries) require multiple data transmissions and computations, leading to performance degradation. Programmable network cards can optimize the execution process of complex queries through hardware acceleration and parallel processing techniques, such as implementing partial join operations or aggregate calculations at the network card level, thereby reducing data transmission and computation overhead and improving the performance of complex queries.

[0045] 8. Data consistency guarantee In distributed queries, data consistency usually needs to be guaranteed through complex software protocols (such as distributed transactions), increasing system complexity and overhead. Programmable network cards can simplify the guarantee mechanism of data consistency through hardware-level atomic operations and consistency protocols, reduce software-level overhead, and improve query reliability.

[0046] The application is based on the wide applicability of the distributed query system of the programmable network card (Programmable NIC) in many fields. The distributed query system is widely used in large-scale data analysis platforms such as Hadoop, Spark and Presto, etc. The programmable network card can accelerate data filtering, aggregation and sorting operations, reduce CPU and network overheads, and significantly improve query performance and reduce latency in data-intensive queries. It can also be used in real-time data processing systems such as stream processing platforms (Apache Kafka, Flink) and real-time databases (TimescaleDB, InfluxDB), where programmable network cards can preprocess data before it reaches the host, reducing data transmission and computation latency, and supporting low-latency queries suitable for real-time monitoring, alerting and decision-making systems. It can also be used in cloud computing platforms (such as AWS, Azure, Google Cloud) and multi-tenant databases (such as Snowflake, CockroachDB), where programmable network cards ensure that different tenants' query tasks do not interfere with each other through hardware-level resource isolation. It also supports dynamic resource allocation and load balancing to improve resource utilization and system stability. It can also be used in distributed database systems (such as Cassandra, MongoDB, CockroachDB), where programmable network cards in the application can accelerate the execution of distributed queries, reduce data transmission and coordination overheads between nodes, and support efficient data sharding and query optimization to improve database scalability and performance. It can also be used in artificial intelligence and machine learning, such as distributed machine learning platforms (such as TensorFlow, PyTorch) and AI inference systems. Programmable network cards can accelerate data preprocessing and feature extraction, improve model training and inference efficiency, support low-latency queries, and are suitable for real-time AI applications. Its core advantage is that through hardware acceleration and resource optimization, it significantly improves query performance, reduces latency, and supports efficient resource management in complex queries and multi-tenant environments. With the continuous growth of data size and computing demand.

[0047] Further, Field Programmable Gate Array (FPGA) is a highly programmable hardware device that can implement various functions by configuring different logic circuits and algorithms. In a distributed query system, customized query processing modules and algorithms can be implemented on FPGAs to achieve efficient processing and optimization of data. The FPGA-based distributed query system utilizes the high programmability and powerful parallel computing capabilities of FPGAs to provide efficient, flexible, and low-power solutions for large-scale and complex data query tasks. By customizing specific query processing modules and algorithms on FPGAs, the speed and efficiency of data processing can be significantly improved, while system latency and energy consumption are reduced. This architecture is particularly suitable for application scenarios that require high performance and real-time response, such as data center networks, high-performance computing clusters, and real-time data analysis. Although the development difficulty is relatively high, the flexibility and reconfigurability of FPGAs enable them to be optimized and adjusted according to specific needs, achieving optimal performance. In addition, the low-power characteristics of FPGAs make them an ideal choice for energy-sensitive applications, reducing operating costs and environmental impact while ensuring high efficiency.

[0048] Further, the programmable switch-based distributed query system utilizes the powerful capabilities of programmable switches to achieve efficient management and query of distributed data by running customized network processing and query algorithms on the switches. The core of this architecture is the programmable switch, which not only has the basic forwarding function of traditional switches but also supports user-defined network protocols and data processing logic, enabling flexible adaptation to various complex application requirements. In addition, the programmable switch-based distributed query system has high flexibility and scalability, allowing dynamic adjustment and optimization according to specific application scenarios and business requirements. For example, in large-scale data center networks, different query processing modules and strategies can be deployed on switches to achieve fast retrieval and analysis of massive data; in high-performance computing clusters, data transmission paths and scheduling strategies can be optimized to improve the concurrency and throughput of the system.

[0049] Embodiment 3: The embodiment of the present application also provides a distributed query system, comprising: a memory and a processor, the memory stores a computer program run by the processor, and the computer program executes the distributed query method when run by the processor.

[0050] Embodiment 4: The present application also provides a storage medium, which stores a computer program, and the computer program executes the distributed query method when run.

[0051] The above described embodiments are only to illustrate the preferred modes of the present application, and are not intended to limit the scope of the present application. Any modification and improvement made by those skilled in the art to the technical solutions of the present application without departing from the design spirit of the present application shall fall within the protection scope of the present application as defined by the claims.

Claims

1. A distributed query method, characterized by, The method comprises the following steps: Step S1, obtaining a user's query operation request; Step S2, obtaining key-value pair data according to the user's query operation request; meanwhile, processing hash conflicts of error multicast sent data packets, and outputting processed correct multicast sent data packets; Step S3, obtaining batch processed data stream operation requests according to the key-value pair data; Step S4, performing remote memory access processing on the batch processed data stream operation requests, and obtaining correct multicast sent data packets; Step S5, obtaining rate limited data packets according to the correct multicast sent data packets; Step S6, sending the rate limited data packets to a remote memory; Step S7, tracking a transmission state of the data packets; The step S2 comprises: receiving the user's query operation request as input, parsing and executing a read, write or delete operation, and outputting an operation result or key-value pair data to be stored; receiving error data packets in the multicast sent data packets, calculating a storage location through a hash table and processing hash conflicts, and outputting processed correct multicast sent data packets. The step S4 comprises: forwarding the batch processed data stream operation requests to a next network node according to predetermined routing information, and outputting forwarded data packets; copying and sending the forwarded data packets to a plurality of target nodes, and outputting multicast sent data packets, wherein, error data packets are processed for hash conflicts.

2. A distributed query apparatus characterized by comprising: The method comprises the following steps: a network card hardware acceleration module, a KV Store integrated module, a query optimization module, a configurable matching table module, a traffic aggregation module, a sending module and a tracking module integrated on a programmable network card; wherein, the network card hardware acceleration module is configured to obtain a user's query operation request; the KV Store integrated module is configured to obtain key-value pair data according to the user's query operation request; meanwhile, process hash conflicts of error multicast sent data packets, and output processed correct multicast sent data packets; the query optimization module is configured to obtain batch processed data stream operation requests according to the key-value pair data; the configurable matching table module is configured to perform remote memory access processing on the batch processed data stream operation requests, and obtain correct multicast sent data packets; the traffic aggregation module is configured to process the correct multicast sent data packets, and obtain rate limited data packets; the sending module is configured to send the rate limited data packets to a remote memory; the tracking module is configured to track a transmission state of the data packets.

3. The distributed query apparatus of claim 2 wherein, The KV Store module comprises a RISC-V Processor component and a Hash Into Memory component; the RISC-V Processor component is configured to receive the user's query operation request as input, parse and execute a read, write or delete operation, and output an operation result or key-value pair data to be stored; the Hash Into Memory component is configured to receive error data packets in the multicast sent data packets, calculate a storage location through a hash table and process hash conflicts, and output processed correct multicast sent data packets.

4. The distributed query apparatus of claim 3 wherein, The data query optimization module comprises a RateLimiter component and a Batching component, wherein the RateLimiter component is configured to rate limit and convert the key-value pair data, and output a plurality of rate-adjusted data streams; and the Batching component is configured to combine a plurality of data stream operation requests into batch processing, and output batch-processed data stream operation requests.

5. The distributed query apparatus of claim 4 wherein, The configurable matching table module comprises a Forwarder and a Multicaster, wherein the Forwarder is configured to forward the batch-processed data stream operation requests to the next network node according to predetermined routing information, and output forwarded data packets; and the Multicaster is configured to copy and send the forwarded data packets to a plurality of target nodes, and output multicast-sent data packets, wherein correct data packets are sent to the traffic aggregation module, and incorrect data packets are sent to the KV Store module.

6. The distributed query apparatus of claim 5 wherein, The traffic aggregation module comprises a TrafficManager, an Aggregator and an RDMA Limiter, wherein the TrafficManager is configured to process the correct multicast-sent data packets according to priorities, and output scheduled data packets; the Aggregator is configured to aggregate the scheduled data packets; and the RDMA Limiter is configured to control the rate of remote direct memory access operations of the aggregated data packets, and output rate-limited data packets.

7. The distributed query apparatus of claim 6 wherein, The sending module comprises a RoCEv2 Crafter component and an RDMA Emitter component, wherein the RoCEv2 Crafter component is configured to encapsulate the rate-limited data packets into a format conforming to the RoCEv2 protocol, and output encapsulated RoCEv2 data packets; and the RDMA Emitter component is configured to directly send the RoCEv2 data packets to a remote memory, and output successfully sent data packets.

8. A distributed query system, characterized in that, The computer program comprises the following steps: A memory and a processor, wherein the memory stores a computer program which is run by the processor, and the computer program performs the distributed query method of claim 1 when run by the processor.

9. A storage medium, characterized by The storage medium stores a computer program which performs the distributed query method of claim 1 when run.

Images