A distributed AI inference distribution network system based on kernel-level packet filtering and a method thereof

Abstract

The application discloses a kind of distributed AI inference distribution network systems and method based on eBPF kernel level packet filtering, it is related to computer network communication and artificial intelligence technical field.The purpose is to solve the problems such as high delay, insufficient security, low resource utilization, poor scalability in existing AI service deployment.System uses four-port architecture (control port, proxy port, public port, API port) to realize function separation, core is deployed XDP program in network driver layer through eBPF technology, completes kernel level API key verification, and the request that does not pass verification is discarded directly in kernel space, improves DDoS protection capability and reduces processing delay;Combined with three-layer authentication mechanism (database authentication→Redis cache authentication→static key bottom-up) to ensure security;Through intelligent load balancing and model routing mechanism, optimal resource allocation is realized according to client health state, model holding condition;Unique connection matching mechanism is used to realize NAT penetration, support intranet AI service access to the outside world;Integrate hybrid P2P architecture, realize client direct connection through libp2p library and STUN / TURN / ICE protocol, further reduce delay and server bandwidth consumption;At the same time, comprehensive system, equipment and network monitoring function is provided, and the index is exposed through RESTful API.The application significantly reduces the response delay of AI inference request, enhances system security and resource utilization, improves scalability and manageability, and is suitable for distributed service deployment scenarios of various AI models.

Description

Technical Field

[0001] This invention relates to the fields of computer network communication technology and artificial intelligence technology, specifically to a distributed AI inference distribution network system and method based on eBPF (extended Berkeley Packet Filter) kernel-level packet filtering technology, and particularly to an AI model inference service distribution system with kernel-level security verification, intelligent load balancing, model routing and P2P hybrid architecture. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLM) and other deep learning models are being used more and more widely in various fields. However, these models typically require a large amount of computing resources, especially GPU resources, making it difficult to run complex AI models directly on local devices. Therefore, deploying AI models in the cloud and providing services via the network has become a common solution.

[0003] AI inference request technical definition: refers to a POST request that conforms to the OpenAI-compatible HTTP format, contains a "model" field, and whose payload is a serialized deep learning model input tensor. Such requests initiated by users need to be transmitted over the network to the server, where the AI ​​model deployed in the cloud or distributed nodes performs inference calculations and returns the result.

[0004] Traditional AI service deployment methods suffer from the following problems: 1. High latency: User requests need to be transmitted over the internet to a remote server before the result is returned, resulting in high response latency. 2. Insufficient security: User data and models may be at risk of leakage during transmission, lacking end-to-end security protection mechanisms. 3. Low resource utilization: Centralized deployment struggles to effectively utilize computing resources distributed across different geographical locations, leading to resource waste. 4. Poor scalability: Traditional architectures struggle to quickly scale up to meet sudden surges in requests. 5. Low authentication efficiency: Traditional API key verification is performed at the application layer, increasing processing latency and making it vulnerable to DDoS attacks.

[0005] Existing technology analysis Currently, there are several AI model-as-a-service solutions on the market, mainly including: 1. AI platforms provided by cloud service providers: such as AWS SageMaker and Google AI Platform. These platforms offer complete machine learning workflows, but users must host their data and models on the cloud service provider's infrastructure, raising concerns about data privacy and security. 2. Open-source model service frameworks: such as TensorFlow Serving and TorchServe. These frameworks are mainly used for model deployment and service-oriented architecture, but lack support for distributed and edge computing. 3. CDN technology: Content Delivery Network (CDN) technology has been widely used to accelerate the distribution of web content, but it is mainly for static content and has limited capabilities for handling dynamic AI inference requests. 4. Traditional reverse proxy and load balancing technologies: such as Nginx and HAProxy. Although they can achieve basic load balancing functions, they have shortcomings in areas such as AI model routing and kernel-level security verification.

[0006] Problems with existing technology Existing technical solutions generally suffer from the following shortcomings: 1. Lack of kernel-level security verification mechanisms: Most existing security verifications are performed at the application layer, increasing processing latency and making them vulnerable to DDoS attacks. Although some solutions employ technologies such as firewalls, these technologies typically operate at the network or transport layer and cannot deeply analyze application layer protocol content. 2. Limited model routing capabilities: Existing technologies lack the ability to intelligently route based on model type and client capabilities, failing to achieve optimal resource allocation. 3. Insufficient NAT traversal capabilities: Many AI services deployed on intranets are difficult for external users to access directly due to the lack of effective NAT traversal technology. 4. Lack of support for hybrid P2P architectures: Most existing solutions adopt a pure client-server architecture, failing to fully utilize direct connections between clients to reduce latency and server bandwidth consumption. 5. Weak resource monitoring and management functions: There is a lack of comprehensive monitoring and management capabilities for distributed AI resources, making it difficult to provide users with detailed resource usage and performance metrics. Summary of the Invention

[0007] Technical issues This invention aims to solve the following technical problems existing in the prior art: 1. How to improve the processing efficiency and response speed of AI inference requests while ensuring security; 2. How to implement kernel-level API key verification to reduce application-layer processing latency and enhance DDoS protection capabilities; 3. How to implement intelligent routing based on AI model type and client capabilities to optimize resource allocation; 4. How to effectively solve the NAT traversal problem of internal network AI services so that external users can access them directly; 5. How to reduce server bandwidth consumption and further reduce request latency through a hybrid P2P architecture; 6. How to provide comprehensive resource monitoring and management functions, and provide users with detailed performance metrics.

[0008] Technical solution To address the aforementioned technical problems, this invention provides a distributed AI inference distribution network system and method based on eBPF kernel-level packet filtering, comprising the following core components and technical solutions:

[0009] This invention employs a unique four-port architecture design, including: 1. Control port: Used for client registration and command issuance, maintaining persistent connections. 2. Proxy port: A temporary connection used for bidirectional data forwarding. 3. Public Port: Serves as the entry point for external users, requiring API key authentication. 4. API Port: Provides RESTful API services for monitoring and management. This architecture achieves functional separation, improving system maintainability and scalability.

[0010] Kernel-level packet filtering technology based on eBPF This invention employs eBPF (extended Berkeley Packet Filter) technology to implement API key verification at the network driver layer, specifically including: 1. XDP (eXpress Data Path) program: The eBPF program is mounted in the earliest stage of the network card driver to ensure that packets are processed before entering the kernel protocol stack; 2. Kernel-level API Key Verification: API key verification is completed by parsing the Authorization header information in the data packet using an eBPF program. The list of valid keys is stored in the eBPF map, which supports atomic updates at runtime via the bpftool tool. Compared to the traditional iptables+ipset scheme, this provides finer-grained kernel-mode access control. It uses BPF_MAP_TYPE_LRU_HASH, with a maximum of 10,000 entries, a key length of 16 bytes, and a value length of 1 byte. 3. Instant Drop Mechanism: Unverified requests are dropped directly in kernel space, preventing invalid requests from entering user space, thus reducing server load and enhancing DDoS protection capabilities. After packets are dropped by XDP_DROP, no logs are generated in the server's user space, thus avoiding the risk of storage leaks caused by log pollution. 4. Dynamic Key Management: The API key list in the eBPF mapping is dynamically updated using the bpftool tool, enabling hot key updates without restarting the service. Atomic replacement using bpf_map_update_elem is employed, ensuring that old connections are unaffected, with an update latency of < 50 µs.

[0011] To demonstrate that "kernel-mode API key verification and model routing calculation" is a necessary technical feature, rather than a commercial gimmick, a comparative experiment was designed: Under the same hardware environment (CPU Intel Xeon 8375C 3.0GHz, memory 32GB, Ubuntu 22.04 LTS, 10 Gbps network bandwidth), "API-Key verification" and "model routing decision" were deployed at different levels, and key performance indicators were tested under the same traffic (100,000 concurrent AI inference requests, 1KB payload per request). The results are shown in the table below: Deployment level CPU utilization P99 Delay SYN-Flood Resistance (Maximum number of attack packets per second) This invention (XDP kernel layer) 11% 18 µs 1.5 million PPS Comparative Example 1 (Nginx Lua Application Layer) 78% 320 µs 300,000 PPS Comparative Example 2 (Userspace Rust Layer) 45% 156 µs 650,000 PPS

[0012] 1. CPU utilization: The XDP kernel layer has a utilization rate of only 11%, far lower than the Nginx Lua layer (78%) and the user-space Rust layer (45%). This is because the kernel mode does not require context switching and directly completes packet parsing and verification at the network driver layer, avoiding the data copying overhead between user mode and kernel mode. 2. P99 Latency: The XDP kernel layer P99 latency is only 18 µs, a 94.3% reduction compared to the Nginx Lua layer (320 µs) and an 88.5% reduction compared to the user-space Rust layer (156 µs). AI inference requests are sensitive to latency. The kernel intercepts invalid requests in advance, preventing them from consuming protocol stack and application layer processing resources. At the same time, model routing decisions are completed in the kernel space, reducing the time for requests to pass through each layer and directly improving the user experience. 3. SYN-Flood Resistance: The XDP kernel layer can withstand 1.5 million PPS of SYN-Flood attacks, which is 5 times that of the Nginx Lua layer (300,000 PPS) and 2.3 times that of the user-space Rust layer (650,000 PPS). Illegal packets are directly dropped in kernel mode, without needing to be passed up to the application layer, thus blocking the consumption of system resources by DDoS attacks at the source and ensuring service availability.

[0013] In summary, performing API key verification and model routing calculation in kernel mode is a necessary technical means to solve the problems of low latency, high security, and low resource consumption for high-concurrency AI inference requests, rather than an optional optimization solution.

[0014] This invention implements an intelligent load balancing and model routing mechanism: 1. Random load balancing algorithm: Randomly selects one of multiple available clients to handle user requests, ensuring even load distribution; 2. Model-aware routing: When a user requests a specific AI model, the system filters out clients that possess that model and selects one for processing; 3. Health Status Monitoring: Continuously monitors client health status via a heartbeat mechanism, automatically removing unusable clients. The heartbeat packet carries GPU utilization, queue length, and inference latency P99 (meaning that 99% of request-response latency does not exceed this number). If there is no response for 3 consecutive times or P99 > 5 s, it is removed; 4. Automatic failover: When a client fails, the system automatically removes it from the pool of available clients and redistributes traffic to other healthy clients.

[0015] 1. Weight calculation formula: Weight w = GPU idle rate × (1 - queue length / maximum queue), where: a. GPU idle rate: The value range is [0,1]. It is derived from the GPU utilization reported by the client heartbeat. GPU idle rate = 1 - GPU utilization (GPU utilization value range is [0,1]). b. Queue length / Maximum queue: Value range [0,1]. The queue length is the number of AI inference tasks currently pending on the client. The maximum queue is preset by the system (default 100). This ratio reflects the congestion level of the client's task queue. c. The weight w ranges from [0,1]. The closer w is to 1, the stronger the client's processing capability and the lighter the load, and the higher the probability of being selected. When w = 0, the client is marked as unavailable and does not participate in load balancing.

[0016] 2. Weight update formula: wn+1 = α×wn + (1- α)×w_current, where: a.wn+1: The updated weight value; b. wn: Historical weight value before the update; c. w_current: Real-time weight value calculated based on current heartbeat data; d.α: Smoothing coefficient, with a value of 0.3, used to reduce the impact of heartbeat data fluctuations on weights, making weight updates more stable.

[0017] Multi-level authentication mechanism This invention employs a three-layer authentication mechanism to ensure system security: 1. Database authentication: The API key validity is first verified by checking the PostgreSQL database; 2. Redis cache authentication: If the database is unavailable, the API key is retrieved from the Redis cache, which has a 5-minute TTL (Time To Live). 3. Static key fallback: If neither the database nor Redis is available, the static API key configured at startup is used as a last resort for verification.

[0018] NAT traversal technology This invention achieves NAT traversal through a unique connection matching mechanism: 1. Proxy Connection Identifier: The server generates a unique proxy_conn_id for each user request. Based on 128-bit UUID4, the collision probability is < 2^-64, and the server maintains an LRU table with a timeout of 30 seconds. 2. Dual-channel connection: The client connects to the server through the proxy port, carrying the proxy_conn_id. The server matches user connections and client connections based on this identifier. 3. Bidirectional data forwarding: Once a connection is successfully matched, the server begins bidirectional data forwarding between the two connections, enabling transparent communication between the internal and external networks.

[0019] Hybrid P2P architecture The hybrid P2P module of this invention includes: a signaling server, a STUN / TURN candidate address collection submodule, and an ICE-based connectivity check submodule, to further optimize performance. 1. libp2p integration: Implements direct connections between clients using Rust's native P2P network library libp2p; 2. NAT Traversal Support: Achieves NAT traversal via STUN / TURN / ICE protocols, and supports AutoNAT to automatically identify NAT types; 3. Intelligent routing selection: Automatically selects P2P direct connection, relay, or TURN server connection based on network conditions; 4. Signaling Server: The server acts as a signaling server, assisting clients in establishing P2P connections.

[0020] Rust code snippets / / Cargo.toml dependency configuration [dependencies] clap = { version = "4.5.13", features = ["derive"]} tokio = {version = "1.39.2", features = ["full"]} tokio-tungstenite = "0.23.1" libp2p = { version = "0.52.0", features = ["tcp", "websocket", "stun", "turn", "ice"]} uuid = { version = "1.8.0", features = ["v4"]} log = "0.4.22" / / Core linker implementation code use libp2p::{ core::{multiaddr::Multiaddr, transport::OrTransport}, dns::TokioDnsConfig, tcp::TokioTcpConfig, websocket::WsConfig, ice::IceTransport, stun::StunServer, turn::TurnServer, swarm::{SwarmBuilder, SwarmEvent}, PeerId, Transport,}; use uuid::Uuid; #[tokio::main] async fn main() -> Result<(), Box<dyn std::error::Error> > { / / Generate unique identifiers for nodes let local_peer_id = PeerId::random(); log::info!("Local Node ID: {}", local_peer_id); / / Building the transport layer (TCP + WebSocket) let transport = OrTransport::new( TokioTcpConfig::new().nodelay(true), WsConfig::new(TokioTcpConfig::new().nodelay(true)), ) .with_dns(TokioDnsConfig::system()?) .map(|either, _ | either) .boxed(); / / Configure STUN / TURN server let stun_server = StunServer::new("stun.l.google.com:19302".parse()?); let turn_server = TurnServer::new( "turn.example.com:3478".parse()?, "turn-username".to_string(), "turn-password".to_string(), ); / / Building ICE transport let ice_transport = IceTransport::new(local_peer_id) .with_stun_server(stun_server) .with_turn_server(turn_server); / / Building the Swarm (P2P network core) let mut swarm = SwarmBuilder::with_tokio_executor(transport, ice_transport, local_peer_id) .build(); / / Listening to local address let listen_addr: Multiaddr = " / ip4 / 0.0.0.0 / tcp / 18080 / ws".parse()?;swarm.listen_on (listen_addr)?; log::info!("Listening address: / ip4 / 0.0.0.0 / tcp / 18080 / ws"); / / Handling network events loop { match swarm.select_next_some().await { SwarmEvent::NewListenAddr { address, ..} => { log::info!("New listening address: {}", address); } SwarmEvent::ConnectionEstablished { peer_id, ..} => { log::info!("Established a connection with node {}", peer_id); / / Generate proxy_conn_id (128-bit UUID4) let proxy_conn_id = Uuid::new_v4().to_string(); log::info!("Connection ID generated: {}", proxy_conn_id);} _ => {} } } }

[0021] This invention provides comprehensive monitoring and observability capabilities: 1. System metrics monitoring: Real-time monitoring of CPU, memory, disk, and network usage; 2. Device performance monitoring: Monitor power consumption of GPU / CPU / ANE (Apple Neural Engine); 3. Network statistics: Provides real-time bandwidth monitoring with session tracing; 4. RESTful API: Expose various monitoring metrics to the outside world through API ports, making it easy to integrate with external monitoring systems.

[0022] Beneficial effects The present invention has the following advantages over the prior art: 1. Reduce latency: By using eBPF kernel-level packet filtering technology, API key verification is moved to the network driver layer, avoiding application layer processing latency. At the same time, P2P direct connection further reduces end-to-end latency. 2. Enhanced security: The kernel-level verification mechanism not only improves verification efficiency but also effectively resists DDoS attacks, while ensuring data transmission security through TLS 1.3 encryption; 3. Optimize resource utilization: Intelligent load balancing and model routing mechanisms ensure that requests are sent to the most suitable client, improving overall resource utilization; 4. Improved scalability: The four-port architecture and hybrid P2P architecture enable the system to easily scale to handle a large number of requests; 5. Improved user experience: NAT traversal technology allows internally deployed AI services to be directly accessed by external users, simplifying the service exposure process; 6. Enhanced manageability: Comprehensive monitoring and management functions provide users with detailed performance metrics and resource usage information, facilitating system maintenance and optimization. Attached Figure Description

[0023] System Architecture Diagram Figure 1 illustrates the overall architecture of the distributed AI inference distribution network system of the present invention. The system adopts a four-port design, including a control port, a proxy port, a public port, and an API port, and the components communicate with each other through a high-speed network connection.

[0024] Four-port architecture diagram Figure 2 illustrates the core four-port architecture design of this invention. The server handles control connections, proxy connections, public user requests, and API management requests through four different ports, achieving clear separation of functions and efficient processing.

[0025] Request processing flowchart Figure 3 illustrates the complete processing flow of a user request in this system. From the moment the user initiates a request, through API key verification, client selection, connection matching, to the final data forwarding, the entire process is clear and straightforward.

[0026] eBPF kernel-level packet filtering principle diagram Figure 4 illustrates the working principle of the eBPF kernel-level packet filtering technology used in this invention. The XDP program intercepts data packets at the network driver layer, parses HTTP header information, verifies the API key, and determines whether to allow the data packet to enter user space. The eBPF program only parses the first packet if the TCP PSH flag is set to 1 and the payload length is greater than 0. If the HTTP header is fragmented, the reassembly is handled by the user-space proxy.

[0027] Hybrid P2P architecture diagram Figure 5 illustrates the hybrid P2P architecture under development in this invention. Based on the traditional client-server architecture, it adds direct connection capabilities between clients. By coordinating the establishment of P2P connections through a signaling server, request latency is further reduced, and server bandwidth consumption is decreased.

[0028] Authentication hierarchy structure diagram Figure 6 illustrates the three-layer authentication mechanism employed in this invention. From database authentication to Redis cache authentication, and then to static key fallback, a multi-layered, highly reliable security authentication system is formed.

[0029] Load balancing and model routing diagram Figure 7 illustrates the intelligent load balancing and model routing mechanism of this invention. The system selects suitable processing nodes based on model requirements and client capabilities, and achieves load balancing through a random selection algorithm.

[0030] Distributed AI inference request processing interaction sequence diagram Figure 8. Sequence diagram of distributed AI inference request processing interaction (including kernel-level verification and P2P direct connection branches).

Claims

1. Independent claim. A method for processing AI inference requests based on kernel-level packet filtering, characterized in that, Includes the following steps: (1) Receive an AI inference request that conforms to the OpenAI-compatible HTTP format, wherein the AI ​​inference request contains a model field and the payload is a serialized deep learning model input tensor; (2) The eBPF program is mounted in the network driver layer through the XDP program to intercept the data packets of the AI ​​inference request. The eBPF program only parses the first packet with the TCP PSH flag set to 1 and the payload length > 0. If the HTTP header is fragmented, it is handed over to the user space proxy to complete the reassembly. (3) The API key is extracted by parsing the Authorization header information in the data packet using the eBPF program; (4) Verify the API key based on the pre-configured eBPF mapping table. The eBPF mapping table adopts the BPF_MAP_TYPE_LRU_HASH type, the key length is 16B, the value length is 1B, the maximum number of entries is 10000, and the whitelist of legitimate keys is stored in the eBPF mapping table. (5) If the API key verification is successful, the data packet is allowed to enter the kernel protocol stack and forwarded to the corresponding processing node; if the verification fails, the data packet is directly discarded in the kernel space and not transmitted to the user space.

2. Independent claim. An AI inference distribution network system based on kernel-level packet filtering, characterized in that, include: (1) Network interception module: configured to mount the eBPF program through the XDP program and intercept the data packets of AI inference request at the network driver layer. The AI ​​inference request is a POST request that conforms to the OpenAI-compatible HTTP format, contains the model field, and has a serialized deep learning model input tensor as the payload. (2) Kernel-level verification module: configured to parse the Authorization header information in the data packet through the eBPF program, extract the API key, and perform key verification based on the eBPF mapping table. The eBPF mapping table adopts the BPF_MAP_TYPE_LRU_HASH type and supports atomic updates through the bpftool tool with an update delay of <50µs. (3) Packet processing module: It is configured to allow verified packets to enter the kernel protocol stack and forward them to the processing node, and to directly discard unverified packets in the kernel space. After the packets are discarded by XDP_DROP, no logs are generated in the server user space.

3. Dependent claims. The AI ​​inference request processing method based on kernel-level packet filtering according to claim 1 is characterized by further comprising the following steps: (1) When it is necessary to establish a data transmission channel for AI inference requests, obtain the network address information of the participants through the signaling server; (2) Perform NAT traversal based on the STUN / TURN / ICE protocol, collect candidate addresses and complete connectivity checks; (3) If the candidate addresses can be directly connected, a P2P direct connection channel is established between the clients, and the AI ​​inference request data and inference results are transmitted through the P2P direct connection channel; (4) If a direct P2P connection cannot be established, data will be transmitted via server relay.

4. Dependent claims. The AI ​​inference distribution network system based on kernel-level packet filtering according to claim 2, characterized in that, It also includes a hybrid P2P module, which comprises a signaling server, a STUN / TURN candidate address collection submodule, and an ICE-based connectivity check submodule, configured as follows: (1) Direct connections between clients are implemented using Rust's native libp2p network library; (2) Supports AutoNAT to automatically identify NAT types and achieve NAT traversal through STUN / TURN / ICE protocols; (3) Automatically select P2P direct connection, relay or TURN server connection mode according to network conditions; (4) Generate a 128-bit UUID4 format proxy_conn_id as a connection identifier. The server maintains an LRU table with a timeout of 30 seconds for connection matching.

5. Dependent claims. The AI ​​inference request processing method based on kernel-level packet filtering according to claim 1, characterized in that, The process includes the following after step (5): (1) Collect the status information of each processing node, including the AI ​​model type supported by the node, GPU utilization, current task queue length and inference latency P99; (2) Calculate the weight of each processing node based on the status information. The weight calculation formula is: w = GPU idle rate × (1 - queue length / maximum queue), where GPU idle rate = 1 - GPU utilization, and the maximum queue is set to 100 by default. (3) A weighted random algorithm is used to select the target processing node. The weight update formula in the weighted random algorithm is: w n ₊1=α×w n +(1-α)×w_current, where α is the smoothing coefficient, with a value of 0.3; (4) If the AI ​​inference request specifies a particular model, the processing nodes that support the model and have a weight > 0 are selected first, and then the target node is selected by weighted random algorithm; if there are no nodes that support the model, the target node is selected from all processing nodes with a weight > 0.

6. Dependent claims. The AI ​​inference distribution network system based on kernel-level packet filtering according to claim 2, characterized in that, It also includes a smart routing module, which is configured as follows: (1) The health status of the processing node is continuously monitored through the heartbeat mechanism. The heartbeat packet carries the GPU utilization, queue length, and inference latency P99. If there is no response for 3 consecutive times or P99>5s, the node is removed. (2) Maintain the list of AI models supported by the node to achieve accurate routing based on model type; (3) Supports automatic failover. When a processing node fails, it is automatically removed from the pool of available nodes and traffic is redistributed to other healthy nodes. (4) Load balancing is achieved based on a weighted random algorithm to ensure that requests are evenly distributed to each processing node.

7. Dependent claims. The AI ​​inference request processing method based on kernel-level packet filtering according to claim 1 is characterized by further including the following monitoring steps: (1) Collect system indicator data in real time, including CPU utilization, memory utilization, disk utilization and network bandwidth usage; (2) Collect equipment indicator data, including power consumption, utilization, memory usage and temperature of GPU / CPU / ANE; (3) Record the processing performance data of AI inference requests, including inference time, request success rate, P99 latency, etc.; (4) Expose the above monitoring metrics to the outside world through RESTful API to support integration with external monitoring systems.

8. Dependent claims. The AI ​​inference distribution network system based on kernel-level packet filtering according to claim 2, characterized in that, It also includes a monitoring and observability module, which is configured as follows: (1) Configure the API port to provide RESTful API services for monitoring and managing the system; (2) Supports real-time bandwidth monitoring with session tracing and records network transmission-related statistical data; (3) The inference delay P99 is calculated using the sliding window algorithm. The sliding window size is set to 1000 samples and the sliding step size is set to 100 samples. (4) Store monitoring data in the database and support historical data query and performance analysis.

9. Dependent claims. The AI ​​inference request processing method based on kernel-level packet filtering according to claim 1 is characterized in that, A four-port architecture is used for data transmission and control, specifically including: (1) The client registration and command issuance are realized through the control port, and a persistent TCP connection is maintained; (2) Establish a temporary connection for bidirectional data forwarding through the proxy port, and close the connection after the AI ​​inference request is processed; (3) The public port serves as the entry point for AI inference requests from external users, and the API key verification of the eBPF program is performed on this port; (4) Provide RESTful API services through the API port for system monitoring and management operations.

10. Dependent claims. The AI ​​inference distribution network system based on kernel-level packet filtering according to claim 2, characterized in that, The system adopts a four-port architecture design, with the four ports being the control port, proxy port, public port, and API port. Each port is physically or logically isolated, with default port numbers of 17000, 17001, 18080, and 18081, corresponding to node communication, data forwarding, request reception, and management and control functions, respectively, to avoid security risks and functional conflicts caused by port reuse.

11. Unity of Invention Statement. Independent claims 1-2 revolve around the core inventive point "eBPF+XDP+API-Key+AI inference request", and all technical features directly serve to solve the core technical problem of "kernel-level security verification and efficient processing of AI inference requests", thus possessing unity of invention.

12. Uniqueness of being statement. Dependent claims 3-10 are all based on the core technical solution of the independent claim, and respectively add technical features such as P2P hole punching, model routing, monitoring API, and four-port architecture. These additional technical features are all optimizations and extensions of the core technical solution, used to further solve derivative problems such as latency optimization, resource scheduling, observability, and architecture scalability, and are closely related to the core technical solution.

Images