Edge device LLM deployment system and method based on memory filling and game optimization

Abstract

The invention relates to an edge device LLM deployment system and method based on memory filling and game optimization, and the system comprises a network state synchronization module, a distributed model deployment module and a model reasoning module. Through dynamic model deployment, cold-start-free model loading and self-adaptive adjustment are realized, and reasoning scheduling is optimized through a layer-device execution graph and shortest path search.

Description

Technical Field

[0001] This invention relates to a technology in the field of neural networks, specifically an edge device LLM deployment system and method based on memory filling and game optimization. Background Technology

[0002] Existing LLM edge deployment solutions have significant limitations: cloud offloading methods increase network dependencies and are not suitable for real-time tasks; model compression methods are prone to performance loss and are difficult to adapt to heterogeneous edge devices; existing edge collaboration frameworks do not fully consider the dynamic changes in device availability and capacity, and cannot simultaneously meet the above three requirements. Summary of the Invention

[0003] To address the aforementioned shortcomings of existing technologies, this invention proposes an edge device LLM deployment system and method based on memory filling and game theory optimization. By integrating model preloading, capacity-aware adjustment, and hierarchical scheduling, it utilizes the idle computing resources of heterogeneous edge devices within the same wireless LAN to achieve cold-start-free model loading and adaptive adjustment through dynamic model deployment. Furthermore, it optimizes inference scheduling through layer-device execution graphs and shortest path search.

[0004] This invention is achieved through the following technical solution:

[0005] This invention relates to an edge device LLM deployment system based on memory filling and game theory optimization, comprising: a network state synchronization module, a distributed model deployment module, and a model inference module, wherein: the network state synchronization module collects and synchronizes the real-time state of devices in the network; the distributed model deployment module preloads the hierarchy of the target large language model into participating devices, determines the initial loading strategy based on device capacity, and adaptively adjusts it according to system dynamics; the distributed model inference module searches for the optimal large language model execution chain based on the preloaded model hierarchy, and then executes the inference task among devices according to the path.

[0006] The network status synchronization module includes: a self-status monitoring unit, a heartbeat beacon broadcasting unit, and a network status monitoring unit. The self-status monitoring unit monitors the available memory, CPU / GPU utilization, and loaded model hierarchies of the mobile device in real time. The heartbeat beacon broadcasting unit encapsulates status data into beacon packets and broadcasts them periodically within the wireless local area network. The network status monitoring unit dynamically updates the list of known peer devices and their status based on the device identifiers and IP addresses in the received beacon packets.

[0007] The beacon packet includes: a timestamp, a device identifier, a device IP address, and a device status, wherein: the timestamp is used to ensure data timeliness, and the device identifier is used to uniquely identify the physical device.

[0008] The distributed model deployment module includes an initial deployment strategy unit, an adaptive adjustment strategy unit, and a memory-filled model loading unit. The initial deployment strategy unit divides the parameters of the large language model to be deployed into several consecutive segments by layer, and then randomly selects at least one segment for loading based on the maximum available free memory capacity. It reallocates the layering of the large language model to be deployed based on real-time free resources to ensure optimal parameter distribution among distributed mobile devices and minimize frequent migrations, thus addressing dynamic changes in the available free memory of mobile devices. The memory-filled model loading unit adaptively and temporarily loads the large language model to be deployed into free memory using a user-invisible memory-filling technology, releasing it immediately when the user needs memory.

[0009] The aforementioned reallocation refers to adjusting the deployment of the large language model's layers on different devices based on the current real-time memory availability of distributed mobile devices, using game theory methods.

[0010] The aforementioned user-invisible memory filling technology refers to: retaining as many parameters of a large language model as possible in memory to achieve fast inference, without interfering with the normal experience of the device's main user. Specifically, this includes:

[0011] A kernel module called Efficient Page Padding is deployed. Using 4KB memory pages as the basic unit, it actively fills the currently free physical memory pages with the partitioned model layer data and maintains a detailed resident record to track these mapping relationships. When the main user starts a new application and needs to allocate memory, this memory module intercepts the kernel's page allocation request through an inline hook mechanism, prioritizes releasing the pages occupied by the model for the application to use and updates the record, thereby ensuring that the smoothness of user operations is not affected.

[0012] Once the application closes and frees up memory, the module immediately repopulates the newly freed-up pages with model data.

[0013] The model inference module includes an optimal execution scheduling unit and an inference task execution unit. The optimal execution scheduling unit searches the hierarchical-device execution graph to determine the optimal inference execution chain based on the large language model inference request. The inference task execution unit performs large language model inference efficiently by alternating between model fragment calculation and instruction packet transmission. Attached Figure Description

[0014] Figure 1 This is a flowchart of the present invention;

[0015] Figure 2 This is a schematic diagram of the model partitioning for an example embodiment;

[0016] Figure 3This is a schematic diagram of the execution path for an example.

[0017] Figure 4 A diagram illustrating efficient page filling technology;

[0018] Figure 5 This is a diagram showing the distribution of experimental equipment for an example.

[0019] Figure 6 This is a schematic diagram illustrating user inference latency in an example.

[0020] Figure 7 This is a schematic diagram illustrating the impact of multiple primary users simultaneously using / closing the application;

[0021] Figure 8 Rate CDF plots were generated for different models in the implementation examples;

[0022] Figure 9 This is a schematic diagram of the VEC energy consumption of different devices in the example. Detailed Implementation

[0023] like Figure 1 As shown, this embodiment relates to an edge device LLM deployment method based on the above system, including:

[0024] Step 1, Initial Deployment Strategy: such as Figure 6 As shown, given a specific large-scale language model (LLM) to be deployed... Where: L is the number of layers in the model; let the set of mobile nodes in the wireless local area network (WLAN) be . Where: N is the number of devices; let This represents the number of LLM layers that each device can accommodate. The core idea of ​​this strategy is to divide the model into multiple equally sized layered segments, and each device randomly selects a segment as the starting point to load the model layers during the initialization phase.

[0025] This embodiment divides the model into k equal-sized fragments based on statistically derived metrics (including the number of active devices in the network and their respective storage capacities). Furthermore, the optimal value of k can be estimated by comprehensively considering the number of model layers to be divided and the total minimum storage capacity derived from statistical analysis of available devices. During initialization, each device randomly selects a starting point within a fragment and loads subsequent model parameters sequentially within the limits of its available memory capacity. For a model with L layers and N devices, where each device can cover an average of c consecutive layers, and a scenario where a device randomly selects an equal-sized fragment for loading but can only partially cover that fragment, find the number of fragments k that maximizes the expected overall coverage. It is known that the LLM model is divided into k equal-sized fragments, each fragment having a length of... Each device randomly selects one of k fragments for deployment. The goal of this embodiment is to determine the optimal value of k to maximize the expected coverage of the entire LLM layer required for real-time inference when a user accesses the network. Since the coverage capability c of a device may not be fully adapted to the fragment length sg, the effective portion contributing to coverage is... .

[0026] The initial deployment strategy specifically includes:

[0027] 1.1 Calculate the contribution of a single device's loaded fragment to the weight coverage of the large model: Coverage refers to the number of layers loaded in the complete large model divided by the total number of layers in the model. Therefore, the expected coverage contribution of a single device to a specific fragment is: Therefore, the total expected coverage of all N devices (normalized by the total number of model layers L) is: .

[0028] 1.2 Maximize Coverage: To maximize total coverage, we need to maximize: Define a function: set up According to the definition, ,but For a fixed value of c, It increases with increasing r, and decreases with increasing sg. When sg is slightly greater than c, At this point, the maximum value is reached. It is close to 1. On the other hand, when sg is much smaller than c, The value is between 0 and The cycle repeats, but because the denominator sg is small, It will become very small.

[0029] 1.3 Solving for the value of k: When sg is the smallest integer greater than or equal to c and divisible by L, To find the maximum value, we need to find the factor of L that is closest to c (whether greater or less than c), because this minimizes the maximum value. This could potentially maximize .set up Given the set of positive integers that divide layer L, the optimal sub-segment length is: The optimal number of partitions is: This option maximizes the expected coverage when devices are evenly and randomly distributed.

[0030] Step 2, Adaptive Adjustment Strategy: Device exiting the network may disrupt the integrity of model parameters or interrupt the existing execution chain, leading to increased inference latency or even complete task failure. To mitigate these impacts, the current deployment scheme needs to be reconfigured immediately to restore optimal system performance. Multiple edge devices should collaboratively participate in the distributed model loading and inference process; their behavior can be modeled as a potential game, specifically including:

[0031] 2.1 With minimizing the overall inference latency and the overhead caused by frequent model parameter switching as the goal of the game theory approach, each device tends to choose the strategy that minimizes its local utility function during the game process, and prioritizes loading the model layer on the current critical inference path. Each device needs to seek a balance between local efficiency (minimizing loading cost) and global cooperation (reducing overall inference latency). In the end, the entire system can self-organize to form an efficient configuration without centralized coordination, which reflects the excellent characteristics of the game theory approach in a distributed learning environment.

[0032] The potential function of the aforementioned potential game: Where: s is the combined strategy of all devices, For the loading cost of device i, The overall inference delay of the system under strategy combination s, For energy consumption, , , These are the weighting coefficients.

[0033] The strategy space of the potential game is: the strategy space of each device i. It consists of a subset of all possible model layers that it can load, satisfying: Where: L is the total number of layers in the model, The memory capacity is constrained for device i. Each device can adjust its strategy by adding, removing, or replacing model parameters, and removing or replacing parameters does not incur additional costs.

[0034] The utility function of the potential game is: The local utility function of device i is defined as: ,in: To start from the current strategy Switch to new strategy The cost, The system inference delay after the policy update, The strategy for all other devices.

[0035] The penalty terms in the potential game include: Loading penalty: the time and energy cost of switching model parameters. Inference penalty: the increased latency caused by the lack of necessary model layers in critical inference paths. Energy penalty: the energy consumption penalty required for inference.

[0036] 2.2 Game Theory Problem Solving: Given a limited policy space for devices, this problem explores how distributed decision-making can optimize the system's potential function. Minimize, thus converging to a stable Nash equilibrium state, specifically including:

[0037] 2.2.1 Global Information Synchronization and Initialization: Each device in the system exchanges global network status information through periodic heartbeat beacon packets, and obtains a global policy profile based on each device. and the current system potential function value It provides each device with the global view needed to make the best local decision.

[0038] The global network status information includes: each device's IP address, available memory, CPU / GPU load, and loaded model layer ID.

[0039] 2.2.2 Device Activation and Policy Space Enumeration: At a certain moment The system activates a device randomly or in a specific order. Activated devices It will enumerate all its feasible strategy space. Its policy space constraint is: each policy It must be a subset of the model layer and satisfy the memory capacity constraint. This generates all possible local policy options, preparing for subsequent utility calculations.

[0040] 2.2.3 Calculation and Evaluation of Local Utility Functions: For the policy space Each candidate strategy ,equipment Calculate the local utility function value if the new strategy is adopted while the strategies of other devices remain unchanged: Calculate the loading cost, that is, estimate the cost from the currently loaded set of layers. Switch to new set After considering the required time and energy consumption, in the new strategy profile The overall latency estimate for performing a typical inference task is then calculated by combining the device's computing power and network communication latency.

[0041] 2.2.4 Optimal Response Strategy Selection: Equipment Compare the utility values ​​of all candidate strategies and select the strategy that minimizes its local utility as its optimal response strategy: From the perspective of the equipment itself, choose the strategy that is most beneficial to it.

[0042] 2.2.5 Global Situation Function Verification and Policy Adoption: Before formally adopting the new policy... Previous equipment This will verify whether this strategy change can reduce the overall situation function. Check if it meets the requirements. The decision-making logic is as follows: Since this game is designed as a potential game, improvements in local utility align with improvements in the overall situation function. Therefore, strategies that minimize local utility typically also lead to a decrease in the overall situation function. This step ensures that the device's selfish behavior ultimately leads to an improvement in global welfare. Policy update: If the verification passes, the device... Then a new model layer is actually loaded, and its new strategy is applied. The beacon packet is broadcast to other devices on the network.

[0043] 2.2.6 System State Iteration and Convergence Judgment: After a device completes its policy update, the system activates the next device, repeating steps 2.2.2 to 2.2.5 and iterating until the system reaches its desired state. When none of the devices can unilaterally change their policies to reduce their local utility (i.e., reduce the global situation function), the system converges to a pure policy Nash equilibrium. At this point, we have... .

[0044] Step 3, as follows Figure 4 As shown, memory-filled model loading is achieved by monitoring the residency status of the model layer and writing it to free memory using the EfficientPagePadding technique. Specifically, in Linux / Unix systems, the kernel's inline hook mechanism is used to load the model layer at the page level, establishing a mapping relationship between the model layer and free memory pages, and recording this relationship in the residency record table. When the number of pages used by the application increases, the physical page allocation process is intercepted, and the residency record of the model layer that was overwritten due to the kernel allocating pages for the application is updated.

[0045] Similarly, when the number of pages occupied by the application decreases, the release process of physical pages is intercepted, and the mapping relationship of the missing model layer is restored.

[0046] Step 4, construct as follows Figure 7 The diagram shows a layer-device execution graph, and the optimal execution chain is determined using a graph-based search method, specifically including:

[0047] 4.1 Construct an execution graph capable of capturing all feasible inference paths: In this graph, each node corresponds to a <layer, device> pair, indicating that a specific model layer is ready to execute on a specific device. For each layer... ,set up This is the set of devices that have preloaded this layer. The directed edges between nodes represent potential transition relationships in the execution sequence, and the edge weights take into account both the computational latency of a device executing a certain layer and the communication latency of transmitting intermediate results between devices.

[0048] 4.2 The problem of searching for the optimal execution chain can be transformed into the problem of finding the shortest path in the execution graph. The classic Dijkstra algorithm is used to solve for the shortest path.

[0049] Step 5: Execution of the reasoning task, specifically including:

[0050] 5.1 Instruction Packet Generation: After determining the optimal execution path for model inference, the system generates an instruction packet to coordinate the distributed computation of model shards. The structure of this instruction packet is as follows: Figure 8 As shown: The packet header contains an identifier specifying the next device to be used for inference, as well as a series of peer instruction fields (which together encode the complete execution plan); the data payload contains the original user prompts (for the initial device) or intermediate features passed between devices.

[0051] 5.2 Model Inference: After receiving the instruction packet, the device first verifies whether the next device ID matches its own identifier. If they match, it parses the first peer instruction to determine the range of model layers to be executed locally, and then updates the next device ID to the device ID specified in the next peer instruction. Next, it performs local inference tasks based on the received instruction packet and calculates and outputs the features.

[0052] 5.3 Command Packet Forwarding: After completing the locally assigned computing task, the device will replace the data fields with output characteristics and forward the updated command packet to the next device in the execution chain.

[0053] Through practical application experiments, the VEC system was implemented on a virtual machine running Ubuntu 20.04. Specifically, the memory filling mechanism was implemented in C and registered as a kernel module, while the model deployment and inference mechanisms were implemented in Python as user-level programs. The inference framework is adapted to devices with different hardware architectures; this invention employs different inference acceleration frameworks to ensure efficient execution of the neural network. For Android smartphones, Alibaba's MNN framework is used—this framework is optimized for device-specific memory constraints, enabling efficient model deployment on mobile devices.

[0054] For Jetson Nano and Jetson OrinNX devices, NVIDIA's TensorRT framework is used—which leverages NVIDIA CUDA technology to optimize neural network models, significantly improving the performance of LLM inference on embedded devices; for Linux and MacOS laptops, the tinygrad and mlx frameworks are used respectively.

[0055] Device discovery: Within a wireless local area network (WLAN), the mDNS protocol is used to implement device discovery. When a new device connects to the network, it broadcasts a multicast data packet containing its device ID and IP address; devices that have registered the same service (i.e., have deployed this invention) receive this broadcast and add the new device to their node list.

[0056] Runtime communication: A combination of UDP heartbeat mechanism and gRPC protocol is used to achieve device status synchronization and model shard execution management. Devices periodically broadcast heartbeat data packets, which include their current load status and information on loaded model layers—load status is divided into four levels based on CPU utilization: idle, low load, medium load, and high load. The heartbeat data packet sending interval is set to 10 seconds. When a device receives a user request and determines the optimal inference execution path, it notifies all collaborating devices via the gRPC protocol. This gRPC data packet contains the node ID and the model shard information to be inferred.

[0057] like Figure 5 As shown in Table 1, this embodiment conducts evaluations on various edge devices, and their hardware configurations are as follows.

[0058] Table 1

[0059] It should be noted that for devices equipped with a dedicated graphics processing unit (GPU), the memory in the table refers to the GPU's maximum video memory (VRAM); for devices with only integrated graphics, this indicator represents the shared memory between the central processing unit (CPU) and the GPU. Experimental Model: This embodiment uses six large language models (LLMs) to conduct extensive experiments to evaluate the performance of the VEC system. Detailed specifications are shown in Table 2.

[0060] Table 2

[0061] Due to the extremely high storage and memory requirements of the 70 billion parameter (70B) and 72 billion parameter (72B) models, deploying their full versions on edge devices is impractical. Therefore, a 4-bit quantized version was used instead in the experiment. All six models support a maximum input length of 32,768 characters, but to ensure experimental feasibility, the input length was limited to 1,024 characters in this embodiment. The evaluation metric is to comprehensively evaluate the efficiency of the VEC system in LLM inference performance. This embodiment uses the token generation rate (TokenGenerationRate) as the core metric, which is the average response time for generating answers during inference. Comparison Method: This embodiment compares the VEC system with two existing edge collaborative inference methods, as follows: EXO: This method uses a simple task scheduling strategy and does not consider the impact of cold start time. EdgeShard: A scheduling strategy based on dynamic programming to find the optimal task execution sequence, but it does not explicitly consider device join / leave events and dynamic resource changes.

[0062] This embodiment uses the DeepSeek and Llama models to evaluate the impact of changes in the number of devices on system latency. Specifically, the experiment is conducted by observing latency changes when mobile phones and desktop devices join or leave the system. Three groups of devices were used to simulate network scenarios with different numbers of devices: 5-device scenario: Huawei MatePad, Huawei P9, Jetson Nano, Xiaomi 14, RTX3060 laptop; 10-device scenario: Huawei MatePad, Huawei P9, Jetson Nano, Xiaomi 14, RTX3060 laptop, Xiaomi 6, Xiaomi Pad 6, Jetson NX, Redmi K70, Macbook M2; 15-device scenario: in addition to the above 10 devices, vivo X9, Pixel 3, Macbook M3, and Macbook M1 were added. To ensure the validity of the test, the Redmi Note 9 (smartphone) and RTX3060 laptop were selected as representative test devices—both have strong performance and can effectively evaluate the impact of a single device joining or leaving the network on the overall latency. The latency metrics involved in the experiment include: Adjustment Time, Inference Time, and Overall Latency. For mobile devices, the metrics are denoted as Pho.Adj, Pho.Inf, and Pho.Oa, respectively, while the metrics for desktop devices follow a similar naming convention.

[0063] like Figure 8As shown, the overall response latency of the system to user inference is mainly dominated by inference time, with adjustment time accounting for only 15.6% of the total latency on average. With the increase in the number of devices, the impact of device addition and removal on the total latency gradually weakens in both the DeepSeek and Llama models—this is because more devices improve system stability, thereby reducing interference caused by the dynamic changes of individual devices. When the number of devices reaches 15, the total latency drops to 52% of that in the 5-device scenario. These results demonstrate that the system can effectively cope with the impact of dynamic changes in device resources; furthermore, thanks to adaptive deployment strategy adjustments, the system's robustness is further enhanced as the number of devices in the network increases.

[0064] This embodiment further evaluates the impact of dynamic resources in scenarios with 10 and 15 devices. During system operation, 1-5 devices are randomly selected to simultaneously run one of three applications (YouTube, TikTok, or WeChat), and the total latency (including tuning latency and inference latency) is measured under different scenarios. Figure 7 As shown, with the increase in the number of concurrent applications, all latency metrics show a moderate upward trend, but the increase does not exceed 27.1%. In addition, different applications have different impacts on system latency: compared with WeChat, YouTube and TikTok consume significantly more video memory (VRAM) resources, thus causing greater overhead to system performance.

[0065] like Figure 9 As shown, once these applications terminate, system resources can be quickly restored thanks to the memory padding mechanism, and the total latency returns to the baseline level. Furthermore, as the number of devices increases, the impact of dynamic resource fluctuations on system latency gradually decreases. These findings indicate that although dynamic resource changes caused by user behavior can affect the system, the combination of the memory padding mechanism and adaptive deployment strategy adjustments can effectively improve system robustness as the number of devices increases.

[0066] Compared to existing technologies, this invention achieves a significant reduction in end-to-end latency through three core technological innovations: First, its innovative distributed model deployment mechanism (combining a statistically based optimal sharding strategy with game theory-driven adaptive adjustment) effectively addresses the dynamic nature of mobile devices, minimizing the impact of device departures or resource fluctuations, resulting in an overall latency reduction of approximately 48% for a system with 15 devices compared to 5 devices. Second, the user-unobtrusive kernel-level memory filling technology maximizes the utilization of idle memory without affecting the main user experience, ensuring that model parameters reside in memory for rapid response; even when multiple devices run high-load applications simultaneously, the system latency increase is controlled within 27.1%. Finally, the optimal inference chain search based on the execution graph and Dijkstra's algorithm precisely schedules heterogeneous device resources, avoiding the cold start and scheduling overhead of traditional methods. Experiments demonstrate that this system reduces end-to-end latency by 44% to 83% compared to existing methods (such as EXO and EdgeShard) in diverse real-world scenarios, while maintaining energy consumption levels comparable to regular web browsing, achieving a balance between low latency, high robustness, and efficient resource utilization. It can utilize field devices to self-organize into temporary high-performance computing nodes, providing critical offline intelligent decision support for fields such as disaster relief, telemedicine, and education. It lays a key technological foundation for achieving large-scale intelligent collaboration and computing power sharing of devices in the future Internet of Things and 6G environment, and is expected to become one of the core enabling technologies to promote the ubiquitous development of edge intelligence.

[0067] The above-described specific implementations can be partially adjusted by those skilled in the art in different ways without departing from the principles and purpose of the present invention. The scope of protection of the present invention is defined by the claims and is not limited to the above-described specific implementations. All implementation schemes within the scope of the claims are bound by the present invention.

Claims

1. An edge device LLM deployment system based on memory filling and game theory optimization, characterized in that, include: The system comprises a network status synchronization module, a distributed model deployment module, and a model inference module. Specifically: the network status synchronization module collects and synchronizes the real-time status of devices in the network; the distributed model deployment module preloads the layers of the target large language model into the participating devices, determines the initial loading strategy based on the device capacity, and adaptively adjusts it according to the system dynamics; the distributed model inference module searches for the optimal execution chain of the large language model based on the preloaded model layers, and then executes the inference task on the devices according to the path.

2. The edge device LLM deployment system based on memory filling and game theory optimization according to claim 1, characterized in that, The network status synchronization module includes: a self-status monitoring unit, a heartbeat beacon broadcasting unit, and a network status monitoring unit. Specifically: the self-status monitoring unit monitors the available memory, CPU / GPU utilization, and loaded model hierarchies of the mobile device in real time; the heartbeat beacon broadcasting unit encapsulates status data into beacon packets and broadcasts them periodically within the wireless local area network; and the network status monitoring unit dynamically updates the list of known peer devices and their status based on the device identifiers and IP addresses in the received beacon packets. The beacon packet includes: a timestamp, a device identifier, a device IP address, and a device status, wherein: the timestamp is used to ensure data timeliness, and the device identifier is used to uniquely identify the physical device.

3. The edge device LLM deployment system based on memory filling and game theory optimization according to claim 1, characterized in that, The distributed model deployment module includes: an initial deployment strategy unit, an adaptive adjustment strategy unit, and a memory-filled model loading unit. The initial deployment strategy unit divides the parameters of the large language model to be deployed into several consecutive segments by layer, and then randomly selects at least one segment for loading based on the maximum available free memory capacity. It reallocates the layering of the large language model to be deployed based on real-time free resources to ensure optimal parameter distribution among distributed mobile devices and minimize frequent migrations, thus addressing dynamic changes in the available free memory of mobile devices. The memory-filled model loading unit adaptively and temporarily loads the large language model to be deployed into free memory using a user-invisible memory-filling technology, releasing it immediately when the user needs memory. The aforementioned reallocation refers to adjusting the deployment of the large language model's layers on different devices based on game theory methods, according to the current real-time memory availability of distributed mobile devices. The aforementioned user-invisible memory filling technology refers to: retaining as many parameters of a large language model as possible in memory to achieve fast inference, without interfering with the normal experience of the device's main user. Specifically, this includes: The kernel module is deployed and uses 4KB memory pages as the basic unit. It actively fills the currently free physical memory pages with the divided model layer data and maintains a detailed resident record to track these mapping relationships. When the main user starts a new application and needs to allocate memory, the memory module intercepts the kernel's page allocation request through an inline hook mechanism, prioritizes the release of pages occupied by the model for the application to use and updates the record, thereby ensuring that the smoothness of user operation is not affected. Once the application closes and frees up memory, the module immediately repopulates the newly freed-up pages with model data.

4. The edge device LLM deployment system based on memory filling and game theory optimization according to claim 1, characterized in that, The model inference module includes an optimal execution scheduling unit and an inference task execution unit. The optimal execution scheduling unit searches the hierarchical-device execution graph to determine the optimal inference execution chain based on the large language model inference request. The inference task execution unit performs large language model inference efficiently by alternating between model fragment calculation and instruction packet transmission.

5. A method for deploying an edge device LLM based on memory filling and game theory optimization according to any one of claims 1-4, characterized in that, include: Step 1, Initial Deployment Strategy: Given a specific large-scale language model (LLM) to be deployed. Where: L is the number of layers in the model; the set of mobile nodes in the wireless local area network is Where: N is the number of devices; let This represents the number of LLM layers that each device can accommodate; Step 2, Adaptive Adjustment Strategy: Device exiting the network may damage the integrity of model parameters or interrupt the existing execution chain, leading to increased inference latency or even complete task failure. To mitigate these impacts, the current deployment scheme needs to be reconfigured immediately to restore the optimal performance of the system. Multiple edge devices should be involved in the distributed model loading and inference process, and their behavior can be modeled as a potential game. Step 3: Monitor the residency status of the model layer through efficient page filling technology and write it to free memory to achieve memory-filled model loading. Specifically, in Linux / Unix systems, the kernel's inline hook mechanism is used to load the model layer at the page level, establish a mapping relationship between the model layer and free memory pages, and record it in the residency record table. When the number of pages used by the application increases, the physical page allocation process is intercepted, and the residency record of the model layer that is overwritten due to the kernel allocating pages for the application is updated. Step 4: Construct a layer-device execution graph and determine the optimal execution chain using a graph-based search method; Step 5: Execution of the reasoning task.

6. The edge device LLM deployment method according to claim 5, characterized in that, Step 1 specifically includes: 1.1 Calculate the contribution of a single device to the weight coverage of the large model: Coverage refers to the number of layers loaded in the complete large model divided by the total number of layers in the model. Therefore, the expected coverage contribution of a single device to a specific segment is: Therefore, the total expected coverage of all N devices is: ; 1.2 Maximize Coverage: To maximize total coverage, we need to maximize: The constraints are: ,function: set up , , For a fixed c, It increases with the increase of r, and decreases with the increase of sg. When sg is slightly greater than c, At this point, the maximum value is reached. When sg is much smaller than c, the value is close to 1. The value is between 0 and Loop between; 1.3 Solving for the value of k: When sg is the smallest integer greater than or equal to c and divisible by L, To obtain the maximum value, we need to find the factor of L that is closest to c, because this minimizes the maximum value. This could potentially maximize ,set up Given the set of positive integers that divide layer L, the optimal sub-segment length is: The optimal number of partitions is: This option maximizes the expected coverage when devices are evenly and randomly distributed.

7. The edge device LLM deployment method according to claim 5, characterized in that, Step 2 specifically includes: 2.1 With minimizing the overall inference latency and the overhead caused by frequent model parameter switching as the goal of the game, each device tends to choose the strategy that minimizes its local utility function during the game process, and prioritizes loading the model layer on the current critical inference path. Each device needs to seek a balance between local efficiency and global cooperation. In the end, the entire system can self-organize to form an efficient configuration without centralized coordination, which reflects the excellent characteristics of the game in the distributed learning environment. 2.2 Game Theory Problem Solving: Given a limited policy space for devices, this problem explores how distributed decision-making can optimize the system's potential function. Minimize, thus converging to a stable Nash equilibrium state, specifically including: 2.2.1 Global Information Synchronization and Initialization: Each device in the system exchanges global network status information through periodic heartbeat beacon packets, and obtains a global policy profile based on each device. and the current system potential function value Provide each device with the global view needed to make the best local decision; The global network status information includes: each device's IP address, available memory, CPU / GPU load, and loaded model layer ID; 2.2.2 Device Activation and Policy Space Enumeration: At a certain moment The system activates a device randomly or in a specific order. Activated devices It will enumerate all its feasible strategy space. Its policy space constraint is: each policy It must be a subset of the model layer and satisfy the memory capacity constraint. This generates all possible local policy options, preparing for subsequent utility calculations; 2.2.3 Calculation and Evaluation of Local Utility Functions: For the policy space Each candidate strategy ,equipment Calculate the local utility function value if the new strategy is adopted while the strategies of other devices remain unchanged: Calculate the loading cost, that is, estimate the cost from the currently loaded set of layers. Switch to new set After considering the required time and energy consumption, in the new strategy profile Below, the overall latency estimate for performing a typical inference task is calculated by combining the device's computing power and network communication latency. 2.2.4 Optimal Response Strategy Selection: Equipment Compare the utility values ​​of all candidate strategies and select the strategy that minimizes its local utility as its optimal response strategy: From the perspective of the equipment itself, choose the strategy that is most beneficial to it; 2.2.5 Global Situation Function Verification and Policy Adoption: Before formally adopting the new policy... Previous equipment This will verify whether this strategy change can reduce the overall situation function. Check if it meets the requirements. The decision-making logic is as follows: Since this game is designed as a potential game, the improvement of local utility is consistent with the improvement of the global situation function. Therefore, the strategy that minimizes local utility usually also leads to a decrease in the global situation function. This step ensures that the selfish behavior of the device will ultimately lead to an improvement in global welfare. Strategy update: If the verification passes, the device... Then a new model layer is actually loaded, and its new strategy is applied. Broadcast to other devices on the network via beacon packets; 2.2.6 System State Iteration and Convergence Judgment: After a device completes its policy update, the system activates the next device, repeating steps 2.2.2 to 2.2.5 and iterating until the system reaches its desired state. When none of the devices reduces their local utility (i.e., reduces the global situation function) by unilaterally changing their policies, the system converges to a pure policy Nash equilibrium, at which point we have: .

8. The edge device LLM deployment method according to claim 7, characterized in that, The potential function of the aforementioned potential game: Where: s is the combined strategy of all devices, For the loading cost of device i, The overall inference delay of the system under strategy combination s, For energy consumption, , , These are the weighting coefficients; The strategy space of the potential game is: the strategy space of each device i. It consists of a subset of all possible model layers that it can load, satisfying: Where: L is the total number of layers in the model, Given the memory capacity constraint of device i, each device can adjust its strategy by adding, removing or replacing model parameters, and removing and replacing parameters will not incur additional costs. The utility function of the potential game is: The local utility function of device i is defined as: ,in: To start from the current strategy Switch to new strategy The cost, The system inference delay after the policy update, For all other devices; The penalty terms in the potential game include: loading penalty: the time and energy cost of switching model parameters; inference penalty: the increased latency caused by the lack of necessary model layers in the critical inference path; and energy consumption penalty: the energy consumption penalty required for inference.

9. The edge device LLM deployment method according to claim 7, characterized in that, Step 4 specifically includes: 4.1 Construct an execution graph capable of capturing all feasible inference paths: In this graph, each node corresponds to a <layer, device> pair, indicating that a specific model layer is ready to execute on a specific device. For each layer... ,set up For the set of devices that have preloaded this layer, the directed edges between nodes represent potential transition relationships in the execution sequence. The weight of the edges takes into account both the computational latency of a device executing a certain layer and the communication latency of transmitting intermediate results between devices. 4.2 The problem of searching for the optimal execution chain can be transformed into the problem of finding the shortest path in the execution graph. The classic Dijkstra algorithm is used to solve for the shortest path.

10. The edge device LLM deployment method according to claim 7, characterized in that, Step 5 specifically includes: 5.1 Instruction Packet Generation: After determining the optimal execution path for model inference, the system generates an instruction packet to coordinate the distributed computing of model shards. The structure of this instruction packet is as follows: The packet header contains an identifier specifying the next device to perform inference, as well as a series of peer instruction fields; the data payload contains the original user prompts or intermediate features passed between devices. 5.2 Model Inference: After receiving the instruction packet, the device first verifies whether the next device ID matches its own identifier. If it matches, it parses the first peer instruction to determine the range of model layers to be executed locally, updates the next device ID to the device ID specified in the next peer instruction, and then performs local inference tasks based on the received instruction packet to calculate and output the features. 5.3 Command Packet Forwarding: After completing the locally assigned computing task, the device will replace the data fields with output characteristics and forward the updated command packet to the next device in the execution chain.

Images