A multi-modal large language model edge deployment method and system based on dynamic loading and shared memory pool

Abstract

This invention belongs to the field of cable technology, specifically relating to an edge deployment method for multimodal large language models based on dynamic loading and a shared memory pool. The method includes the following steps: S1, lightweight initialization; S2, on-demand dynamic loading; S3, streaming voice pipeline; S4, intelligent resource scheduling. This invention, through a shared memory pool and dynamic loading mechanism, allows a 16GB edge device to run a multimodal model equivalent to a 30GB traditional deployment, resolving the core contradiction between memory and model complexity. Furthermore, by utilizing a user interaction time window to hide model loading, the entire process is seamless with no waiting pop-ups or lag, and the user is completely unaware of the model scheduling process. Additionally, by automatically adapting to edge devices with different memory configurations such as 8GB / 16GB, there is no need to redesign deployment schemes for different hardware. Moreover, through end-to-end local inference, with no data uploads to the cloud, it is stable and usable in weak network / no network environments, perfectly adapting to privacy-sensitive scenarios.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence model deployment technology, specifically relating to a method and system for edge deployment of multimodal large language models based on dynamic loading and shared memory pool. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) have become an important tool for human-computer interaction. Meanwhile, the multimodal capabilities of large models (including text, speech, and image processing) have become a key factor in improving user experience. However, these advanced AI models typically require massive computing resources, often involving billions of parameters, making them difficult to deploy directly on edge computing devices with limited computing power and memory.

[0003] Taking Rockchip as an example, Rockchip's RK series devices, as domestically produced high-performance edge computing chips, have made edge AI applications possible. However, their limited memory is a significant gap compared to current mainstream multimodal large models (which typically require more than 32GB of memory). Existing solutions on the market mainly adopt a cloud inference model, running the model on a cloud server and obtaining the output of the large model through network communication, with the edge device only acting as an input / output terminal. While this approach can solve the resource limitation problem, it brings disadvantages such as high network latency, privacy leakage risks, and offline unavailability, especially performing poorly in scenarios such as smart homes and in-vehicle systems where real-time performance and privacy are sensitive.

[0004] In summary, current multimodal model edge deployment solutions suffer from five irreversible defects, which severely restrict the implementation of the technology.

[0005] 1. Heavy reliance on cloud inference: The model is deployed on cloud servers, and edge devices are only used as input and output terminals. This results in problems such as high network latency, leakage of user data privacy, and complete unavailability in weak network / no network environments. It cannot be adapted to real-time and privacy-sensitive scenarios such as smart homes and vehicles. 2. Limitations of model compression technology: Optimization techniques such as quantization, pruning, and knowledge distillation are only for optimizing a single model and cannot solve the problem of multimodal model collaborative deployment. They will also significantly reduce the model inference accuracy and directly degrade the user interaction experience. 3. Waste of resources in fixed deployment: Traditional solutions load all modal models at once, and unused modalities continue to occupy memory, resulting in hardware resource utilization of less than 30%, and edge devices cannot run large multimodal models at the same time; 4. Laggy loading experience: The model loading process requires a "loading" prompt to pop up, resulting in long waiting times for users and extremely poor smoothness of multimodal interaction; 5. Increased hardware wear and tear: As the NPU operates under continuous high load, the chip temperature rises rapidly, which can easily lead to problems such as overheating and frequency reduction, and shortened equipment lifespan.

[0006] In summary, existing technologies cannot achieve efficient, smooth, secure, and offline local deployment of multimodal large language models on resource-constrained edge devices. There is an urgent need for an innovative deployment solution to overcome the core contradictions of memory limitations, resource utilization, and user experience. Summary of the Invention

[0007] The purpose of this invention is to provide a method and system for edge deployment of multimodal large language models based on dynamic loading and shared memory pools. It aims to address the challenges of limited computing power and memory resources on edge computing devices, particularly the insufficient memory and low computational efficiency faced by Rockchip RK series devices when deploying multimodal large language model systems. This invention proposes a deployment scheme with dynamic model loading and resource optimization. Through innovative memory management mechanisms and asynchronous loading strategies, it maximizes the utilization of limited hardware resources while ensuring a good user experience, enabling complex multimodal AI systems to run efficiently on edge devices.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a method for edge deployment of a multimodal large language model based on dynamic loading and a shared memory pool, comprising the following specific steps: S1. Lightweight initialization: Only load the core weights of the large language model, create a multimodal unified shared memory pool, preheat the NPU computing unit, and minimize startup memory usage; S2. On-demand dynamic loading: Based on the modal type input by the user, the corresponding model is loaded in the background, completely hiding the loading delay by taking advantage of the gaps in user operation; S3, Streaming Speech Pipeline: When the large language model streams and outputs text, the background loads the TTS model, performs semantic segmentation on the text, and then synthesizes speech in a pipeline. S4 Intelligent Resource Scheduling: Release memory as soon as the model is finished, and automatically degrade non-critical functions based on device load, temperature, and memory status to ensure core interactions.

[0009] An edge deployment system for a multimodal large language model based on dynamic loading and a shared memory pool includes: a large language model core module, a multimodal input processing module, a voice interaction module, a memory resource management module, and an inference scheduling module; The core module of the large language model is resident in memory and is responsible for core dialogue reasoning and context maintenance. Multimodal input processing module: identifies input type, schedules image encoder, and loads and infers ASR model; Voice interaction module: Completes TTS model loading, text semantic segmentation, speech synthesis and output; Memory resource management module: Maintains a shared memory pool, enabling memory allocation, reclamation, and fragmentation prevention; Inference and scheduling module: Monitors device status and performs dynamic loading, priority scheduling, and function degradation.

[0010] Compared with the prior art, the beneficial effects of the present invention are: This invention resolves the core contradiction between memory and model complexity by using a shared memory pool and a dynamic loading mechanism, enabling a 16GB edge device to run a multimodal model equivalent to a 30GB traditional deployment.

[0011] This invention hides model loading by utilizing a user interaction time window, resulting in no waiting pop-ups or lag throughout the process, so that users are completely unaware of the model scheduling process.

[0012] This invention automatically adapts to edge devices with different memory configurations, such as 8GB / 16GB, eliminating the need to redesign deployment schemes for different hardware.

[0013] This invention enables full-process local inference without uploading data to the cloud, ensuring stable operation in weak network / no network environments and perfectly adapting to privacy-sensitive scenarios.

[0014] This invention avoids continuous high load on the NPU through dynamic scheduling, effectively controls chip temperature, prevents overheating and frequency throttling, and extends hardware lifespan.

[0015] This invention enables rapid expansion of new modalities such as video, gestures, and sensor data, making it compatible with edge AI applications across all scenarios, including smart homes, automotive, industrial, and medical fields. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This diagram illustrates the steps of a multimodal large language model edge deployment method based on dynamic loading and a shared memory pool.

[0017] Figure 2 This is a schematic diagram of the module composition of a multimodal large language model edge deployment system based on dynamic loading and shared memory pool.

[0018] Figure 3 This is a schematic diagram showing the comparison test data before and after the edge deployment of a multimodal large language model based on dynamic loading and a shared memory pool. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figure 1-3 This invention provides the following technical solution: a method for edge deployment of multimodal large language models based on dynamic loading and shared memory pools, taking Rockchip RK3588 (16GB memory) edge device as an example, including the following steps: 1. System initialization: Obtain LLM, image encoder, ASR, and TTS model files; load the LLM core weights into NPU memory; initialize the shared memory pool with a 16GB hardware configuration; warm up the NPU computing unit; load the model path and memory allocation strategy configuration parameters, keeping only the LLM core model resident.

[0021] The pre-allocation calculation method for the shared memory pool is as follows: During system initialization, the size of the shared memory pool is determined based on a comprehensive consideration of the device's total memory, operating system usage, application memory usage, persistent resident memory usage of large language models, and security redundancy space. Its calculation method can be expressed as: Shared memory pool size = (Total device memory - Memory used by operating system and basic services - Memory used by resident applications - Memory used by resident large language models - Memory for safety redundancy) × Adjustment factor.

[0022] The adjustment coefficient can be set between 0.85 and 0.95 according to the equipment stability requirements. The safety redundancy memory should ideally be 5% to 10% of the total equipment memory to prevent system lag or model loading failure caused by sudden memory fluctuations. The size of the shared memory pool should be no less than the peak memory of the model that occupies the most memory among the image encoder model, speech recognition model, and speech synthesis model, plus 10% to 20% of the switching redundancy space.

[0023] For example, on an edge computing device with 16GB of memory, after deducting the operating system, basic services, persistent application components, and persistent large language model components, most of the remaining available memory can be allocated to a shared memory pool, preferably configured to be around 1.5GB to 2GB, for the dynamic loading and unloading of the image encoder, ASR model, and TTS model. This approach ensures sufficient runtime space for dynamic models while preventing the shared memory pool from becoming too large and crowding out system and core model resources.

[0024] The specific steps for NPU warm-up are as follows: 1. Construct a set of preset test input data and perform one or more empty inferences on the loaded large language model. The test input data can be a short sequence of test text tokens. 2. During the empty inference process, enable the NPU to complete operator initialization, cache establishment, memory mapping, and execution queue establishment so that no cold start is required during subsequent real inference. For image encoder models, ASR models, or TTS models that are frequently used later, a lightweight warm-up can also be performed in the same way when they are first loaded, thereby reducing the additional waiting time when they are used for the first time.

[0025] 2. Dynamic processing of multimodal input: Image input: When a user uploads an image, the background loads the image encoder, which completes the loading process during image selection, upload, and preview. After inferring and generating image features, the memory is immediately released. Voice input: When the user records, the ASR model is loaded in the background, and the memory is released immediately after the speech-to-text conversion is completed; Text input: Directly enters LLM inference without loading additional models.

[0026] 3. Streaming output and speech synthesis pipeline: When LLM generates the first batch of text tokens, it loads the TTS model; it performs semantic segmentation of the text based on punctuation marks, and constructs a text generation-segmentation-speech synthesis pipeline; it outputs the text and synthesized speech synchronously, and decides whether to retain the TTS model after playback based on the frequency of use.

[0027] 4. Dynamic resource scheduling and temperature control: Real-time monitoring of memory, load, and chip temperature; hot loading of high-frequency models; reduction of TTS sampling rate when memory usage exceeds the threshold; insertion of task intervals when temperature is too high; switching to basic dialogue mode when resources are exhausted to ensure core interaction.

[0028] This invention employs a core architecture of persistent core model and dynamic loading of non-core modal models. It leverages four key mechanisms—shared memory pool, interaction time window utilization, streaming pipeline, and intelligent resource scheduling—to achieve efficient deployment of multimodal large language models on resource-constrained edge devices. During system initialization, only the core weights of the large language model are loaded. A unified shared memory pool is adaptively created based on the edge device's hardware configuration, and the NPU computing unit is preheated, reducing startup memory usage from the outset. The system identifies user input types (text, image, and voice) in real time, dynamically loading the corresponding modal processing model in the background. It cleverly utilizes user interaction gaps such as image uploads, voice recordings, and streaming text generation to hide loading delays. Once the model completes inference, memory is immediately released to the shared memory pool, preventing resource idleness.

[0029] During the streaming output stage of the large language model, the system loads the TTS model in the background. For edge chips that do not support dynamic size input, the text is segmented for semantic integrity based on punctuation and paragraph marks, constructing a closed-loop pipeline of text generation-segmentation-speech synthesis. At the same time, the system continuously monitors the device load, chip temperature, and memory usage. A hot-loading caching strategy is established based on the model usage frequency. When memory is low, non-critical components are automatically degraded. When resources are extremely limited, the system switches to the basic dialogue mode. The entire inference process is performed locally without cloud data transmission.

[0030] To ensure stable operation of the system on resource-constrained devices, the system continuously monitors the total memory usage and the remaining percentage of the shared memory pool, and automatically downgrades non-critical components based on the monitoring results. Specifically, the following threshold strategies can be adopted: 1. When the total system memory usage reaches 80% or more, or the remaining space in the shared memory pool is less than 20%, reduce the output quality of the TTS model, such as reducing the sampling rate, reducing the segment length of a single speech synthesis, or reducing the background buffer usage; 2. When the total system memory usage reaches 87.5% or more, or the remaining space in the shared memory pool is less than 12%, unload the currently idle auxiliary models, and appropriately shorten the dialogue context cache length; 3. When the total system memory usage reaches 92.5% or more, or the remaining space in the shared memory pool is less than 8%, pause the real-time speech playback function, retain only text output, and pause the immediate execution of new image understanding or speech recognition tasks, instead queuing them; When the total system memory usage reaches 95% or more, or the remaining space in the shared memory pool is less than 5%, the system enters the basic function mode, retaining only the core dialogue capabilities of the large language model, and shutting down all non-critical auxiliary components until system resources are restored to a safe range.

[0031] The innovation of this invention lies in breaking through the limitations of traditional fixed deployment and single model optimization. It is the first to create a four-dimensional deployment mechanism that combines unified scheduling of shared memory pool, zero-perception loading of interactive time window, streaming collaborative processing, and intelligent temperature control degradation, thus solving the core pain point of insufficient memory and computing power of edge devices.

[0032] This principle enables 16GB edge devices to efficiently run multimodal models equivalent to 30GB traditional deployments, significantly improving hardware resource utilization; zero-perception loading ensures smooth interaction without lag or waiting; local inference enables privacy protection and offline availability; dynamic scheduling avoids continuous high load on the NPU, effectively controls chip temperature, and extends device lifespan, making it widely adaptable to edge AI applications in various scenarios such as smart homes, automotive, and industry.

[0033] Regarding hardware resource utilization, this invention, through dynamic loading and timely release, concentrates computing power on tasks that truly need to be executed, reducing invalid memory usage and NPU computing power waste. This increases the overall NPU utilization rate from 46% to 68%, reduces the average memory usage rate throughout the process from 78% to 56%, and reduces memory usage during task idle time from 13.2GB to 9.7GB. Regarding model loading time, traditional solutions require loading all models at once during system startup, resulting in a long startup wait time. The dynamic loading of this invention saves considerable time. Although the loading time of a single model objectively exists, it is largely covered by the user's natural interaction process, achieving an overall effect where the user has no noticeable waiting time. Figure 3 For comparison of test data: Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method and system for edge deployment of multimodal large language models based on dynamic loading and shared memory pools, characterized in that, Includes the following steps: S1. Lightweight initialization: During system initialization, only the core weights of the large language model are loaded, a unified shared memory pool for multimodal models is created, and the NPU computing unit of the edge device is preheated. S2. On-demand dynamic loading: Based on the type of text, image, or voice modality input by the user, the background dynamically loads the corresponding modality processing model, hiding the model loading delay by utilizing the user interaction operation time window; S3, Streaming Speech Pipeline: During the stage of streaming text output from the large language model, the background loads the TTS model and constructs a text semantic segmentation-speech synthesis pipeline; S4. Intelligent Resource Scheduling: After the modal processing model completes inference, it immediately releases memory to the shared memory pool and performs intelligent resource scheduling and function degradation based on device load, chip temperature, and memory usage.

2. The method and system for edge deployment of multimodal large language models based on dynamic loading and shared memory pools as described in claim 1, characterized in that: The unified shared memory pool in S1 adaptively pre-allocates memory space according to the hardware memory configuration of the edge device, automatically adapting to edge computing devices with 8GB and 16GB memory specifications, thus avoiding memory fragmentation.

3. The method and system for edge deployment of a multimodal large language model based on dynamic loading and shared memory pool as described in claim 1, characterized in that: In S2, the image encoder model is dynamically loaded when the user input is an image, the ASR model is dynamically loaded when the input is speech, and the loading of non-core modal models is skipped when the input is text, thus realizing on-demand loading and immediate release.

4. The method and system for edge deployment of multimodal large language models based on dynamic loading and shared memory pools as described in claim 1, characterized in that: The user interaction operation time window in S2 includes the stages of image selection and upload, voice recording, and large language model text streaming generation. There are no explicit loading prompts throughout the process, achieving zero-perception of model loading.

5. The method and system for edge deployment of a multimodal large language model based on dynamic loading and shared memory pool as described in claim 1, characterized in that: The S3 streaming speech pipeline targets edge chips that do not support dynamic size input. It performs semantic integrity segmentation on the output text of the large language model based on punctuation marks, paragraph marks, and line breaks, and then inputs it into the TTS model to complete speech synthesis.

6. The edge deployment method and system for a multimodal large language model based on dynamic loading and shared memory pool as described in claim 1, characterized in that: In S2, a hot-loading cache strategy is established based on the frequency of modal model usage for dynamic loading on demand. In S4, the high-frequency model used by the streaming speech pipeline resides in memory, while the low-frequency model is released immediately after inference.

7. The method and system for edge deployment of a multimodal large language model based on dynamic loading and shared memory pool as described in claim 1, characterized in that: When the memory pressure of the S4 streaming speech pipeline system exceeds the limit, it automatically degrades non-critical components; when resources are extremely limited, it switches to the basic dialogue mode to ensure that the core interactive capabilities of the large language model are not affected.

8. The method and system for edge deployment of multimodal large language models based on dynamic loading and shared memory pools according to claim 1, characterized in that: The S4 streaming voice pipeline avoids continuous high-load operation of the NPU by dynamically loading the model and scheduling tasks at intervals, and achieves intelligent temperature control of the chip to prevent overheating and frequency reduction.

9. A multimodal large language model edge deployment system based on dynamic loading and shared memory pool, characterized in that: include: The large language model core module includes a multimodal input processing module, a voice interaction module, a memory resource management module, and an inference scheduling module. The memory resource management module is used to maintain a unified shared memory pool and realize memory allocation and reclamation for multimodal models; The inference scheduling module is used to perform dynamic model loading, pipeline scheduling, and dynamic resource adjustment based on user input modalities and device status.

10. The edge deployment system for a multimodal large language model based on dynamic loading and shared memory pool as described in claim 9, characterized in that: The system is compatible with Rockchip RK series, Huawei Ascend, and Cambricon edge computing chips, enabling end-to-end local inference without cloud data transmission and supporting offline operation.

Images