Deployment method and system of large language model, electronic equipment and storage medium

Abstract

The invention relates to a deployment method and system of a large language model, electronic equipment and a storage medium. The method comprises the steps that a model deployment task for at least one target large model is received, a target deployment node for executing the model deployment task is selected from a plurality of model deployment nodes, a target reasoning framework is integrated on the target deployment node, and if it is detected that a model file needed by the target large model exists in a local cache, the target large model is deployed. If detecting that the model file required by the target large model does not exist in the local cache, obtaining the model file from the distributed storage module, loading the model file to the target reasoning framework, starting the reasoning service of the target reasoning framework on the target deployment node, importing the target large model, and obtaining the model file required by the target large model on the target deployment node. And completing a model deployment task. According to the scheme provided by the invention, the deployment efficiency and the deployment effect of the large language model can be remarkably improved.

Description

Technical Field

[0001] This application relates to the field of model deployment technology, and in particular to methods, systems, electronic devices and storage media for deploying large language models. Background Technology

[0002] With the rapid development of artificial intelligence technology, the number of parameters in large language models (LLMs) is growing exponentially, and the size of their corresponding model files is expanding rapidly, often reaching tens or even hundreds of gigabytes. This exponential growth poses a severe challenge to the storage, management, and deployment of large language models.

[0003] During the deployment of large models, mainstream inference frameworks such as Ollama, vLLM (Virtual Large Language Model), and SGLang (Structured Generation Language) place extremely high demands on storage systems when processing ultra-large-scale model files.

[0004] However, traditional storage systems have many obvious drawbacks when dealing with such scenarios. For example, in terms of performance, they are prone to I / O (Input / Output) bottlenecks and slow model loading speeds. In terms of cost, hardware procurement and maintenance costs are high, and storage utilization is low. In multi-cloud and hybrid cloud environments, data management is complex, cross-cloud deployment compatibility is poor, and different inference frameworks suffer from data inconsistencies and performance instability when combined with traditional storage.

[0005] It can be seen that current storage systems are unable to handle extremely large model files, thus failing to meet the need for efficient deployment of large language models. Summary of the Invention

[0006] To address or partially address the problems existing in related technologies, this application provides a method, system, electronic device, and storage medium for deploying large language models, which can efficiently deploy large language models.

[0007] The first aspect of this application provides a method for deploying a large language model, applied to a large language model deployment system, the large language model deployment system including multiple model deployment nodes and a distributed storage module, the method comprising: The system receives a model deployment task for at least one target large model, and selects a target deployment node from multiple model deployment nodes to execute the model deployment task; the target deployment node integrates a target inference framework. If the model file required by the target large model is detected in the local cache, then the model file is loaded into the target inference framework; If the local cache does not contain the model file required by the target large model, the model file is retrieved from the distributed storage module and loaded into the target inference framework. On the target deployment node, the inference service of the target inference framework is started, and the target large model is imported to complete the model deployment task.

[0008] In one instance, before receiving a model deployment task for at least one target large model, the process includes: Create the distributed storage module, which includes a metadata engine and an object storage unit; The metadata engine is used as the metadata storage space of the distributed storage module, and the object storage unit is used as the data block storage space of the distributed storage module. Establish the association between the metadata storage space and the data block storage space.

[0009] In one instance, before receiving a model deployment task for at least one target large model, the process includes: Install the target inference framework on each model deployment node; Configure the model storage path for the target inference framework; The model storage path is mapped to the local cache directory of the distributed storage module, so that the target inference framework can perform read and write operations on the local cache of the distributed storage module.

[0010] In one instance, receiving a model deployment task for at least one target large model, and selecting a target deployment node from a plurality of model deployment nodes to execute the model deployment task, includes: Receive model deployment tasks for at least one target large model; Analyze the model deployment task to extract the model type and deployment resources of the target large model; Based on the model type and the deployment resources, a target deployment node is selected from multiple model deployment nodes to execute the model deployment task.

[0011] In one instance, if it is detected that the local cache does not contain the model file required by the target large model, then retrieving the model file from the distributed storage module includes: If the local cache does not contain the model file required by the target large model, a data query request is generated. In response to the data query request, the model file is obtained from the metadata storage space and the data block storage space; The model file is stored in the local cache so that the target inference framework can retrieve the model file from the local cache.

[0012] In one instance, the distributed storage module includes a metadata engine and an object storage unit. The step of retrieving the model file from the metadata storage space and the data block storage space in response to the data query request includes: In response to the data query request, obtain the login information of the logged-in user; Based on the user login information, the legitimacy of the logged-in user is verified; When the legality verification is successful, the location information of the model file is extracted from the metadata storage space, and the model file is read from the data block storage space based on the location information.

[0013] In one instance, it also includes: Upon receiving the version update request for the target large model, the updated model file is obtained; the model file includes change metadata and change data blocks; The changed metadata is encrypted and written to the metadata storage space of the target deployment node, and the changed data block is encrypted and written to the data block storage space of the target deployment node; Based on the pre-built mirror relationship, the changed metadata and the changed data blocks are synchronized to other model deployment nodes.

[0014] A second aspect of this application provides a large language model deployment system, the large language model deployment system comprising multiple model deployment nodes and a distributed storage module, including: The task receiving module is used to receive model deployment tasks for at least one target large model, and select a target deployment node from multiple model deployment nodes to execute the model deployment task; the target deployment node integrates a target inference framework; The first model file loading module is used to load the model file required by the target large model into the target inference framework if the local cache is detected to contain the model file required by the target large model. The second model file loading module is used to retrieve the model file from the distributed storage module and load the model file into the target inference framework if the local cache does not contain the model file required by the target large model. The model deployment module is used to start the inference service of the target inference framework on the target deployment node and import the target large model to complete the model deployment task.

[0015] A third aspect of this application provides an electronic device, comprising: Processor; and A memory that stores executable code, which, when executed by the processor, causes the processor to perform the method described above.

[0016] A fourth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method described above.

[0017] The fifth aspect of this application provides a computer program product comprising computer instructions that, when executed by a processor, implement the method described above.

[0018] The technical solution provided in this application may include the following beneficial results: This application applies to a large language model deployment system, which includes multiple model deployment nodes and a distributed storage module. The system receives model deployment tasks for at least one target large model and selects a target deployment node from among the multiple deployment nodes to execute the task. The target deployment node integrates a target inference framework. If the local cache contains the model file required by the target large model, the model file is loaded into the target inference framework. If the local cache does not contain the required model file, the model file is retrieved from the distributed storage module and loaded into the target inference framework. On the target deployment node, the inference service of the target inference framework is started, and the target large model is imported to complete the model deployment task.

[0019] Compared to related technologies, the technical solution of this application offers two key advantages: First, it utilizes a distributed storage module to uniformly manage model files, enabling cross-node and cross-cloud environment data access. The target deployment node can automatically retrieve model files, reducing manual intervention and effectively lowering operational complexity. Second, when a model file exists in the local cache, it is directly loaded into the target inference framework, avoiding repeated data readings and improving overall system throughput. When the model file is not present in the local cache, it is retrieved from the distributed storage module, ensuring reliable data retrieval even during initial deployment or cache misses, thus enhancing the automation and reliability of model deployment. Therefore, the technical solution of this application significantly improves the deployment efficiency and effectiveness of large language models.

[0020] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0021] The above and other objects, features and advantages of this application will become more apparent from the more detailed description of exemplary embodiments thereof in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments thereof.

[0022] Figure 1 This is a flowchart illustrating a method for deploying a large language model according to an embodiment of this application; Figure 2 This is a schematic diagram of a large language model deployment system shown in an embodiment of this application; Figure 3 This is a schematic diagram of server communication shown in an embodiment of this application; Figure 4 This is another flowchart illustrating a method for deploying a large language model, as shown in an embodiment of this application. Figure 5 This is a schematic diagram illustrating the model file transfer process in an embodiment of this application; Figure 6 This is a schematic diagram illustrating the structure of a large language model deployment system according to an embodiment of this application; Figure 7 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application. Detailed Implementation

[0023] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0024] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0025] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0026] With the rapid development of artificial intelligence technology, the number of parameters in large language models (LLMs) is growing exponentially, and the size of their corresponding model files is expanding rapidly, often reaching tens or even hundreds of gigabytes. This exponential growth poses a severe challenge to the storage, management, and deployment of large language models.

[0027] During the deployment of large models, mainstream inference frameworks such as Ollama, vLLM (Virtual Large Language Model), and SGLang (Structured Generation Language) place extremely high demands on storage systems when processing ultra-large-scale model files.

[0028] However, traditional storage systems have many obvious drawbacks when dealing with such scenarios.

[0029] From a performance perspective, traditional local storage systems are limited by the hardware performance of a single node, and are prone to I / O bottlenecks when concurrently loading large model files. This leads to a significant decrease in model loading speed, severely impacting the deployment efficiency of large models and inference response speed. For example, when multiple inference instances access the same large model file simultaneously, traditional storage typically cannot provide sufficient bandwidth and concurrent processing capabilities, resulting in a substantial increase in loading latency.

[0030] From a cost perspective, to meet the storage needs of large models, enterprises often need to purchase a large number of high-performance storage devices, such as high-end SSDs (Solid-State Drives). This not only increases hardware investment but also brings high operation and maintenance costs. Due to the large size of large model files, traditional storage systems have low space utilization, which easily leads to a lot of storage waste and further increases the total cost of ownership.

[0031] In multi-cloud or hybrid cloud environments, the drawbacks of traditional storage become more apparent. The distribution, synchronization, and management of large-scale model data across different cloud platforms become complex, making efficient sharing and unified management difficult. Furthermore, differences in storage interfaces and protocols across different cloud environments increase compatibility challenges and implementation difficulties in cross-cloud deployments.

[0032] Furthermore, mainstream inference frameworks also have pain points when storing and loading large model files. The Ollama framework stores model files in a local directory by default, requiring manual file copying during multi-node deployments, which is not only time-consuming but also prone to data inconsistencies. While the vLLM framework supports distributed deployment, it has extremely high requirements for the performance and consistency of the storage system, which traditional storage systems often struggle to meet, easily leading to read errors or performance fluctuations. The SGLang framework is extremely sensitive to storage throughput and latency when handling very large model files; traditional storage systems struggle to provide stable high-performance support, thus affecting the model's inference performance.

[0033] It can be seen that current storage systems are unable to handle extremely large model files, thus failing to meet the need for efficient deployment of large language models.

[0034] To address the aforementioned problems, this application provides a method for deploying large language models, enabling efficient deployment of such models. The technical solutions of this application's embodiments are described in detail below with reference to the accompanying drawings.

[0035] See Figure 1 , Figure 1 This is a flowchart illustrating a method for deploying a large language model according to an embodiment of this application. The method is applied to a large language model deployment system, which includes multiple model deployment nodes and a distributed storage module.

[0036] Step 101: Receive a model deployment task for at least one target large model, select a target deployment node from multiple model deployment nodes to execute the model deployment task, and integrate the target inference framework on the target deployment node.

[0037] In this embodiment, the large language model deployment system can receive model deployment tasks for at least one target large model and select a target deployment node from multiple model deployment nodes to execute the model deployment task. After selection, the target inference framework integrated on the target deployment node is determined so that the corresponding target large model can be loaded and run in subsequent steps.

[0038] Optionally, a large language model deployment system refers to a system for managing, deploying, and monitoring large language models. It is mainly used to coordinate resource allocation and task execution in a multi-node computing environment, and can monitor the running status of each model deployment node in real time.

[0039] A model deployment task refers to a user-initiated deployment instruction for a specific target large model. This task must include at least the model type, model identifier, and deployment resources required by the model.

[0040] The target large model refers to the large language model that needs to be deployed. It has a large number of parameters, a relatively complex network structure, and rich semantic understanding and generation capabilities, and can perform tasks such as natural language processing, text generation, semantic understanding, and reasoning.

[0041] A model deployment node refers to a unit with computing and storage capabilities. Each model deployment node can independently deploy a large language model. A target inference framework is typically installed on the model deployment node, and it maintains a communication connection with the distributed storage module. It serves as both a computation execution unit and one of the storage carriers for model files, enabling efficient loading of model files according to task requirements.

[0042] The target deployment node is the actual execution node selected from multiple model deployment nodes based on the model deployment task. For example, after receiving a model deployment task, the large language model deployment system comprehensively considers factors such as the load, network bandwidth, cache hit rate, and CPU (Central Processing Unit) availability of each model deployment node to determine the most suitable node to execute the current model deployment task. The selected target deployment node is responsible for loading the model file of the target large model and starting the target inference framework instance to complete the model's inference service.

[0043] A distributed storage module refers to a unified storage system for managing model files. It can store both the structural information and the actual data content of the model files. Multiple model deployment nodes share the same model file through the distributed storage module, thereby achieving data access and consistency management across nodes and cloud environments, and avoiding duplicate storage of model files.

[0044] In this application, the distributed storage module can be the JuiceFS file system. JuiceFS is a distributed file system characterized by high performance, high availability, high scalability, and good cross-platform compatibility, making it well-suited for cloud environments. This application deeply integrates the JuiceFS file system with several mainstream large-model inference frameworks, overcoming the challenges of storage and management in large-model deployment. Compared to related technologies, it significantly improves model file reading efficiency, reduces storage costs and operational complexity, and achieves efficient utilization of storage resources and optimization of large-model inference performance through dynamic resource scheduling and intelligent caching mechanisms. This fills a technological gap in this field and provides reliable technical support for the large-scale application and deployment of large models.

[0045] A target inference framework refers to a framework deployed on a target deployment node for loading and executing inference services for large language models. The target inference framework supports loading and running various model formats. For example, mainstream large model inference frameworks such as Ollam, vLLM, and SGLang are responsible for loading model files, receiving input requests, performing computational inference, and returning results. Different inference frameworks differ in performance, compatibility, and deployment methods. This application aims to achieve efficient integration of mainstream inference frameworks with the JuiceFS storage system to improve the overall effect of large model deployment. In practical applications, the large language model deployment system binds the Ollam, vLLM, and SGLang inference frameworks to the target deployment node. After determining the target deployment node, it calls the corresponding inference instance of the inference framework, quickly loads the model file, and starts the inference service, achieving automated deployment of large language models.

[0046] Step 102: If the model file required by the target large model is detected in the local cache, the model file is loaded into the target inference framework.

[0047] In this embodiment, when the required model file is detected in the local cache, the large language model deployment system can directly load the locally cached model file into the target inference framework, thereby quickly deploying the large language model.

[0048] Optionally, local caching refers to the storage space on the target deployment node used to temporarily store model files of large language models. It is usually deployed on the node's local SSD. SSD has a fast read and write speed and low latency, which helps to accelerate the access and loading of model files and improve inference efficiency.

[0049] Model files refer to the various data files required when loading and running a large language model, including but not limited to the model parameters and configuration information of the large language model. For example, model parameters include weight parameters, and configuration information includes model structure, input dimensions, output dimensions, etc.

[0050] As an example, before loading, the model files in the local cache can be validated, such as for version consistency, integrity, and permissions. During the loading process, the target inference framework can choose different loading methods based on the size and format of the model file. For example, for small to medium-sized model files, a one-time loading method can be used to store the entire model in the local cache. For very large model files, a layered loading strategy can be used, loading the index information first, and then loading the corresponding data blocks based on the index information.

[0051] Step 103: If the local cache does not contain the model file required by the target large model, the model file is retrieved from the distributed storage module and loaded into the target inference framework.

[0052] In this embodiment, when the local cache does not contain the model file required by the target large model, the large language model deployment system automatically obtains the model file from the distributed storage module and loads the model file into the target inference framework.

[0053] As an example, when the model file is not detected in the local cache, the large language model deployment system first sends a data query request to the distributed storage module to obtain the metadata of the model file. Based on the metadata information, it determines the location of the model file and downloads the model file from the corresponding location.

[0054] Step 104: On the target deployment node, start the inference service of the target inference framework and import the target large model to complete the model deployment task.

[0055] In this embodiment of the application, in steps 102 and 103 above, the model file of the target large model has been successfully loaded onto the local cache of the target deployment node. The large language model deployment system can start the inference service of the target inference framework on the target deployment node and import the target large model to complete the entire model deployment task.

[0056] Optionally, the process of importing the target large model mainly involves registering the model file to the target inference framework, which then performs model initialization, loading, and other operations, enabling the target large model to perform calculation, prediction, and other tasks normally.

[0057] This application applies to a large language model deployment system, which includes multiple model deployment nodes and a distributed storage module. The system receives model deployment tasks for at least one target large model and selects a target deployment node from among the multiple deployment nodes to execute the task. The target deployment node integrates a target inference framework. If the local cache contains the model file required by the target large model, the model file is loaded into the target inference framework. If the local cache does not contain the required model file, the model file is retrieved from the distributed storage module and loaded into the target inference framework. On the target deployment node, the inference service of the target inference framework is started, and the target large model is imported to complete the model deployment task.

[0058] Compared to related technologies, the technical solution of this application offers two key advantages: First, it utilizes a distributed storage module to uniformly manage model files, enabling cross-node and cross-cloud environment data access. The target deployment node can automatically retrieve model files, reducing manual intervention and effectively lowering operational complexity. Second, when a model file exists in the local cache, it is directly loaded into the target inference framework, avoiding repeated data readings and improving overall system throughput. When the model file is not present in the local cache, it is retrieved from the distributed storage module, ensuring reliable data retrieval even during initial deployment or cache misses, thus enhancing the automation and reliability of model deployment. Therefore, the technical solution of this application significantly improves the deployment efficiency and effectiveness of large language models.

[0059] To facilitate understanding of the deployment method of the large language model in the embodiments of this application, the deployment system of the large language model in the embodiments of this application will be described in detail below.

[0060] See Figure 2 and Figure 3 , Figure 2 This is a schematic diagram of a large language model deployment system shown in an embodiment of this application. Figure 3 This is a schematic diagram of server communication shown in an embodiment of this application.

[0061] In this application, the large language model deployment system is a system that seamlessly integrates an inference framework and a distributed storage module. The inference frameworks used in the large language model deployment system are Ollam, vLLM, and SGLang, and the distributed storage module is the JuiceFS file system, as illustrated in the example.

[0062] like Figure 2 As shown, from a logical architecture perspective, the large language model deployment system can be generally divided into an inference framework layer, a JuiceFS file system and storage layer, and a security and management layer. The inference framework layer integrates multiple mainstream inference frameworks, each accessing model files in the JuiceFS file system through the JuiceFS mount point. The JuiceFS file system and storage layer includes local SSD caching and a distributed cache pool, significantly accelerating model file read speeds. The security and management layer provides comprehensive protection, encompassing mechanisms such as transmission encryption, storage encryption, access control, and multi-cloud synchronization.

[0063] like Figure 3As shown, in terms of server type, the large language model deployment system mainly consists of GPU (Graphics Processing Unit) computing servers, large model file storage servers, and database servers. The GPU computing servers are the core of the system's computing, specifically responsible for the operation and inference tasks of the large language model, and deploy various large model inference frameworks such as Ollam, vLLM, and SGLang. The large model file storage servers are used to store massive amounts of model files, physically separated from the computing layer. This layer employs storage solutions such as MinIO (Distributed Object Storage System), S3 (Simple Storage Service), Ceph (Distributed File System), and OOS (Object Storage Service). These storage solutions are suitable for storing large model files and have advantages such as low cost and high scalability. The database server is responsible for managing the metadata information of the model files. It mainly uses databases such as Redis (Remote Dictionary Server), MySQL (MySQL Structured Query Language), PostgreSQL (PostgreSQL Structured Query Language), and SQLite (Structured Query Language Lite), and uses it as the metadata engine of the JuiceFS file system.

[0064] First, configure the distributed storage module.

[0065] In this embodiment of the application, before receiving a model deployment task for at least one target large model, a distributed storage module is created. The distributed storage module includes a metadata engine and an object storage unit. The metadata engine is used as the metadata storage space of the distributed storage module, and the object storage unit is used as the data block storage space of the distributed storage module. The association between the metadata storage space and the data block storage space is established.

[0066] The JuiceFS file system consists of a metadata engine and an object storage backend. It not only has no special limitations on hardware configuration, but is also compatible with a variety of mainstream operating systems, including Linux, BSD, macOS, and Windows. The appropriate operating system platform can be selected according to the actual deployment environment and system architecture, thereby ensuring the portability and deployment flexibility of the file system.

[0067] To ensure the deployment performance of the large language model, this application deploys the metadata engine on a high-performance server, such as one equipped with a multi-core CPU and sufficient memory, and deploys the object storage unit on a cloud object storage service. Simultaneously, SSDs are configured as local caches on the JuiceFS client nodes, with the cache capacity configured based on the size of the large language model's model files and access frequency, generally not less than 100GB.

[0068] The configuration process for the metadata engine can involve selecting a suitable metadata engine, such as Redis (a remote dictionary server), MySQL (a Structured Query Language database), or PostgreSQL (a PostgreSQL object-relational database). Taking Redis as an example, install the Redis service on a cloud server and configure it accordingly, such as setting maximum memory and persistence strategies, to ensure efficient storage and fast access to metadata.

[0069] The configuration process for an object storage unit can be as follows: Select a suitable object storage service, such as Amazon S3 (Amazon Simple Storage Service), AliyunOSS (Alibaba Cloud Object Storage Service), or Tencent Cloud Object Storage Service. Create the corresponding bucket and configure access keys, access permissions, and other information to ensure that the JuiceFS file system can access the object storage unit normally.

[0070] The process of installing the JuiceFS client and creating the JuiceFS file system can be as follows: Install the JuiceFS client on the model deployment node where the inference framework needs to be deployed. Use the `juicefs format` command to create the file system. The command format is as follows: `juicefs format [--storage][--bucket][--access-key][--secret-key]`. This command specifies the object storage type, such as S3 or OSS, the bucket path of the object storage, the key to access the object storage, and finally creates the connection address for the metadata engine and the JuiceFS file system name.

[0071] After the JuiceFS file system is created, you can use the `juicefs mount` command to mount it to the local cache directory. The command format is: `juicefsmount [--cache-dir][--cache-size]`. This command specifies the local cache directory, which is recommended to be set on an SSD, and the cache size should also be specified. Additionally, you can create a JuiceFS file system and mount it to a local directory path on the node. This local directory path will serve as the access path for the inference framework to read model files.

[0072] In practical applications, various model files, such as model parameters and configuration files from DeepSeek and Qwen, can be uniformly stored in the JuiceFS file system. The JuiceFS file system stores the metadata of model files in a metadata engine (such as Redis), while the actual data blocks are stored in object storage units (such as AliyunOSS), achieving separate storage of metadata and actual data blocks and improving management efficiency.

[0073] The metadata engine serves as a metadata storage space, storing metadata information for model files, such as their path, size, shard list, version number, and permissions. The object storage unit serves as a data block storage space, storing the actual data blocks of the model files, such as model parameters. This application establishes and maintains a mapping relationship between metadata and data blocks, ensuring that during access, the corresponding data block can be located through the metadata, enabling read and write operations.

[0074] By following the steps above, the target deployment node can complete the initialization and mounting of the JuiceFS file system, enabling unified management and fast access to model files.

[0075] Secondly, configure the inference framework and integrate it with the distributed storage module.

[0076] To ensure the proper functioning of different inference frameworks, it is necessary to ensure that their operating environment meets the minimum hardware requirements, such as sufficient CPU, memory, and GPU resources. Simultaneously, it is essential to guarantee that the inference framework can access the mounted JuiceFS file system directory, meaning it must have the appropriate read and write permissions for the JuiceFS file system directory.

[0077] In this embodiment of the application, before receiving a model deployment task for at least one target large model, a target inference framework is installed on each model deployment node, the model storage path of the target inference framework is configured, and the model storage path is mapped to the local cache directory of the distributed storage module, so that the target inference framework can perform read and write operations on the local cache of the distributed storage module. Furthermore, during the read and write process, the target inference framework can access the model files in the distributed storage through the model storage path without modifying the data.

[0078] The integration configuration of the Ollam framework can include the following steps: (1) Install the Ollam client: Visit the Ollam official website, download the corresponding installation package according to the operating system type, and complete the installation according to the official instructions.

[0079] (2) Configure the Ollam model storage path: Modify the Ollam configuration file to point the model storage path to the JuiceFS file system mounted to the local cache directory. For example, in a Linux system, you can edit the Ollam configuration file, such as / etc / ollama / config.yaml, and set model-dir: / ollama / models.

[0080] (3) Start the Ollam service: Use the ollama serve command to start the service. At this time, the Ollam framework will automatically read and store model files from the configured model storage path.

[0081] The integration configuration of the vLLM framework can include the following steps: (1) Install vLLM: Install the vLLM related dependency packages using pip (Pip Installs Packages or Pip Installs Pythonpackages, Python package installer) command, such as pip install vllm.

[0082] (2) Configure the vLLM model storage path: When deploying a large language model using vLLM, specify the model storage path and mount it to the local cache directory using the JuiceFS file system. For example, start the vLLM service using the command python-mvllm.entrypoints.api_server--model / vllm / models / deepseek-6b, where / vllm / models / deepseek-6b is the storage path for the deepseek-6b model stored on JuiceFS.

[0083] In this embodiment, the seamless integration of various mainstream large-model inference frameworks with the JuiceFS file system is achieved through the above-described method, thereby significantly improving the loading and access speed of large model files, ensuring efficient and stable deployment of large language models, and effectively reducing the overall storage cost of the system. Simultaneously, this embodiment supports efficient management and consistent maintenance of large language model files in multi-cloud and hybrid cloud environments, providing a unified and reliable storage solution for different inference frameworks, simplifying the deployment process of large language models, and improving the scalability and operational efficiency of the system.

[0084] Figure 4 This is another flowchart illustrating a method for deploying a large language model, as shown in an embodiment of this application. Figure 4 relatively Figure 1 The technical solutions of the embodiments of this application are described in more detail.

[0085] Step 401: Receive a model deployment task for at least one target large model.

[0086] In this embodiment of the application, users can initiate model deployment tasks through an interactive interface.

[0087] For example, users can directly select the target large model, specify the deployment node, set the inference framework and other task parameters in the interactive interface, or users can input natural language text in the interactive interface, and the large language model deployment system will parse the corresponding information from it and trigger the model deployment task.

[0088] Step 402: Analyze the model deployment task and extract the model type and deployment resources of the target large model.

[0089] When the large language model deployment system receives a model deployment task, it parses the task content and extracts the model type and deployment resources contained therein. The extracted model type and deployment resources are used in subsequent steps such as target deployment node selection, model file loading, local cache configuration, and inference framework startup to ensure that the entire model deployment meets the task requirements.

[0090] For example, the model type is a category of target large model, such as large language model①, large language model②, large language model③, etc.

[0091] Deployment resources include central processing unit (CPU), graphics processing unit (GPU), memory capacity, storage resources, network bandwidth, operating system type, and other hardware or software environment requirements for deployment nodes.

[0092] Step 403: Based on the model type and deployment resources, select the target deployment node from multiple model deployment nodes to execute the model deployment task.

[0093] In this embodiment of the application, the large language model deployment system can select the most suitable target deployment node from the available model deployment nodes to perform the model deployment task based on the model type and deployment resources.

[0094] As an example, the large language model deployment system first obtains detailed street information for each model deployment node, including its hardware resource status, operating system type, current load, network bandwidth, and cache status. Then, it matches the extracted model types with each model deployment node, filtering out at least one candidate deployment node. If multiple candidate deployment nodes exist, they are sorted according to deployment resources, thus selecting the node with the best performance as the target deployment node.

[0095] Step 404: If the local cache contains the model file required by the target large model, then load the model file into the target inference framework.

[0096] In this embodiment, the model file can be preheated to improve the data loading speed, so that when the model file required by the target large model is detected in the local cache, the model file can be quickly loaded into the target inference framework.

[0097] As an example, when deploying a large DeepSeek model within the Ollam framework, data preheating is performed before deployment by executing the command `juicefswarmup / mnt / juicefs / deepseek / ollama / models / deepseek-6b`, pre-loading the model file into the JuiceFS local cache. When the Ollam service starts and `ollama run deepseek-6b` is executed to load the model, Ollam reads directly from the cache. Testing showed that the initial load, which originally took 30 seconds, was reduced to 15 seconds after preheating.

[0098] When deploying the Qwen-7b model within the vLLM framework, the command `juicefs warmup / mnt / juicefs / qwen / vllm / models / qwen-7b` is run to warm up the data before starting the vLLM service. During vLLM startup, the JuiceFS local cache is accessed through the specified model storage path. Because the cache already contains model data, the loading time is reduced from 25 seconds to 12 seconds. Furthermore, frequently accessed model layer data can be quickly retrieved from the cache during subsequent inference, improving overall inference efficiency.

[0099] Step 405: If the local cache does not contain the model file required by the target large model, the model file is retrieved from the distributed storage module and loaded into the target inference framework.

[0100] In this embodiment of the application, if it is detected that the model file required by the target large model does not exist in the local cache, a data query request is generated. In response to the data query request, the model file is obtained from the metadata storage space and the data block storage space, and the model file is stored in the local cache so that the target inference framework can obtain the model file from the local cache.

[0101] Furthermore, in response to a data query request, the system obtains the login information of the logged-in user, verifies the legitimacy of the logged-in user based on the login information, and extracts the location information of the model file from the metadata storage space when the legitimacy verification is successful. Based on the location information, the system reads the model file from the data block storage space.

[0102] The data query request refers to a request message generated by the large language model deployment system when it detects that the model file required by the target large model does not exist in the local cache. This message is used to read the model file from the distributed storage module. The data query request must include at least the unique identifier of the target large model, the name of the required file, the request initiation time, and the request node identifier.

[0103] User login information refers to the identity information entered by a user when logging in through a terminal device that is connected to the large language deployment model, such as account name, role information, and terminal device identifier.

[0104] Legality verification refers to the process by which a large language model deployment system verifies a user's identity and access permissions based on their login information. The legality verification process includes two stages: user authentication and access authorization. For example, first, it verifies whether the user's account name is a registered and legitimate account, and checks the account's login status and credential validity period. Then, it verifies whether the user's role information meets the permissions required to access the target large model. If all verification conditions are met, the user's login information is deemed to have passed legality verification. If any one of the conditions is not met, the user's login information is deemed to have failed legality verification, data access is denied, and a corresponding error message is returned, thereby improving the security of model deployment.

[0105] Location information refers to the index data of model files stored in the metadata storage space of the distributed storage module, used to record the storage location of model files in object storage units. For example, location information includes the storage path and data block number of the model file.

[0106] As an example, refer to Figure 5 , Figure 5This is a schematic diagram illustrating the model file transfer process in an embodiment of this application. When the Ollam framework needs to load a target large model, it accesses the local cache directory of the JuiceFS file system through the POSIX interface (Portable Operating System Interface) according to the configured model storage path. After receiving the access request, the JuiceFS client of the JuiceFS file system first checks whether the required model file exists in the local cache. If it exists, it reads it directly from the cache and returns it to Ollam. If it does not exist, the JuiceFS client queries the metadata engine for the location information of the model file, then reads the model file from the object storage unit, caches the model file in the local cache, and then returns it to Ollam. When the vLLM framework needs to load a target large model, after starting the service and specifying the model storage path, the vLLM framework accesses the model file in the local cache directory of the JuiceFS file system through the JuiceFS file system interface. The JuiceFS file system also accelerates data access through a caching mechanism. It first checks the local cache; if a match is found, it returns the model file directly; otherwise, it reads the model file from the object storage unit and caches it.

[0107] Step 406: On the target deployment node, start the inference service of the target inference framework and import the target large model to complete the model deployment task.

[0108] After the model file is successfully loaded and initialized, the target inference framework enters the inference service ready state. At this point, the large language model deployment system can send a small number of test requests to initialize the inference service, allowing the target large model to complete the necessary compilation, optimization, and caching preparation, reducing the latency of the first inference. Subsequently, inference operations are performed based on the model file, the target large model is imported, and the target large model is added to the list of available models, completing the model deployment task.

[0109] In a multi-node deployment of the Ollam framework, three model deployment nodes are mounted on the same JuiceFS file system. When model deployment node ① loads the target large model via `ollam run deepseek-6b`, its cache stores some key data of the target large model. When model deployment nodes ② and ③ load the target large model for the first time, they directly retrieve the data from the cache of model deployment node ① through a distributed caching mechanism, without needing to reread it from the object storage unit. Testing showed that the initial load time for model deployment nodes ② and ③ was reduced from 30 seconds for individual loads to 10 seconds, and the load time remained stable as the number of nodes increased, avoiding the performance degradation issues associated with multi-node loading in traditional storage. In a multi-node deployment of the vLLM framework, multiple vLLM instances share the local cache of the JuiceFS file system. Once a vLLM instance has loaded a specific parameter block of the Qwen-7b model, other vLLM instances can access that parameter block directly from the local cache of the JuiceFS file system, increasing the overall cluster cache hit rate to over 90%, significantly reducing the access pressure on object storage, and ensuring stable performance during high-concurrency inference.

[0110] Furthermore, if the target large model undergoes a version update, the large language model deployment system can receive the update request and obtain the updated model file, which includes change metadata and change data blocks. The change metadata is encrypted and written to the metadata storage space of the target deployment node, while the change data blocks are encrypted and written to the data block storage space, ensuring data security and integrity. Subsequently, based on a pre-built mirror relationship, the change metadata and change data blocks are synchronized to other model deployment nodes, achieving cross-node consistent updates.

[0111] The change metadata records the new version information, modification time, permissions, etc. of the model file, while the change data block contains the actual model parameters and configuration files.

[0112] In this application, the JuiceFS file system and target inference framework are designed with an encryption mechanism to protect the model files during transmission through an encrypted channel, preventing them from being stolen or tampered with during network transmission.

[0113] As an example, when deploying DeepSeek models using the Ollam framework, the JuiceFS file system enables transport and storage encryption. The Ollam framework and JuiceFS clients transmit model data encrypted via SSL (Secure Sockets Layer) / TLS (Transport Layer Security), ensuring that data is not intercepted during the `ollampull` and `ollamrun` processes. Model files are encrypted when stored in object storage units; if an object storage unit is not authorized for access, the model file cannot be parsed. Only Ollam nodes configured with the correct decryption key can read the data, guaranteeing the security of the model data.

[0114] For example, permissions can be set for the model storage path ` / mnt / juicefs / deepseek / ollama / models`, allowing only the Ollam service user to read and write, while other users have no access. Developers can upload and update models through authorized JuiceFS clients, while the Ollam service on inference nodes can only read, preventing accidental deletion or tampering of model files. Access keys for object storage units are strictly managed; only the model deployment node deploying Ollam can obtain the correct decryption key, ensuring that model files are not accessed illegally.

[0115] As another example, when deploying the Qwen-7b model using the vLLM framework, the JuiceFS file system's encryption mechanism requires the vLLM framework to interact with the metadata engine and object storage units through encrypted channels. The vLLM framework receives inference request data through its API (Application Programming Interface), and its interaction with the JuiceFS file system is also encrypted to prevent sensitive data from being leaked during transmission.

[0116] For example, an IP (Internet Protocol) whitelist restricts access to the metadata engine to only the model deployment node that has deployed vLLM, and chmod 700 permissions are set for the model storage path, allowing only the vLLM framework running user to access it. Simultaneously, the object storage unit only allows access via a specific access key from the JuiceFS client, ensuring the secure and reliable source of model files loaded by the vLLM framework and preventing unauthorized nodes from accessing, loading, or tampering with model files.

[0117] After encrypting the changed metadata and writing it to the metadata storage space of the target deployment node, and encrypting the changed data block and writing it to the data block storage space, the changed metadata and changed data block can also be synchronized to other model deployment nodes based on the pre-built mirror relationship to achieve cross-node consistent updates.

[0118] In the application, the version update of the target large model often involves data synchronization between multiple model deployment nodes, that is, data synchronization between the target deployment node and other model deployment nodes, and the target deployment node and other model deployment nodes may belong to the same cloud environment or different cloud environments.

[0119] In order to achieve efficient updates of the target large model and maintain data consistency in a multi-cloud or hybrid cloud environment, the large language model deployment system of this application needs to have a mirroring function. Through the mirroring function, the updated model files are automatically synchronized to other model deployment nodes.

[0120] As an example, after a DeepSeek model deployed on Alibaba Cloud using the Ollam framework undergoes a version update, the updated (new version) model file is stored in Alibaba Cloud's JuiceFS file system. Since a mirror relationship between Alibaba Cloud and the AWS (Amazon Web Services) JuiceFS file system is pre-built, executing the command `juicefs mirror deepseek-fsaws-deepseek-fs--bucketoss: / / aws-deepseek-bucket` automatically synchronizes the updated model file to the AWS JuiceFS file system. The Ollam service on the AWS node requires no manual intervention; after restarting, the updated DeepSeek model can be loaded via the original model storage path. This achieves synchronized updates of Ollam-deployed models across cloud environments. The entire synchronization process takes only about 3 minutes, and the integrity of the model files is rigorously verified after synchronization, ensuring data consistency and availability.

[0121] As another example, in a cross-cloud deployment scenario of the vLLM framework, after the Qwen-7b model used by Tencent Cloud's vLLM service completes a version update, it is synchronized to Alibaba Cloud via the mirroring function of the JuiceFS file system. When Alibaba Cloud's vLLM service makes inference requests, it automatically reads the synchronized model file without needing to re-upload or reconfigure the path, thus ensuring the consistency of model versions deployed in different cloud environments.

[0122] As can be seen, the cross-cloud deployment of the Ollam framework and the vLLM framework relies on the mirroring function of the JuiceFS file system. In fact, the mirroring function in this application is a cross-storage system mirroring mechanism based on metadata synchronization and data block replication. This mirroring mechanism ensures that the data in the target file system remains consistent with the source file system by establishing a mapping relationship between the source file system (such as Tencent Cloud JuiceFS) and the target file system (Alibaba Cloud JuiceFS).

[0123] The mirroring function can be triggered in two modes: manual and automatic.

[0124] The manual trigger mode executes the corresponding command, such as `juicefs mirror tencent-qwen-fsaliyun-qwen-fs--bucketoss: / / aliyun-qwen-bucket`.

[0125] The automatic trigger mode is configured as a scheduled task or listens for changes in the source file system. In vLLM cross-cloud deployment scenarios, it is usually configured to "automatically trigger after source file update" to ensure rapid synchronization after model file updates.

[0126] At the metadata level, update operations on model files in the source file system (such as adding model files, modifying configuration files, or performing version iterations) are recorded in the metadata engine in real time. The mirroring function can periodically scan the metadata of the source file system, identify changed metadata items (such as file path, file size, modification time, access permissions, etc.), and synchronize the changed metadata to the metadata engine of the target file system to maintain the consistency of the file structure at both ends.

[0127] At the data block level, for the actual data block pointed to by the metadata (such as the parameter file block of the Qwen-7b model), the mirroring function can check whether the corresponding data block exists in the object storage unit of the target file system. If the target file system does not have the data block or the data content is inconsistent with the source file system, the corresponding data block is read from the object storage unit of the source file system and written to the object storage unit of the target file system through encrypted transmission to ensure the integrity and consistency of the model data content.

[0128] As an example, the execution steps for mirroring the Qwen-7b model on various cloud servers, such as Tencent Cloud and Alibaba Cloud, are as follows: Scan the model storage path of the Qwen-7b model in the source file system, such as scanning and obtaining the metadata of / mnt / juicefs / qwen / vllm / models / qwen-7b. Compare this metadata with the metadata of the corresponding directory in the target file system to identify newly added, modified, or deleted files. Write the updated metadata in the source directory (such as information on newly added model shard files) to the metadata engine of the target file system. For newly added or modified files, read data blocks from Tencent Cloud COS, send them to Alibaba Cloud OSS via encrypted transmission (such as SSL / TLS), and record the location of the data blocks in the target object storage. After synchronization, verify the consistency between the target file and the source file by hash value comparison, such as calculating the MD5 hash of the file (Message-Digest Algorithm 5), to ensure that the data is not corrupted.

[0129] The aforementioned mirroring mechanism enables data synchronization in multi-cloud or hybrid cloud environments. However, to prevent inconsistencies in model files caused by network fluctuations, latency, or concurrent access, this application further designs a data consistency management mechanism to ensure data consistency and availability during cross-cloud deployment.

[0130] Under this mechanism, after deleting an old version of a DeepSeek model file in Alibaba Cloud's Ollam framework, the metadata engine records the deletion operation in real time and synchronizes the change information to all nodes mounted with the JuiceFS file system. At this point, when the Ollam service on the AWS node attempts to access the old version of the model file, it will receive a "file not found" response, thus avoiding loading an invalid model. Simultaneously, the JuiceFS file system can periodically verify the source and target file systems, with verification data covering both metadata and actual data content. If differences arise between the two ends due to network fluctuations or synchronization interruptions, the large language model deployment system automatically performs repair operations, ensuring that the Ollam framework always accesses consistent and valid model files in multi-cloud deployment scenarios. For the vLLM framework, when the configuration file of the Qwen-7b model is modified, the metadata engine immediately synchronizes the change information to other nodes. All vLLM instances can automatically obtain the latest configuration file the next time they load the configuration file, thus ensuring the consistency of inference parameters and avoiding inference result deviations caused by configuration differences. By accessing the corresponding interface, the synchronization status and consistency verification results of the model files can be monitored in real time, further ensuring the data reliability and consistency of the vLLM framework during cross-cloud deployment.

[0131] Finally, based on the above content, complete examples of the Ollam framework and the vLLM framework are provided to fully illustrate the large language model deployment process of this application.

[0132] 1) Deploying DeepSeek large models based on the JuiceFS file system and Ollam framework S11, Preparation Phase.

[0133] Configure the cloud environment: Install the CentOS 7 operating system (Community Enterprise Operating System 7) on the Alibaba Cloud ECS server (Elastic Compute Service), ensuring the server has sufficient CPU, memory, and network bandwidth. Simultaneously, activate Alibaba Cloud OSS object storage service and create a storage bucket named "deepseek-models".

[0134] Install Redis: Install Redis 6.2 on another ECS server, configure the maximum memory to 10GB, and enable AOF (Append-Only File) persistence mode as the metadata engine for the JuiceFS file system.

[0135] Install the JuiceFS client: On the ECS server where the Ollam framework is deployed, install the JuiceFS client using the official script. The command is curl -sSL https: / / d.juicefs.com / install|sh.

[0136] To install the Ollam framework: Download the Linux installation package for the Ollam framework and execute `sudodpkg-iollama_xxx.deb` to install it.

[0137] S12, deploy the JuiceFS file system.

[0138] To create a file system, execute the command `juicefs format --storageoss --bucketoss: / / deepseek-models --access-key --secret-keyredis: / / :6379 / 0deepseek-fs`, which includes the Alibaba Cloud OSS access key and the IP address of the Redis server.

[0139] Mounting the file system: Create a local mount directory / mnt / juicefs / deepseek, and execute the command juicefsmount --cache-dir / var / juicefs / cache --cache-size 500GB redis: / / :6379 / 0 / mnt / juicefs / deepseek to mount the JuiceFS file system to this directory.

[0140] S13, deploy the Ollam framework and DeepSeek large models.

[0141] Configure the Ollam model storage path: Edit the Ollam framework configuration file / etc / ollama / config.yaml, add model-dir: / mnt / juicefs / deepseek / ollama / models, save and exit.

[0142] Start the Ollama service: Execute the command `sudo systemctl start tollama` to start the service and set it to start automatically on boot.

[0143] Pulling and importing DeepSeek models: Execute the command ollamapulldeepseek-6b. Ollama will automatically pull the model files from the network and store them in the directory / mnt / juicefs / deepseek / ollama / models.

[0144] S14, Model Loading and Running.

[0145] Loading the model: Executing the command `ollamarundeepseek-6b` will cause the Ollam framework to load the DeepSeek model from the JuiceFS file system. Due to JuiceFS's caching mechanism, subsequent loading speeds are significantly faster after the initial load.

[0146] Interaction and Monitoring: Interact with the DeepSeek model via command line, input questions and get answers. Simultaneously, use the `juicefsstats` command to monitor JuiceFS performance metrics, such as cache hit rate and I / O throughput, to ensure system proper operation.

[0147] S15, Data Management and Optimization.

[0148] Data warm-up: Execute the command juicefswarmup / mnt / juicefs / deepseek / ollama / models / deepseek-6b to warm up the model files and improve subsequent loading speed.

[0149] Distributed caching configuration: On the other two nodes deploying the Ollam framework, mount the JuiceFS file system and configure a distributed cache. By editing the JuiceFS configuration file and setting `cache-group:deepseek-group`, multiple nodes can share cache information.

[0150] Multi-cloud synchronization: Deploy the same JuiceFS file system in the AWS cloud environment, configure the mirror relationship, and mirror the JuiceFS file system on Alibaba Cloud to AWS to achieve cross-cloud synchronization of model data.

[0151] S16, Security and Permissions Configuration.

[0152] Configure security policy: Enable JuiceFS's transmission and storage encryption features by adding the --encrypt-rsa-key parameter when creating the file system to encrypt data using an RSA key.

[0153] To set access control: Execute the command `chown-Rollama:ollama / mnt / juicefs / deepseek / ollama` to set the owner of the model directory to the Ollam service user, and then execute `chmod 700 / mnt / juicefs / deepseek / ollama` to restrict access to the model files to only this user.

[0154] 2) Deploying the Qwen large model based on JuiceFS and vLLM S21, Preparation Phase.

[0155] Configure the cloud environment: Install the Ubuntu 20.04 operating system on the Tencent Cloud CVM server and configure a GPU (such as NVIDIA A100) to meet the hardware requirements of vLLM. Activate Tencent Cloud COS object storage and create a "qwen-models" bucket.

[0156] Install MySQL: Install MySQL 8.0 on a CVM server as the metadata engine for JuiceFS, create the database and grant access.

[0157] Install the JuiceFS client and vLLM: Install the JuiceFS client on the server where vLLM is deployed, and install vLLM using the pipinstallvllm command.

[0158] S22, Deploy the JuiceFS file system.

[0159] To create a file system: Execute the command `juicefsformat --storagecos --bucketcos: / / qwen-models --access-key --secret-keymysql: / / :@:3306 / juicefsqwen-fs`, where `qwen-models` and `secret-key` are the Tencent Cloud COS access keys, and `secret-key` is the MySQL login information.

[0160] Mount the file system: Create the directory / mnt / juicefs / qwen, and execute the command juicefsmount --cache-dir / var / juicefs / qwen-cache --cache-size 1TBmysql: / / :@:3306 / juicefs / mnt / juicefs / qwen to mount it.

[0161] S23, Deploy vLLM and Qwen large model.

[0162] Upload Qwen model files: Upload the relevant files of the Qwen-7b model (such as config.json, pytorch_model-00001-of-00002.bin, etc.) to the / mnt / juicefs / qwen / vllm / models / qwen-7b directory.

[0163] To start the vLLM service: Execute the command python-mvllm.entrypoints.api_server--model / mnt / juicefs / qwen / vllm / models / qwen-7b--port8000 to start the vLLM API service, which will load the Qwen model from the JuiceFS path.

[0164] S24, Model Loading and Running.

[0165] Accessing the vLLM service: Access the vLLM API interface (e.g., http: / / :8000 / generate) via an HTTP request to send an inference request to the Qwen model. vLLM will efficiently load model data from JuiceFS for inference.

[0166] Performance monitoring: Use the monitoring interface provided by vLLM and the stats command of JuiceFS to monitor model inference performance and storage system performance, and make optimizations and adjustments based on the monitoring results.

[0167] S25, Data Management and Optimization.

[0168] Data warm-up: Execute the command juicefswarmup / mnt / juicefs / qwen / vllm / models / qwen-7b to warm up the model file.

[0169] Distributed caching: Configure distributed caching on multiple vLLM deployment nodes to improve cache utilization.

[0170] Consistency Management: Periodically execute the juicefsfsck command to check the consistency of the JuiceFS file system and ensure the integrity of model data.

[0171] S26, Security and Permissions Configuration.

[0172] Configure security policy: Enable JuiceFS's transport encryption and storage encryption to protect the security of model data during transport and storage.

[0173] Configure access control: Set permissions for the / mnt / juicefs / qwen / vllm / models directory to allow only the vLLM service user to read it. Execute chmod 400 / mnt / juicefs / qwen / vllm / models / * to ensure that the model data is not modified illegally.

[0174] The technical solution of this application not only combines the JuiceFS file system with mainstream large model inference frameworks such as Ollama, vLLM, and SGLang, and solves the storage problem of different inference frameworks when deploying large models through a unified storage management method, thus achieving efficient deployment of large models, but also realizes automatic synchronization of model files in multi-cloud and hybrid cloud environments through the mirroring function of the JuiceFS file system, and ensures data consistency by relying on the data consistency management mechanism of the JuiceFS file system.

[0175] Furthermore, JuiceFS file system's data encryption, access control, and permission control provide comprehensive security protection for the deployment of large language models. At the same time, JuiceFS file system supports horizontal scaling, allowing for flexible addition of storage nodes and cache capacity according to business needs, easily addressing the challenge of continuously growing model file sizes and meeting the long-term requirements of large-scale model deployment.

[0176] Corresponding to the aforementioned application function implementation method embodiments, this application also provides a large language model deployment system, electronic device, and corresponding embodiments.

[0177] Figure 6 This is a schematic diagram of the structure of a large language model deployment system shown in an embodiment of this application.

[0178] See Figure 6A large language model deployment system, comprising multiple model deployment nodes and a distributed storage module, including: The task receiving module 601 is used to receive model deployment tasks for at least one target large model, and select the target deployment node to execute the model deployment task from multiple model deployment nodes; the target deployment node integrates a target inference framework. The first model file loading module 602 is used to load the model file into the target inference framework if the local cache detects that the model file required by the target large model exists. The second model file loading module 603 is used to obtain the model file from the distributed storage module and load the model file into the target inference framework if the local cache does not contain the model file required by the target large model. The model deployment module 604 is used to start the inference service of the target inference framework on the target deployment node and import the target large model to complete the model deployment task.

[0179] As an optional example of this application, the system also includes a first deployment module; The first deployment module is used to create a distributed storage module, which includes a metadata engine and object storage units. The metadata engine is used as the metadata storage space of the distributed storage module, and the object storage units are used as the data block storage space of the distributed storage module. The association between the metadata storage space and the data block storage space is established.

[0180] As an optional example of this application, the system also includes a second deployment module; The second deployment module is used to install the target inference framework on each model deployment node; configure the model storage path of the target inference framework; and map the model storage path to the local cache directory of the distributed storage module so that the target inference framework can perform read and write operations on the local cache of the distributed storage module.

[0181] As an optional example of this application, the task receiving module 601 is used to receive a model deployment task for at least one target large model; parse the model deployment task to extract the model type and deployment resources of the target large model; and select the target deployment node to execute the model deployment task from multiple model deployment nodes based on the model type and deployment resources.

[0182] As an optional example of this application, the second model file loading module 603 includes: The request generation submodule is used to generate a data query request if the model file required by the target large model is not detected in the local cache. The data query submodule is used to retrieve model files from the metadata storage space and the data block storage space in response to data query requests; The model file loading submodule is used to store model files in the local cache so that the target inference framework can retrieve the model files from the local cache.

[0183] As an optional example of this application, the distributed storage module includes a metadata engine and an object storage unit, and a data query submodule, which is used to obtain the login information of the logged-in user in response to a data query request; to perform a validity verification of the logged-in user based on the login information; and to extract the location information of the model file from the metadata storage space when the validity verification is successful, and to read the model file from the data block storage space based on the location information.

[0184] As an optional example of this application, the system also includes a model update module; The model update module is used to receive version update requests from the target large model and obtain the updated model file. The model file includes change metadata and change data blocks. The change metadata is encrypted and written to the metadata storage space of the target deployment node, and the change data blocks are encrypted and written to the data block storage space of the target deployment node. Based on the pre-built mirror relationship, the change metadata and change data blocks are synchronized to other model deployment nodes.

[0185] Regarding the system in the above embodiments, the specific ways in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0186] Figure 7 This is a schematic diagram of the structure of an electronic device shown in an embodiment of this application.

[0187] See Figure 7 The electronic device 700 includes a memory 710 and a processor 720.

[0188] The processor 720 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. Memory 710 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 720 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 710 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, memory 710 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital versatile optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0189] The memory 710 stores executable code, which, when processed by the processor 720, can cause the processor 720 to execute part or all of the methods described above.

[0190] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0191] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) that, when executed by a processor of an electronic device (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0192] This application also provides a computer program product, which includes computer instructions that, when executed by a processor, implement the method described above.

[0193] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A large language model deployment method, characterized in that, The method, applied to a large language model deployment system, which includes multiple model deployment nodes and a distributed storage module, comprises: The system receives a model deployment task for at least one target large model, and selects a target deployment node from multiple model deployment nodes to execute the model deployment task; the target deployment node integrates a target inference framework. If the model file required by the target large model is detected in the local cache, then the model file is loaded into the target inference framework; If the local cache does not contain the model file required by the target large model, the model file is retrieved from the distributed storage module and loaded into the target inference framework. On the target deployment node, the inference service of the target inference framework is started, and the target large model is imported to complete the model deployment task.

2. The method according to claim 1, characterized in that, Before receiving a model deployment task for at least one target large model, the following are included: Create the distributed storage module, which includes a metadata engine and an object storage unit; The metadata engine is used as the metadata storage space of the distributed storage module, and the object storage unit is used as the data block storage space of the distributed storage module. Establish the association between the metadata storage space and the data block storage space.

3. The method according to claim 1, characterized in that, Before receiving a model deployment task for at least one target large model, the following are included: Install the target inference framework on each model deployment node; Configure the model storage path for the target inference framework; The model storage path is mapped to the local cache directory of the distributed storage module, so that the target inference framework can perform read and write operations on the local cache of the distributed storage module.

4. The method according to any one of claims 1-3, characterized in that, The step of receiving a model deployment task for at least one target large model and selecting a target deployment node from multiple model deployment nodes to execute the model deployment task includes: Receive model deployment tasks for at least one target large model; Analyze the model deployment task to extract the model type and deployment resources of the target large model; Based on the model type and the deployment resources, a target deployment node is selected from multiple model deployment nodes to execute the model deployment task.

5. The method according to claim 2, characterized in that, If it is detected that the local cache does not contain the model file required by the target large model, then the model file is obtained from the distributed storage module, including: If the local cache does not contain the model file required by the target large model, a data query request is generated. In response to the data query request, the model file is obtained from the metadata storage space and the data block storage space; The model file is stored in the local cache so that the target inference framework can retrieve the model file from the local cache.

6. The method according to claim 5, characterized in that, The distributed storage module includes a metadata engine and an object storage unit. The step of retrieving the model file from the metadata storage space and the data block storage space in response to the data query request includes: In response to the data query request, obtain the login information of the logged-in user; Based on the user login information, the legitimacy of the logged-in user is verified; When the legality verification is successful, the location information of the model file is extracted from the metadata storage space, and the model file is read from the data block storage space based on the location information.

7. The method according to claim 2, characterized in that, Also includes: Upon receiving the version update request for the target large model, the updated model file is obtained; the model file includes change metadata and change data blocks; The changed metadata is encrypted and written to the metadata storage space of the target deployment node, and the changed data block is encrypted and written to the data block storage space of the target deployment node; Based on the pre-built mirror relationship, the changed metadata and the changed data blocks are synchronized to other model deployment nodes.

8. A large language model deployment system, characterized in that, The large language model deployment system includes multiple model deployment nodes and a distributed storage module, including: The task receiving module is used to receive model deployment tasks for at least one target large model, and select a target deployment node from multiple model deployment nodes to execute the model deployment task; the target deployment node integrates a target inference framework; The first model file loading module is used to load the model file required by the target large model into the target inference framework if the local cache is detected to contain the model file required by the target large model. The second model file loading module is used to retrieve the model file from the distributed storage module and load the model file into the target inference framework if the local cache does not contain the model file required by the target large model. The model deployment module is used to start the inference service of the target inference framework on the target deployment node and import the target large model to complete the model deployment task.

9. An electronic device, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the method as described in any one of claims 1-7.

Images