Model offline deployment method and device, electronic equipment, medium and product

Abstract

The invention discloses a model offline deployment method and device, electronic equipment, a readable storage medium and a computer program product. The method comprises the following steps: acquiring a model weight file of an own model; performing standardization processing on the model weight file according to a predefined model description file to obtain model configuration information; splitting the model configuration information according to the model weight file to obtain a hierarchical model file; storing the hierarchical model file in a vehicle end storage system; and when the vehicle end performs reasoning, dynamically loading the target model layer file currently required in the storage system to a vehicle end memory, and executing corresponding reasoning operation based on the target model layer file. The method can solve the problem that an incompatible own large model is difficult to efficiently deploy off line in a resource-limited vehicle-mounted environment.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence model deployment technology, specifically to a method, apparatus, electronic device, readable storage medium, and computer program product for offline model deployment. Background Technology

[0002] With the development of smart cockpits and in-vehicle assistants, offline deployment of large language models on the vehicle has become crucial for improving the interactive experience. Currently, the industry commonly uses open-source frameworks as the deployment foundation, such as the Ollam platform, which supports the local execution of various mainstream models. However, in actual in-vehicle scenarios, when enterprises want to deploy their self-developed proprietary models, the inconsistency between the model structure and the platform's native supported formats usually requires manual modification of the framework's underlying code or the writing of complex adaptation scripts. This results in a cumbersome deployment process, high technical barriers, and difficulty in ensuring the model's compatibility and operational efficiency on different in-vehicle hardware, severely restricting the rapid iteration and stable service of in-vehicle intelligent applications. Summary of the Invention

[0003] In view of the above problems, this application provides a model offline deployment method, device, electronic device, readable storage medium and computer program product, which can solve the problem that it is difficult to efficiently deploy incompatible large models offline in resource-constrained vehicle environments.

[0004] Firstly, this application provides a method for offline deployment of a model, including: Obtain the model weight file for your own model; The model weight file is standardized according to a predefined model description file to obtain model configuration information; The model configuration information is split according to the model weight file to obtain a hierarchical model file; The layered model file is stored in the vehicle-side storage system; When performing inference on the vehicle, the target model layer file currently required in the storage system is dynamically loaded into the vehicle's memory, and corresponding inference operations are performed based on the target model layer file.

[0005] In the above technical solution, this method can directly identify the deployment target by explicitly obtaining the core weight file of the enterprise's proprietary model, thus providing a unique and accurate processing basis for the subsequent standardization and splitting process. Secondly, this method can also process the weight file based on a preset description file, achieving model format unification. This allows proprietary models that were originally incompatible with the vehicle deployment platform to meet platform adaptation requirements without modifying the underlying framework code, thereby lowering the threshold for format adaptation. Furthermore, this method can also split the complete large model based on the structural characteristics of the weight file, obtaining multiple lightweight layered files. This avoids excessive vehicle storage consumption by a single large file, while facilitating subsequent on-demand loading and adapting to situations with limited vehicle storage resources. Furthermore, this method can directly store the split, layered files on the vehicle, ensuring that the model data is completely stored locally. This allows the vehicle to access the model even in offline environments without a network, meeting the offline availability requirements of in-vehicle scenarios. Finally, this method can accurately load the target layer required for the current inference based on the layered files, without loading the entire model, thereby significantly reducing memory usage, avoiding lag caused by insufficient vehicle memory, and ensuring the compatibility of in-vehicle models across different hardware.

[0006] In some implementations, the model configuration information includes at least the model identifier, model architecture type, model hierarchical structure, precision configuration of each model layer, and weight file path mapping relationship corresponding to each model layer.

[0007] In the above technical solution, the method can clearly define the core components of the model configuration information, making the model standardization more thorough and the breakdown more precise, thereby solving the problems of adaptation and loading of self-owned model vehicle deployment at the underlying level.

[0008] In some implementations, the step of splitting the model configuration information according to the model weight file to obtain a hierarchical model file includes: The model weight file is parsed to obtain a parsed file; Based on the hierarchical structure defined in the model description file, the parsing file is split into multiple hierarchical files; Based on the preset specified layer application mixed precision quantization standard and the layer file, mixed precision quantization processing is performed to obtain the layer model file.

[0009] In the above technical solution, the method can make the model splitting not only fit the actual structure of the model, but also further compress the size of the layered files through accuracy optimization. This ensures that the split layered model files are completely matched with the model operation logic, avoids hierarchical confusion during subsequent inference, and can significantly reduce the storage occupation and memory consumption of each layered file while ensuring that the inference accuracy is basically up to standard, thus more accurately adapting to the limited storage and memory resources of the vehicle environment.

[0010] In some implementations, dynamically loading the target model layer file currently required in the storage system into the vehicle's memory includes: When performing inference on the vehicle, the target model layer file required for the current inference progress in the layered model file is dynamically loaded into the vehicle's memory, while model layer files not required for the current inference progress are retained in the vehicle's storage system.

[0011] In the above technical solution, this method enables on-demand loading of model layer files during inference on the vehicle side. Only the target model layer files necessary for the current inference progress are loaded into memory, while model layer files not currently needed remain in the storage system. This operation minimizes real-time memory usage, avoiding memory resource shortages or even lag caused by loading the complete model. Simultaneously, it eliminates the need for frequent reading and writing of all model files, reducing access pressure on the storage system and further adapting to the resource-constrained characteristics of the in-vehicle environment, ensuring a smooth and stable inference process.

[0012] In some implementations, dynamically loading the target model layer file required for the current inference progress from the hierarchical model file into the vehicle's memory includes: The target model layer file required for the current inference progress is mapped to the vehicle's memory by calling the memory mapping function.

[0013] In the above technical solution, this method can directly map the target model layer file to memory through a memory mapping function, eliminating the need for complete file reading and data copying operations. This significantly reduces the time overhead of data loading and improves the efficiency of dynamic loading. Simultaneously, this mapping method enables a direct association between file data and memory addresses, reducing memory redundancy and further optimizing the utilization of onboard memory resources, ensuring fast access and efficient operation of the model layer file during inference.

[0014] In some implementations, performing the corresponding inference operation based on the target model layer file includes: The target vehicle computing hardware is invoked through a predefined unified hardware abstraction interface, and corresponding inference operations are performed through the target vehicle computing hardware and the target model layer file.

[0015] In the above technical solution, the method can bypass the underlying differences of different vehicle computing hardware by using a predefined unified hardware abstraction interface, and directly call the adapted target hardware to perform inference operations. This avoids writing separate calling logic for different hardware, ensures that the proprietary model can run stably on various vehicle hardware, and enables the target hardware and the target model layer file to cooperate efficiently.

[0016] In some implementations, the method further includes, prior to invoking the target in-vehicle computing hardware via a predefined unified hardware abstraction interface: Obtain the vehicle's onboard computing hardware configuration information; A unified hardware abstraction interface is defined based on the vehicle computing hardware configuration information. Multiple vehicle computing hardware components are determined based on the vehicle computing hardware configuration information; A computing plugin instance is created for each of the in-vehicle computing hardware devices, and the computing plugin instance is initialized; wherein the computing plugin instance inherits the unified hardware abstraction interface.

[0017] In the above technical solution, this method first obtains the configuration information of the in-vehicle computing hardware, ensuring that the definition of the unified hardware abstraction interface and the creation of computing plugin instances are perfectly aligned with the actual hardware environment of the vehicle, thus avoiding the problem of interface-hardware incompatibility. Simultaneously, this method can create and initialize computing plugin instances that inherit from the unified interface for each hardware component, thereby unifying the hardware calling standard, avoiding the need for repetitive development of calling logic for different hardware, and enabling each hardware component to have its own dedicated adaptation plugin. This ensures efficient hardware response during subsequent calls, fundamentally improving the compatibility, adaptation efficiency, and operational stability of model inference across different in-vehicle hardware.

[0018] In some embodiments, the method further includes: When performing corresponding inference operations based on the target model layer file, determine the model layer file to be used for the next inference progress. The model layer file to be used is preloaded into the vehicle-side cache.

[0019] In the above technical solution, this method can pre-determine and preload the model layer files required for the next stage into the vehicle-side cache during the current inference operation, avoiding the waiting time of loading the next layer file from the storage system after the current inference is completed. This reduces interruptions during the inference process and significantly improves overall inference efficiency. Simultaneously, the fast cache read capability also helps reduce access pressure on the vehicle-side storage system, making the model inference experience in in-vehicle scenarios smoother.

[0020] Secondly, this application provides an offline deployment device for a model, comprising: The acquisition unit is used to acquire the model weight file of your own model; The first processing unit is used to standardize the model weight file according to a predefined model description file to obtain model configuration information. The second processing unit is used to split the model configuration information according to the model weight file to obtain a hierarchical model file; Storage unit, used to store the hierarchical model file in the vehicle-side storage system; The loading unit is used to dynamically load the target model layer file currently required in the storage system into the vehicle memory when performing inference on the vehicle side; The inference unit is used to perform corresponding inference operations based on the target model layer file.

[0021] In the above technical solution, the device can obtain the core weight file of the enterprise's own model, thus clarifying the target and laying an accurate foundation for subsequent processing. Then, it processes the weight file based on the preset description file to unify the model format, so that the proprietary model that was originally incompatible with the vehicle deployment platform can meet the adaptation requirements without modifying the underlying framework code, effectively reducing the format adaptation threshold. At the same time, according to the structural characteristics of the weight file, the complete large model is split into multiple lightweight layered files, which not only saves vehicle-side storage resources but also facilitates subsequent on-demand loading. Subsequently, the layered files are stored on the vehicle, allowing the model data to be stored locally, ensuring that the vehicle can still call the model in an offline environment without network access, meeting the offline usage requirements of vehicle scenarios. Finally, during the inference process, only the target model layer currently required is loaded, significantly reducing memory usage to avoid lag, while ensuring the model's compatibility on different vehicle hardware.

[0022] Thirdly, this application provides an electronic device including a memory and a processor, the memory storing a computer program, and the processor running the computer program to cause the electronic device to perform the offline deployment method of the model described in any one of the first aspects.

[0023] Fourthly, this application provides a readable storage medium storing a computer program, which, when executed by a processor, performs the offline deployment method of the model described in any one of the first aspects.

[0024] Fifthly, this application provides a computer program product comprising a computer program that, when executed by a processor, performs the offline deployment method for the model described in any one of the first aspects.

[0025] The beneficial effects of this application are as follows: It enables the definition of architecture and layering based on the model metadata specification in `model_schema.json`, decoupling proprietary models from the Ollam runtime and supporting automated processing and deployment. Simultaneously, by leveraging on-demand loading and memory mapping mechanisms for the model layer, it overcomes in-vehicle memory limitations, optimizing memory usage. Furthermore, based on the unified NPUPlugin interface and the dynamic expansion capabilities of PluginFactory, it also enables flexible adaptation to hardware plugins and seamless integration with new hardware. Overall, this method not only solves the compatibility issues between proprietary models and deployment frameworks and hardware environments but also meets the high-efficiency, stable, and scalable performance requirements of offline model deployment in in-vehicle scenarios through resource optimization and automated processes. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart illustrating the offline deployment method of the model in some embodiments of this application; Figure 2 This is a schematic diagram of the proprietary model standardization and adaptation process in some embodiments of this application; Figure 3 This is a schematic diagram illustrating the process of model layer memory mapping and on-demand loading in the vehicle-side layered dynamic loading engine in some embodiments of this application; Figure 4 This is a schematic diagram of the heterogeneous computing plug-in adaptation process in some embodiments of this application; Figure 5 This is a schematic diagram of the structure of the offline deployment device for models in some embodiments of this application; Figure 6 This is a schematic diagram of the structure of an electronic device in some embodiments of this application. Detailed Implementation

[0028] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0029] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0030] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more (including two), similarly, "multiple sets" refers to two or more sets (including two sets), and "multiple pieces" refers to two or more pieces (including two pieces) unless otherwise explicitly defined.

[0031] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0032] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0033] Currently, Ollam, an open-source platform, is commonly used in the industry for deploying large language models in in-vehicle scenarios. However, while it supports local deployment of mainstream models such as LLaMA, DeepSeek, and Tongyi Qianwen, deploying enterprise-owned trained models (such as self-developed Transformer architecture models) typically requires addressing incompatibility issues between the enterprise's own model and Ollam's natively supported models in terms of weight format and layer structure. This necessitates non-standardized integration methods, such as manually modifying the Ollam kernel code, writing customized loading scripts, or modifying the underlying code of llama.cpp, resulting in a cumbersome deployment process, high technical barriers, and a lack of compatibility guarantees. Furthermore, Ollam's ability to coordinate and schedule heterogeneous hardware such as in-vehicle CPUs, NPUs, and GPUs is limited, leading to insufficient utilization of heterogeneous resources. Overall, existing technologies lack a vehicle-side collaborative optimization system for enterprise-owned models and the Ollam platform, ultimately resulting in complex deployment processes, significant resource waste, and poor real-time interactive experience for in-vehicle models.

[0034] To address the aforementioned technical issues, this application provides an offline model deployment method. This method enables one-click conversion of proprietary models through model description files and automated toolchains, significantly reducing manual coding workload and lowering development costs, allowing even non-algorithm engineers to complete deployment. Simultaneously, by leveraging layered dynamic loading and heterogeneous computing adaptation, it allows proprietary models with large parameter counts to run stably on low-memory automotive hardware, supporting mid-range automotive chips to support large parameter models, covering mainstream automotive terminal devices, and expanding hardware compatibility. Furthermore, by using a plug-in mechanism to adapt proprietary model operators to the automotive NPU, it avoids modifications to the Ollam core code, thereby improving computational efficiency while achieving convenient ecosystem integration. This method specifically addresses the problems of cumbersome deployment processes, poor compatibility, and insufficient resource utilization in existing technologies.

[0035] like Figure 1 As shown, some embodiments of this application provide a method for offline model deployment, which includes: S101. Obtain the model weight file of your own model; S102. Standardize the model weight file according to the predefined model description file to obtain the model configuration information; S103. Based on the model weight file, the model configuration information is split to obtain a hierarchical model file; S104. Store the layered model file in the vehicle-side storage system; S105. When performing inference on the vehicle side, dynamically load the target model layer file currently required in the storage system into the vehicle side memory, and perform corresponding inference operations based on the target model layer file.

[0036] In some embodiments, a proprietary model refers to a dedicated large language model that is independently developed and trained by an enterprise, has specific functions, or is adapted to specific scenarios. Its model structure, weight format, etc., are usually different from the models natively supported by open source platforms.

[0037] In some embodiments, a model weight file refers to a file containing parameter information learned during the training of the proprietary model, which determines the model's inference ability and performance.

[0038] In some embodiments, a model description file refers to a predefined file used to standardize model-related information, including model architecture, hierarchical structure, weight mapping rules, and other content.

[0039] In some embodiments, model configuration information refers to structured information obtained after standardizing the model weight file, including at least model identifier, architecture type, hierarchical structure, precision configuration of each layer, and weight file path mapping relationship.

[0040] In some embodiments, a hierarchical model file refers to multiple lightweight files obtained by splitting the model configuration information based on the structural characteristics of the model weight file, with each file corresponding to one or more levels of the model.

[0041] In some embodiments, the vehicle-side storage system refers to a hardware device or system on a vehicle used for storing data, capable of persistently storing hierarchical model files and supporting model retrieval in an offline environment.

[0042] In some embodiments, the target model layer file refers to a specific layer file selected from the layered model files that is necessary for the current inference progress during the vehicle-side inference process.

[0043] In the above embodiments, this method can directly identify the deployment object by explicitly obtaining the core weight file of the enterprise's proprietary model, thus providing a unique and accurate processing basis for the subsequent standardization and splitting process. Secondly, this method can also process the weight file based on a preset description file to achieve model format unification, allowing proprietary models that were originally incompatible with the vehicle deployment platform to meet platform adaptation requirements without modifying the underlying framework code, thereby lowering the threshold for format adaptation. Furthermore, this method can also split the complete large model based on the structural characteristics of the weight file, obtaining multiple lightweight layered files. This avoids excessive vehicle-side storage consumption by a single large file, while facilitating subsequent on-demand loading and adapting to situations with limited vehicle storage resources. Furthermore, this method can directly store the split, layered files on the vehicle, ensuring that the model data is completely stored locally. This allows the vehicle to access the model even in offline environments without a network, meeting the offline availability requirements of in-vehicle scenarios. Finally, this method can accurately load the target layer required for the current inference based on the layered files, without loading the entire model, thereby significantly reducing memory usage, avoiding lag caused by insufficient vehicle-side memory, and ensuring the compatibility of in-vehicle models across different hardware.

[0044] In some embodiments, the model configuration information includes at least the model identifier, model architecture type, model hierarchical structure, precision configuration of each model layer, and weight file path mapping relationship corresponding to each model layer.

[0045] In the above embodiments, the method can clearly define the core components of model configuration information, making model standardization more thorough and splitting more precise, thereby solving problems such as adaptation and loading of self-owned model vehicle deployment at the underlying level.

[0046] In some embodiments, the model configuration information is split according to the model weight file to obtain a hierarchical model file, including: The model weight file is parsed to obtain the parsed file; Based on the hierarchical structure defined in the model description file, the parsing file is split into multiple hierarchical files; Based on the preset specified layer application mixed precision quantization standard and layer file, mixed precision quantization processing is performed to obtain the layer model file.

[0047] In some embodiments, when a user issues a command on the vehicle terminal (such as “plan a route from point A to point B”), Ollam runtime triggers: based on the model_schema.json standardized adaptation model, it completes metadata parsing and hierarchical file recognition (i.e., proprietary model standardized adaptation: parsing schema and hierarchical files).

[0048] In the above embodiments, the method enables the model splitting to not only fit the actual structure of the model but also to further compress the size of the layered files through accuracy optimization. This ensures that the split layered model files are completely matched with the model's computational logic, avoiding hierarchical errors during subsequent inference. Furthermore, it significantly reduces the storage footprint and memory consumption of each layered file while ensuring that the inference accuracy is basically up to standard, making it more accurately suited to the limited storage and memory resources of the in-vehicle environment.

[0049] In some embodiments, dynamically loading the target model layer file currently required in the storage system into the vehicle's memory includes: When performing inference on the vehicle, the target model layer files required for the current inference progress are dynamically loaded into the vehicle's memory, while model layer files not required for the current inference progress are retained in the vehicle's storage system.

[0050] In some embodiments, this method can map model layers to memory on demand through a hierarchical dynamic loading mechanism. That is, it maps active layers via mmap while preloading the next layer.

[0051] In the above embodiments, this method enables on-demand loading of model layer files during in-vehicle inference, loading only the target model layer files necessary for the current inference progress into memory, while leaving the non-currently needed model layer files in the storage system. This operation minimizes real-time memory usage, avoiding memory resource shortages or even lag caused by loading the complete model. Simultaneously, it eliminates the need for frequent reading and writing of all model files, reducing access pressure on the storage system and further adapting to the resource-constrained characteristics of the in-vehicle environment, ensuring a smooth and stable inference process.

[0052] In some embodiments, dynamically loading the target model layer file required for the current inference progress from the hierarchical model file into the vehicle's memory includes: The target model layer file required for the current inference progress is mapped to the vehicle's memory by calling the memory mapping function.

[0053] In the above embodiments, this method can directly map the target model layer file to memory through a memory mapping function, eliminating the need for complete file reading and data copying operations. This significantly reduces the time overhead of data loading and improves the efficiency of dynamic loading. Simultaneously, this mapping method enables a direct association between file data and memory addresses, reducing memory redundancy and further optimizing the utilization of onboard memory resources, ensuring fast access and efficient operation of the model layer file during inference.

[0054] In some embodiments, performing corresponding inference operations based on the target model layer file includes: The target vehicle computing hardware is invoked through a predefined unified hardware abstraction interface, and corresponding inference operations are performed through the target vehicle computing hardware and the target model layer file.

[0055] In some embodiments, the method can leverage a heterogeneous computing plugin to call the vehicle-side NPU to accelerate inference and ultimately return the results to the user.

[0056] In the above embodiments, the method can bypass the underlying differences of different vehicle computing hardware by using a predefined unified hardware abstraction interface, and directly call the adapted target hardware to perform inference operations. This avoids writing separate calling logic for different hardware, ensures that the proprietary model can run stably on various vehicle hardware, and enables the target hardware and target model layer files to cooperate efficiently.

[0057] In some embodiments, before invoking the target in-vehicle computing hardware through a predefined unified hardware abstraction interface, the method further includes: Obtain the vehicle's onboard computing hardware configuration information; Define a unified hardware abstraction interface based on the vehicle computing hardware configuration information; Multiple on-board computing hardware components are identified based on the on-board computing hardware configuration information; Create a computing plugin instance for each in-vehicle computing hardware and initialize the computing plugin instance; the computing plugin instance inherits the unified hardware abstraction interface.

[0058] In the above embodiments, this method first obtains the configuration information of the in-vehicle computing hardware, ensuring that the definition of the unified hardware abstraction interface and the creation of computing plugin instances are perfectly aligned with the actual hardware environment of the vehicle, thus avoiding the problem of interface-hardware incompatibility. Simultaneously, this method can create and initialize computing plugin instances that inherit from the unified interface for each hardware component, thereby unifying the hardware calling standard, avoiding the repeated development of calling logic for different hardware, and enabling each hardware component to have its own dedicated adaptation plugin. This ensures that the hardware can respond efficiently during subsequent calls, fundamentally improving the compatibility, adaptation efficiency, and operational stability of model inference across different in-vehicle hardware.

[0059] In some embodiments, the method further includes: When performing corresponding inference operations based on the target model layer file, determine the model layer file to be used for the next inference progress. The model layer files to be used are preloaded into the vehicle-side cache.

[0060] In the above embodiments, this method can pre-determine and preload the model layer files required for the next stage into the vehicle-side cache during the current inference operation, avoiding the waiting time of loading the next layer file from the storage system after the current inference is completed. This reduces interruption intervals during inference and significantly improves overall inference efficiency. Simultaneously, the fast cache read capability also helps reduce access pressure on the vehicle-side storage system, making the model inference experience in in-vehicle scenarios smoother.

[0061] To make the purpose, technical solution and advantages of this application clearer, the technical solution in this application will be clearly and completely described below.

[0062] In some embodiments, Figure 2 This demonstrates a proprietary model standardization and adaptation process system.

[0063] In some embodiments, the system designs a JSON-formatted model description file, Model_Schema.json, which can decouple the model architecture from the Ollam runtime by standardizing the metadata of its own model (such as model name, architecture type, information of each layer, correspondence of weight files, etc.), providing a basis for the automated parsing and quantization configuration generation of weight files.

[0064] For example, the sample content of a JSON-formatted model description file (model_schema.json) is shown below: { "model_name":"VehicleLLM-13B", "architecture":"CustomTransformer", "layers":[ {"name":"embedding_layer","type":"embedding","precision":"fp16"}, {"name":"transformer_layer_0","type":"attention","quantization":"int8"} ], "weight_files":{ "embedding":"embedding.bin", "transformer":["layer_0.bin","layer_1.bin","...","layer_23.bin"] }}

[0065] In some embodiments, the system also builds an automated conversion toolchain that can parse the weight files of its own models (such as PyTorch's .pt files) and split them into layered files according to the definition of model_schema.json (each layer is stored separately); at the same time, it can call Ollama's quantization library to apply mixed precision quantization to specified model layers (for example, the embedding layer retains FP16 precision, and the Transformer layer is quantized to INT8).

[0066] For example, the command to invoke an automated conversion toolchain is as follows: pythonmodel_converter.py --input_modelcustom_model.pt --schemamodel_schema.json --output_dirollama_models / VehicleLLM-13B.

[0067] In some embodiments, Figure 3 This paper illustrates a flowchart of model layer memory mapping and on-demand loading in a vehicle-side hierarchical dynamic loading engine. The memory management mechanism in the vehicle-side hierarchical dynamic loading engine includes: Layered storage mechanism: This mechanism can split the model into N files (such as layer_0.bin to layer_N-1.bin) by layer and store them in the / ollama / models / VehicleLLM directory of the vehicle eMMC; Dynamic loading mechanism: This mechanism can be used at the Ollam runtime via the mmap() system call to map only the current inference layer to memory, while inactive layers remain in disk storage state, significantly reducing memory usage; Layer switching optimization mechanism: This mechanism can utilize the CPU cache to preload the weights of the next layer, with inter-layer switching latency of less than 15ms, ensuring inference continuity.

[0068] In some embodiments, Figure 4 A schematic diagram of a heterogeneous computing plug-in adaptation process is shown.

[0069] In some embodiments, the method can define a unified NPU Plugin interface and declare standard methods such as initialization, model loading, inference, and release.

[0070] For example, the class definition code for the NPU Plugin unified interface is shown below: class NPUPlugin { public: virtual bool init(const char* config) = 0; virtual void* load_model(const char* path) = 0; virtual bool run_inference(void* model, const void* input, void*output) = 0; virtual void release_model(void* model) = 0; virtual ~NPUPlugin() = default; }

[0071] In some embodiments, the method can implement plugins for different computing hardware (such as CPU and GPU) based on the unified interface, so that upper-layer logic can call it without distinguishing hardware type.

[0072] For example, the implementation code of CPUPlugin and GPUPlugin inheriting the NPUPlugin interface is shown below: class CPUPlugin : public NPUPlugin { public: bool init(const char* config) override { return true;} void* load_model(const char* path) override { return nullptr;} bool run_inference(void* model, const void* input, void* output)override { return true;} void release_model(void* model) override {} }; class GPUPlugin : public NPUPlugin { public: bool init(const char* config) override { / * Initialize CUDA * / return true;} void* load_model(const char* path) override { / * Load GPU model * / return nullptr;} bool run_inference(void* model, const void* input, void* output)override { return true;} void release_model(void* model) override {} }

[0073] In some embodiments, this method can design a static factory class PluginFactory and dynamically create corresponding plugin instances based on a specified hardware type string.

[0074] For example, the code for the PluginFactory static factory class used to dynamically create hardware plugin instances is shown below: class PluginFactory { public: static NPUPlugin* create(const char* type) { if (strcmp(type, "cpu") == 0) return new CPUPlugin(); if (strcmp(type, "gpu") == 0) return new GPUPlugin(); return nullptr; } }

[0075] In some embodiments, the method can obtain the required plugins through a factory at runtime, complete the initialization according to the configuration file, call a unified interface to perform model inference, and finally release resources.

[0076] For example, the code in the main program that obtains the plugin through the factory, completes initialization, and performs inference is shown below: int main() { const char* npu_type = "gpu"; / / Get from configuration or environment NPUPlugin* plugin = PluginFactory::create(npu_type); if (plugin&&plugin->init("config.json")) { void* model = plugin->load_model("model.onnx"); plugin->run_inference(model, input, output); plugin->release_model(model); } delete plugin; / / Ensure resources are released return 0; }

[0077] Figure 5 A schematic diagram of a model offline deployment device is shown. It should be understood that this device is related to... Figure 1 The method executed in the middle corresponds to the steps involved in the aforementioned method. The specific functions and effects of the device can be found in the description above. To avoid repetition, detailed descriptions are omitted here.

[0078] The offline deployment device for this model includes: Unit 210 is used to obtain the model weight file of the proprietary model; The first processing unit 220 is used to standardize the model weight file according to the predefined model description file to obtain model configuration information; The second processing unit 230 is used to split the model configuration information according to the model weight file to obtain a hierarchical model file; Storage unit 240 is used to store the hierarchical model file in the vehicle-side storage system; The loading unit 250 is used to dynamically load the target model layer file currently required in the storage system into the vehicle memory when performing inference on the vehicle side; The inference unit 260 is used to perform corresponding inference operations based on the target model layer file.

[0079] In some embodiments, the model configuration information includes at least the model identifier, model architecture type, model hierarchical structure, precision configuration of each model layer, and weight file path mapping relationship corresponding to each model layer.

[0080] In some embodiments, the second processing unit 230 includes: Parsing subunit 231 is used to parse the model weight file to obtain a parsed file; The splitting subunit 232 is used to split the parsed file into multiple hierarchical files according to the hierarchical structure defined in the model description file; Processing subunit 233 is used to perform mixed precision quantization processing according to the preset specified layer application mixed precision quantization standard and layer file to obtain layer model file.

[0081] In some embodiments, the loading unit 250 is specifically used to dynamically load the target model layer file required for the current inference progress in the layered model file into the vehicle memory when performing inference on the vehicle side, and to retain the model layer file not required for the current inference progress in the vehicle storage system.

[0082] In some embodiments, the loading unit 250 is further configured to map the target model layer file required for the current inference progress to the vehicle-side memory by calling a memory mapping function.

[0083] In some embodiments, the inference unit 260 is specifically used to call the target vehicle computing hardware through a predefined unified hardware abstraction interface, and to perform corresponding inference operations through the target vehicle computing hardware and the target model layer file.

[0084] In some embodiments, the offline deployment apparatus for the model further includes: The acquisition unit 210 is also used to acquire the vehicle-side vehicle computing hardware configuration information before the inference unit 260 calls the target vehicle computing hardware through a predefined unified hardware abstraction interface. Unit 270 is used to define a unified hardware abstraction interface based on the vehicle computing hardware configuration information. The determining unit 270 is also used to determine multiple on-board computing hardware based on the on-board computing hardware configuration information; The initialization unit 280 is used to create computing plugin instances for each vehicle computing hardware and initialize the computing plugin instances; wherein, the computing plugin instances inherit the unified hardware abstraction interface.

[0085] In some embodiments, the offline deployment apparatus for the model further includes: The determining unit 270 is also used to determine the model layer file to be used for the next inference progress when performing corresponding inference operations based on the target model layer file; The preloading unit 290 is used to preload the model layer file to be used into the vehicle-side cache.

[0086] like Figure 6As shown, this application provides an electronic device 300, which includes a processor 301 and a memory 302. The processor 301 and the memory 302 are interconnected and communicate with each other through a communication bus 303 and / or other forms of connection mechanism (not shown). The memory 302 stores a computer program that can be executed by the processor 301. When the computing device is running, the processor 301 executes the computer program to perform the method in any of the aforementioned optional implementations.

[0087] This application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the method in any of the aforementioned optional implementations.

[0088] The computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0089] This application provides a computer program product, which includes a computer program that, when run by a processor, executes the method in any of the aforementioned optional implementations.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and they should all be covered within the scope of the claims and specification of this application. In particular, as long as there is no conflict, the various technical features mentioned in the embodiments can be combined in any way. This application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A model offline deployment method, characterized in that, The method comprises the following steps: obtaining a model weight file of a self-owned model; standardizing the model weight file according to a pre-defined model description file to obtain model configuration information; splitting the model configuration information according to the model weight file to obtain a layered model file; storing the layered model file in a vehicle-side storage system; when performing reasoning on the vehicle side, dynamically loading a target model layer file currently required in the storage system to a vehicle-side memory, and performing corresponding reasoning operation based on the target model layer file.

2. The model offline deployment method of claim 1, wherein, The model configuration information at least comprises a model identifier, a model architecture type, a model layer structure, precision configuration of each model layer, and a weight file path mapping relationship corresponding to each model layer.

3. The model offline deployment method of claim 1, wherein, The splitting the model configuration information according to the model weight file to obtain a layered model file comprises the following steps: parsing the model weight file to obtain a parsed file; splitting the parsed file into multiple layered files according to a hierarchical structure defined in the model description file; performing mixed precision quantization processing according to a pre-set specified layer mixed precision quantization standard and the layered files to obtain a layered model file.

4. The model offline deployment method of claim 1, wherein, The dynamically loading the target model layer file currently required in the storage system to the vehicle-side memory comprises the following steps: when performing reasoning on the vehicle side, dynamically loading a target model layer file required by a current reasoning progress in the layered model file to a vehicle-side memory, and retaining a model layer file not required by a current reasoning progress in the vehicle-side storage system.

5. The model offline deployment method of claim 4, wherein, The dynamically loading the target model layer file currently required by the reasoning progress to the vehicle-side memory comprises the following steps: mapping the target model layer file currently required by the reasoning progress to the vehicle-side memory by calling a memory mapping function.

6. The model offline deployment method of claim 1, wherein, The performing corresponding reasoning operation based on the target model layer file comprises the following steps: calling a target vehicle-mounted computing hardware through a pre-defined unified hardware abstraction interface, and performing corresponding reasoning operation through the target vehicle-mounted computing hardware and the target model layer file.

7. The model offline deployment method of claim 6, wherein, Before the calling the target vehicle-mounted computing hardware through the pre-defined unified hardware abstraction interface, the method further comprises the following steps: obtaining vehicle-mounted computing hardware configuration information of the vehicle side; defining a unified hardware abstraction interface according to the vehicle-mounted computing hardware configuration information; determining a plurality of vehicle-mounted computing hardware according to the vehicle-mounted computing hardware configuration information; creating a computing plug-in instance for each vehicle-mounted computing hardware, and initializing the computing plug-in instance; wherein the computing plug-in instance inherits the unified hardware abstraction interface.

8. The model offline deployment method of claim 1, wherein, The method further comprises the following steps: when performing corresponding reasoning operation based on the target model layer file, determining a to-be-used model layer file required by a next reasoning progress; preloading the to-be-used model layer file to a vehicle-side cache.

9. A model offline deployment apparatus characterized by comprising: The model offline deployment device comprises: an obtaining unit configured to obtain a model weight file of a self-owned model; a first processing unit configured to standardize the model weight file according to a pre-defined model description file to obtain model configuration information; A second processing unit is configured to split the model configuration information according to the model weight file to obtain a layered model file; A storage unit is configured to store the layered model file in an on-vehicle storage system; A loading unit is configured to dynamically load a target model layer file currently required in the storage system to an on-vehicle memory when performing inference on the vehicle; An inference unit is configured to perform corresponding inference operation based on the target model layer file.

10. An electronic device, comprising: The electronic device includes a memory configured to store a computer program and a processor configured to execute the computer program to cause the electronic device to perform the model offline deployment method in any one of claims 1 to 8.

11. A readable storage medium, characterized by, The readable storage medium stores a computer program, and the computer program is executed by the processor to perform the model offline deployment method in any one of claims 1 to 8.

12. A computer program product, characterised in that, The computer program product includes a computer program, and the computer program is executed by the processor to perform the model offline deployment method in any one of claims 1 to 8.

Images