Intelligent inference system, method, equipment and medium
By introducing LoRA model adapter and intelligent router mechanism in a low video memory environment, efficient deployment and inference of multiple expert models is achieved, video memory bottleneck problem is solved, inference response speed is improved, and it is suitable for scenarios such as edge computing and mobile devices.
Patent Information
- Application Number
- CN202510161209.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-13
- Publication Date
- 2025-06-06
AI Technical Summary
In low-video memory environments, when executing large-scale deep learning models, memory resources become a bottleneck that limits model inference performance, resulting in increased inference latency and slower system response, especially in the deployment of multi-expert models.
By introducing the adapter design and intelligent router mechanism of the LoRA model, the separate and independent deployment of the dock model and the adapter model is realized, supporting multiple adapter models to be inferred in parallel on one dock model, dynamically loading the adapter model, and optimizing the loading strategy of the adapter through the intelligent router.
It significantly reduces the use of video memory, improves the inference response speed, meets the inference needs of low resources and high response, and is especially suitable for scenarios such as edge computing and mobile devices.
Smart Images

Figure CN120106213A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computing power scheduling, and in particular to an intelligent inference system, method, device and medium. Background Art
[0002] With the rapid development of deep learning technology, large-scale deep learning models are widely used in more and more fields, such as natural language processing, computer vision, and speech recognition. However, during the inference process, especially when executing large-scale deep learning models on low-resource devices (such as smartphones, IoT devices, edge computing nodes, etc.), graphics memory resources often become a bottleneck that limits model inference performance.
[0003] Existing deep learning models usually need to load a large number of parameters to complete inference tasks, which places very high demands on the hardware's video memory. In an environment with limited video memory, running these large models becomes very difficult, especially when multiple expert models need to be dynamically loaded during the inference process. The use of video memory becomes more intense, resulting in increased inference latency and slower system response speed. This is especially a serious problem for application scenarios with high real-time requirements (such as mobile devices, edge computing, etc.).
[0004] In order to solve the problem of video memory bottleneck, some methods have been proposed in the prior art, such as model compression, model distillation, and quantization. These methods reduce the pressure on video memory by reducing the number of model parameters and reducing the complexity of the model. However, these methods are usually accompanied by a certain performance loss, and the inference accuracy is often reduced, which cannot fully meet the dual requirements of efficient inference and high accuracy. In addition, for the application of multi-expert models, although these models can provide higher inference accuracy through the combination of different experts, the deployment of multi-expert models in a low video memory environment faces the challenge of frequently loading different expert models. Especially during the cold start process, the loading time of the model significantly affects the inference speed, resulting in a high inference delay. Therefore, how to efficiently deploy multi-expert models in a low video memory environment and reduce video memory occupancy and inference delay while ensuring inference accuracy has become a technical problem that needs to be solved urgently. Summary of the invention
[0005] One purpose of the present application is to provide an intelligent inference system, method, device and medium, at least to solve the technical problem of low efficiency of expert model inference.
[0006] To achieve the above objectives, some embodiments of the present application provide the following aspects:
[0007] In the first aspect, some embodiments of the present application also provide an intelligent inference system, including a base model, which is used to perform the inference process of shared parameters; multiple adapter models, each of which is used to perform the inference process of an inference task; the base model is separated from the adapter model and independently deployed, supporting parallel inference of multiple adapter models on one base model; and an intelligent router, which is used to receive and analyze inference tasks, select the corresponding adapter model according to preset rules, and load the adapter model to the corresponding inference node.
[0008] On the second aspect, some embodiments of the present application also provide an intelligent inference method, which is applied to the system as described above, and the method includes receiving and analyzing the inference task through an intelligent router; analyzing the inference task according to preset rules or a lightweight machine learning model, and selecting a corresponding adapter model; loading a base model according to the inference task, and performing an inference process of shared parameters; selecting a corresponding adapter model according to the inference task, loading the adapter model to the inference node, and executing the inference task.
[0009] In a third aspect, some embodiments of the present application further provide an electronic device, comprising: one or more processors; and a memory storing computer program instructions, wherein the computer program instructions, when executed, cause the processor to perform the steps of the method described above.
[0010] In a fourth aspect, some embodiments of the present application further provide a computer-readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method as described above.
[0011] Compared with the related technology, the solution provided in the embodiment of the present application realizes the efficient deployment and inference of multiple expert models in a low video memory environment by introducing the adapter design and intelligent router mechanism of the LoRA model, which can significantly reduce the use of video memory and improve the inference response speed, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS
[0012] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings represent similar elements, and unless otherwise stated, the figures in the drawings do not constitute proportional limitations.
[0013] Figure 1 A schematic diagram of the structure of an intelligent inference system provided according to an embodiment of the present application;
[0014] Figure 2 A flowchart of an intelligent inference method provided according to an embodiment of the present application;
[0015] Figure 3 The present invention is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION
[0016] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0017] First embodiment
[0018] The first embodiment of the present application relates to an intelligent inference system. Figure 1 As shown, the system may include:
[0019] The base model is used for the inference process of shared parameters. The base model contains the basic parameters and core calculation logic shared by all inference tasks, so it undertakes the calculation tasks of the unchanged part of the inference process. By concentrating these common parameter calculations in the base model, repeated calculations can be avoided each time a different adapter model is loaded, improving the overall calculation efficiency.
[0020] Since the base model is only loaded once and is responsible for the inference of shared parameters, the repeated occupation of video memory is reduced, which is particularly suitable for low-end devices and edge computing environments with limited video memory resources. The efficient deployment and independent calculation of the base model enable it to work in parallel with multiple adapter models, thereby improving the inference speed and system response time. When multiple inference tasks are in parallel, the base model reduces waiting time and improves multi-tasking capabilities by parallel calculation of shared parameters.
[0021] Multiple adapter models, each of which is used to perform an inference process of an inference task. An adapter model is a sub-model customized according to a specific inference task, and each adapter model is specifically used to perform a specific inference task, such as classification, recognition or other operations on specific input data.
[0022] Each adapter model only loads specific parameters related to its task, and uses LoRA technology (low-rank adaptation) to optimize its parameters, thereby significantly reducing the memory usage. This enables multiple adapters to work in parallel on the same base model to support the inference requirements of different tasks.
[0023] The adapter model is not loaded at initialization, but dynamically loaded when the inference task requires it. This on-demand loading mechanism effectively reduces the system's memory consumption and improves resource utilization efficiency. Especially in low-memory environments, it reduces the waste of memory resources. The use of LoRA technology makes the adapter model have a lower parameter count, further reducing the memory usage without significantly losing inference accuracy. This is especially important in the deployment of multi-expert models to ensure efficient and accurate inference.
[0024] The base model is deployed separately from the adapter model, supporting parallel inference of multiple adapter models on one base model. Separate deployment means that the base model and the adapter model are physically or logically independent. The base model handles shared, common parameters and inference logic, while the adapter model is responsible for handling inference of specific tasks.
[0025] The base model and the adapter model are deployed on independent storage spaces and computing resources, so changes to the adapter model will not affect the base model, and vice versa. This separation design provides flexibility, allowing the system to dynamically select the adapter model based on demand without frequently modifying the base model.
[0026] The base model and multiple adapter models can work in parallel at the same time. This parallel inference mode allows each adapter model to focus on processing different tasks, while the base model is only responsible for providing shared computing resources and parameters. Different adapter models can independently perform their respective tasks with the support of the base model, while saving video memory resources and computing resources. Through this parallel processing, the processing efficiency of the inference system can be greatly improved, especially in the scenario of multi-task parallel inference, reducing waiting and switching time.
[0027] The intelligent router is used to receive and analyze the inference task, select the corresponding adapter model according to the preset rules, and load the adapter model to the corresponding inference node. The intelligent router dynamically selects the appropriate adapter model according to the characteristics of the inference task (such as input data type, computing load, etc.), avoiding invalid adapter loading and reducing the waste of video memory.
[0028] It is not difficult to find that compared with the related art, the solution provided in the embodiment of the present application realizes the efficient deployment and inference of multiple expert models in a low video memory environment by introducing the adapter design and intelligent router mechanism of the LoRA model, which can significantly reduce the use of video memory and improve the inference response speed at the same time, and has broad application prospects. Since the LoRA adapter model is deployed separately from the base model, the adapter is loaded only when needed, which greatly reduces the use of video memory. The intelligent distribution mechanism of the router reduces the cold loading time of the adapter and improves the real-time performance of the inference process. Multiple adapter models can be deployed in parallel on the same base model to support the inference requirements of multiple tasks. The system is particularly suitable for devices with limited video memory, such as edge computing devices, mobile devices, etc.
[0029] Second embodiment
[0030] The second embodiment of the present application relates to an intelligent inference system. The second implementation is an improvement on the first embodiment, and the specific improvements are:
[0031] Furthermore, the adapter model adopts LoRA model technology to reduce the number of parameters of the adapter model by low-rank decomposition.
[0032] LoRA (Low-Rank Adaptation) is a method for efficiently fine-tuning large pre-trained models by introducing low-rank matrices to reduce the computational cost of model training and inference. Specifically, the LoRA method does not directly adjust all the parameters of the large model, but introduces a low-rank matrix (usually composed of the product of several smaller matrices) to adjust the parameters of the model. The introduction of this low-rank matrix significantly reduces the number of parameters and reduces the computational burden, while retaining sufficient model expression capabilities.
[0033] The LoRA model technology is used to build a small adapter model for each expert model. The role of each adapter model is to fine-tune the shared parameters of the base model according to specific task requirements to achieve task-specific inference goals.
[0034] Build independent adapter models for each task (such as image classification, speech recognition, etc.), which have smaller parameters, so they can be loaded and unloaded independently, avoiding high video memory usage. Since the adapter model parameters are significantly reduced, LoRA reduces video memory usage compared to traditional full parameter fine-tuning methods and can run efficiently on devices with limited video memory.
[0035] Multiple LoRA adapter models can work in parallel on the same base model, and each adapter model only loads the low-rank parameters it needs, greatly reducing resource waste and loading time, and improving inference speed.
[0036] Furthermore, the adapter model does not load video memory resources when being deployed, but loads video memory resources when performing an inference task.
[0037] The adapter model runs efficiently in a low-memory environment, so the adapter model parameters are not loaded into the video memory during system initialization or deployment. Only the base model needs to be loaded at startup, and the adapter model is only loaded when there is an inference demand. This lazy loading strategy avoids wasting video memory resources and frees up more video memory space for subsequent inference tasks.
[0038] Traditional system designs may load all models at system startup, regardless of whether they will be used immediately. In contrast, the adapter model of this design does not occupy video memory when deployed, avoiding excessive occupation of video memory, especially on devices with multi-tasking parallel processing or limited hardware resources (such as embedded devices, IoT devices, etc.).
[0039] When the system receives an inference task, it dynamically loads the corresponding adapter model into the video memory according to the requirements of the task. This method ensures that video memory resources are allocated only when needed, avoids preloading unnecessary adapter models, and reduces the pressure on the video memory. At this time, the system loads the corresponding adapter according to the characteristics of the current task (such as task type, input data, etc.) to ensure that the task can be executed efficiently. When dynamically loading the adapter model, combined with the advance scheduling mechanism of the intelligent router module, the adapter model that may be used can be preloaded based on historical data or task characteristics before the inference task arrives. In this way, during actual inference, the cold loading delay will be greatly reduced, ensuring a faster inference process.
[0040] Furthermore, the system includes: when performing the inference task, loading the base model to infer the shared parameters, and loading the corresponding adapter model according to the specific requirements of the inference task.
[0041] When performing an inference task, the system first loads the base model to process the shared parameters, and then loads the corresponding adapter model according to the requirements of the specific inference task to perform task-specific inference. The base model plays a core role in the system and is responsible for the inference process of shared parameters in all inference tasks. The base model generally contains the common part of the model and most of the common computing parameters, and it carries the shared computing part of the task.
[0042] The base model is loaded into the video memory at the beginning of the inference task and is responsible for all general computing tasks. The base model is a relatively stable component that does not change according to the task type, so it is uniformly loaded once when the task is executed and reused. After loading the base model, the system can perform all shared parameter calculations in the task that do not depend on a specific task, reducing the resource consumption of repeated loading. For example, multiple inference tasks may require the same neural network layer. The base model provides shared calculation results of these layers, avoiding repeated calculations of the same content between different tasks, and significantly improving the computing efficiency of the system.
[0043] When performing inference, although the base model can provide the calculation of shared parameters, different inference tasks usually require specific task parameters or features, which are usually provided by the adapter model. Therefore, the system will dynamically load the corresponding adapter model according to the specific needs of the current inference task.
[0044] The adapter model is optimized through LoRA technology (low-rank adaptation) and has a small number of parameters. When a specific task needs to be processed, the system will dynamically load the corresponding adapter model into the video memory according to the task type, input data, etc. The adapter model is separated from the base model and deployed independently to avoid unnecessary video memory usage.
[0045] Each adapter model is designed for a specific task, and it can perform specific calculations related to the task according to the input data type, target output, etc. of the inference task. For example, an image classification task may require an adapter for image feature extraction, while a speech recognition task may require an adapter for speech signals. Depending on the inference task, the intelligent router will dynamically select and load the appropriate adapter model to ensure that the inference task can proceed smoothly. This on-demand loading mechanism not only reduces the use of video memory, but also improves the efficiency of inference. Because each adapter model is loaded only when needed, the system's video memory and computing resources are maximized, and performance loss caused by unnecessary loading is avoided.
[0046] Furthermore, the intelligent router adopts a selection mechanism based on a combination of static rules and small models to analyze the inference task and select the corresponding adapter model.
[0047] Static rules are a set of rules preset by the system to quickly make decisions and select the appropriate adapter model based on the input characteristics and requirements of the inference task. Static rules are usually based on common and predictable input data patterns and task types. Static rules can quickly select the most appropriate adapter model based on the input characteristics of the inference task, such as data type, task category, etc.
[0048] Small models refer to lightweight machine learning models that are used to dynamically analyze inference tasks and select the most suitable adapter model. Unlike traditional deep learning models, small models usually have lower computational complexity and parameter count, and are suitable for fast reasoning. Small models can make more precise decisions based on the actual features of the input data (such as text, images, sounds, etc.) to select the best adapter model.
[0049] By combining the advantages of static rules and small machine learning models, the system can rely on fast static rules to make choices in most cases, thereby improving the speed of task processing; in complex or dynamic task situations, it can switch to small machine learning models for more accurate inference task analysis. This combination allows the system to respond quickly when the amount of inference tasks is large, while also having the ability to handle complex tasks.
[0050] The input of the inference task is analyzed by the intelligent router module. The intelligent router first quickly identifies the type and basic characteristics of the task through static rules, and selects the adapter model (i.e., adapter) based on the static rules or a small machine learning model, and distributes it to the inference node loaded with the corresponding adapter model according to the requirements of the task. The adapter model is only loaded when the task requires it, and the most suitable adapter is selected according to the characteristics of the task.
[0051] The intelligent router sends the inference task to the corresponding inference node according to the selected adapter model. Through this mechanism, the task can be quickly located on the most suitable computing resource, avoiding unnecessary switching and waiting between multiple nodes, and significantly improving the response speed of the system.
[0052] In traditional systems, loading adapter models is usually an expensive operation, especially on low-memory devices. Based on the historical data and preset rules of the task, commonly used adapter models are loaded in advance, thus avoiding the need for cold loading every time an inference task arrives. The pre-loaded adapter model is ready when the inference task arrives, greatly reducing cold start latency and improving inference speed.
[0053] Further, the loading of the adapter model into the corresponding inference node includes: the intelligent router loading the adapter into the corresponding inference node in advance based on historical data and rules before the inference task arrives.
[0054] Cold loading means that when an inference task arrives, the system needs to load the adapter model from the storage medium into the video memory or computing resources. This process usually produces a long delay, especially on devices with limited video memory or tight computing resources, resulting in a slow system response. In traditional systems, whenever an inference task arrives, the adapter model often needs to be loaded from an external storage device, which increases the waiting time of the task and affects the real-time performance of the system, especially when the task volume is large or the inference node resources are insufficient.
[0055] Through the optimization mechanism of the smart router module, commonly used adapter models are loaded into the inference node in advance, reducing cold loading delays. Specifically, the smart router will pre-determine which adapter models may be used in future inference tasks based on historical data and task characteristics, and load these adapter models into the inference node before the inference task arrives. This preloading mechanism is based on the historical data of the task (such as past task types, input data characteristics, etc.) and rules (such as the frequency and category of inference tasks), ensuring that the inference node already has the adapter model required to execute the task when the actual task arrives, thereby avoiding cold start delays.
[0056] The smart router analyzes the data of historical inference tasks, including task type, input features, frequency, and other information. This data helps the router predict which adapter models may be used in future inference tasks. For example, if a certain type of task frequently appears in past inferences, the router can infer that the task may occur again in the future and load the relevant adapter model in advance.
[0057] Based on the analysis of historical data, the router schedules using preset rules (such as task frequency, common categories, etc.) and loads the most likely adapter model to the inference node in advance. In this way, when the inference node processes the inference task, it can directly use the loaded adapter model, avoiding the need for cold loading every time a task arrives.
[0058] Before the inference task arrives, the smart router loads the adapter model required for the task to the specified inference node in advance. This process not only ensures that the inference node is ready when it receives the task, but also avoids overloading and repeated loading of resources through optimized loading order and strategy. Once the task arrives, the smart router is able to quickly distribute the task to the inference node loaded with the adapter model. Since the relevant adapter model has been loaded in advance, the inference node can execute the task immediately without additional delay.
[0059] Furthermore, the system also includes: a dynamic load balancing module, which is used to adjust the loading strategy of the adapter model and allocate computing power resources of the inference node according to the real-time load situation of the inference task.
[0060] The dynamic load balancing module monitors the inference tasks in real time and analyzes the computing load and memory usage of the current inference node. When multiple inference tasks are performed concurrently, each task may have different requirements for the system's computing resources. The load balancing module continuously evaluates the load status of each node to ensure that the allocation of tasks does not cause overload of a single node.
[0061] By analyzing the complexity of the task, the amount of input data, and the demand for computing power, we can determine whether the task is "heavy" or "light". For example, some tasks may require high computing power, while other tasks may only require light computing resources. The system will dynamically adjust the resource allocation strategy to adapt to the needs of different tasks and ensure system load balancing.
[0062] Based on the load of the inference task, the dynamic load balancing module decides whether additional adapter models need to be loaded and when to load them. For example, when the inference task load is high, the module may prioritize loading the adapter model that is most likely to be used to ensure that computing resources are fully utilized. The system dynamically loads adapter models and unloads unnecessary adapter models based on the current load, avoiding the system from wasting graphics memory resources by loading unnecessary adapter models, ensuring that the computing power of the inference node is focused on the current task, and improving the efficiency of computing resource usage.
[0063] The dynamic load balancing module is also responsible for intelligently scheduling the computing resources of the inference nodes according to the computing requirements of the tasks. The system monitors the processing power of each node to ensure that nodes with lighter loads are not overloaded, while nodes with heavier loads receive more computing power support. Through a reasonable load balancing strategy, the module ensures that multiple inference tasks are evenly distributed on each node to avoid affecting the task processing efficiency due to resource constraints on a certain node. The load balancing strategy can dynamically adjust the distribution of tasks according to the real-time load of the node (such as processor usage, memory usage, etc.) to maximize the utilization of system resources.
[0064] It is not difficult to find that the embodiments of the present application are widely used in deep learning inference of low-memory devices, such as smart phones, IoT devices, edge computing and other scenarios. In these scenarios, the video memory resources are limited and the real-time response requirements are high. The design of multi-expert model adapters and intelligent routers meets the low-resource, high-response inference requirements.
[0065] It is worth mentioning that all modules involved in this embodiment are logic modules. In practical applications, a logic unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, in order to highlight the innovative part of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application, but this does not mean that there are no other units in this embodiment.
[0066] Third embodiment
[0067] The third embodiment of the present application relates to an intelligent inference method, which is applied to the system in the above embodiment, such as Figure 2 As shown, the method includes:
[0068] S101, receiving and analyzing inference tasks through an intelligent router;
[0069] S102, analyzing the inference task according to preset rules or a lightweight machine learning model, and selecting a corresponding adapter model;
[0070] S103, loading the base model according to the inference task, and performing an inference process of shared parameters;
[0071] S104, selecting a corresponding adapter model according to the inference task, loading the adapter model into the inference node, and executing the inference task.
[0072] It is not difficult to find that this embodiment is a method embodiment corresponding to the first embodiment, and this embodiment can be implemented in conjunction with the first embodiment. The relevant technical details mentioned in the first embodiment are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied in the first embodiment.
[0073] The step division of the above methods is only for the purpose of clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent; adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the scope of protection of this patent.
[0074] In addition, some embodiments of the present application also provide an electronic device. The electronic device may be a digital computer in various forms, such as a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, etc. The electronic device may also be a mobile device in various forms, such as a personal digital processing, a cellular phone, a smart phone, a wearable device, and other similar computing devices.
[0075] The electronic device includes: one or more processors; and a memory storing computer program instructions, wherein when the computer program instructions are executed, the processor executes the steps of the method provided in any one or more of the above embodiments. Figure 3 An exemplary structural diagram of the electronic device is disclosed. Figure 3 As shown, the electronic device includes: one or more processors 1101, memory 1102, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Among them, the components shown in this article, their connections and relationships, and their functions are only used as examples, and are not intended to limit the implementation of the present application described and / or required herein.
[0076] The electronic device may further include: an input device 1103 and an output device 1104. The processor 1101, the memory 1102, the input device 1103 and the output device 1104 may be connected via a bus or other means. Figure 3 The example of connecting through bus is taken in the following.
[0077] The input device 1103 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 1104 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.
[0078] To provide interaction with a user, the electronic device may be a computer. The computer has: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball), through which the user can provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0079] In the embodiments of the present application, a computer program / instruction is stored on a computer-readable medium, and when the computer program / instruction is executed by a processor, the steps of the method provided by any one or more of the above embodiments are implemented. The computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently without being assembled into the device. The above computer-readable medium carries one or more computer-readable instructions.
[0080] The memory 1102 can be used as a non-transient computer-readable storage medium, which can be used to store non-transient software programs, non-transient computer executable programs and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 1102, so as to implement the program instructions / modules corresponding to the method provided by any one or more embodiments in the embodiments of the present application.
[0081] The memory 1102 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1102 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 1102 may optionally include a memory remotely arranged relative to the processor 1101, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0082] It should be noted that the computer-readable medium described in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.
[0083] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.
[0084] Computer program code for performing the operations of the present application may be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0085] In the above-described embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. For example, it can be implemented by using an application specific integrated circuit (ASIC), a general-purpose computer or any other similar hardware device. In certain embodiments, the software program of the present application can be executed by a processor to implement the above steps or functions. Similarly, the software program of the present application (including relevant data structures) can be stored in a computer-readable recording medium, for example, a RAM memory, a magnetic or optical drive or a floppy disk and similar devices. In addition, some steps or functions of the present application can be implemented by hardware, for example, as a circuit that cooperates with a processor to perform each step or function.
[0086] The computer program product provided in the embodiment of the present application includes one or more computer programs / instructions, and when the computer program / instructions are executed by the processor, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.
[0087] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0088] The scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes that fall within the meaning and scope of the equivalent elements of the claims are included in the present application. Any figure mark in the claims should not be regarded as limiting the claims involved. In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the device claim may also be implemented by one unit or device through software or hardware. The words "first", "second", etc. are only used to distinguish the description, and do not indicate any particular order, nor can they be understood as indicating or implying relative importance.
[0089] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily mention changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims, and the above embodiments should be regarded as exemplary and non-restrictive.
Claims
1. An intelligent inference system, characterized in that: The system comprises: The base model is used to carry out the inference process of shared parameters; A plurality of adapter models, each of the adapter models being used for performing an inference process of an inference task; The base model is deployed separately from the adapter model, and supports parallel inference of multiple adapter models on one base model; The intelligent router is used to receive and analyze the inference task, select the corresponding adapter model according to a preset rule, and load the adapter model into the corresponding inference node.
2. The system according to claim 1, characterized in that The adapter model adopts the LoRA model technology and reduces the number of parameters of the adapter model by low-rank decomposition.
3. The system according to claim 2, characterized in that The adapter model does not load video memory resources when it is deployed, but loads video memory resources when performing an inference task.
4. The system according to claim 3, characterized in that The system comprises: When performing the inference task, the base model is loaded to infer the shared parameters, and the corresponding adapter model is loaded according to the specific requirements of the inference task.
5. The system according to any one of claims 1 to 4, characterized in that: The intelligent router adopts a selection mechanism based on a combination of static rules and small models to analyze the inference task and select the corresponding adapter model.
6. The system according to claim 5, characterized in that The loading of the adapter model into the corresponding inference node comprises: The intelligent router loads the adapter to the corresponding inference node in advance based on historical data and rules before the inference task arrives.
7. The system according to claim 1, characterized in that The system further comprises: The dynamic load balancing module is used to adjust the loading strategy of the adapter model and allocate the computing power resources of the inference node according to the real-time load situation of the inference task.
8. An intelligent inference method, applied to the system as claimed in any one of claims 1 to 7, characterized in that: The method comprises: Receive and analyze inference tasks through intelligent routers; Analyze the inference task according to preset rules or a lightweight machine learning model, and select a corresponding adapter model; The base model is loaded according to the inference task to perform the inference process of the shared parameters; the corresponding adapter model is selected according to the inference task, the adapter model is loaded into the inference node, and the inference task is executed.
9. An electronic device, characterized in that: The electronic device comprises: one or more processors; and A memory storing computer program instructions which, when executed, cause the processor to perform the steps of the method of claim 8.
10. A computer readable medium having a computer program / instructions stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method of claim 8 are implemented.
Citation Information
Cited By
Large language model reasoning method and device, equipment and storage medium
CN121189489A
Large language model reasoning method and device, equipment and storage medium
CN121189489B
