Reasoning optimization method, optimization device, equipment, storage medium and program product

Abstract

The invention provides a reasoning optimization method, an optimization device, equipment, a storage medium and a program product, and relates to the technical field of machine learning. The model-as-a-service reasoning optimization method comprises the steps that a first request set and a second request set are generated in response to received calling requests of calling multiple fine tuning large model reasoning by multiple users, the first request set corresponds to reasoning of a basic large model, and the second request set corresponds to reasoning of the fine tuning large model; reasoning the first request set based on the shared freezing parameter to obtain a first batch processing result, and reasoning the second request set based on the non-shared parameter to obtain a second batch processing result; and combining the first batch processing result and the second batch processing result to obtain a target result, and feeding back the target result to the plurality of users. According to the technical scheme, different tenants share the frozen parameters to perform batch processing of the reasoning tasks, the GPU memory resource usage amount of the multi-tenant large model is reduced, and the reasoning cost is reduced.

Description

Technical Field

[0001] The present disclosure relates to the technical field of machine learning, and in particular to a model-as-a-service inference optimization method, a model-as-a-service inference optimization device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the development of big model technology, application scenarios have gradually evolved from general scenarios to industry customized scenarios. In order to save the cost of fine-tuning and using customized big models for users in different industries, more and more cloud computing vendors have begun to provide MaaS (Model as a Service) services, among which big model reasoning is a key business of MaaS services. However, due to the huge number of parameters of big models, their reasoning latency is long, so accelerators such as GPU (Graphics Processing Unit) can be used for processing. However, resources such as GPU memory are limited. In cloud computing multi-tenant scenarios where multiple tenants share the same cloud computing vendor resources, it is necessary to maintain a complete customized big model for each user in the GPU system for reasoning, resulting in large GPU memory overhead and high costs.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The purpose of the present disclosure is to provide a model-as-a-service reasoning optimization method, a model-as-a-service reasoning optimization device, an electronic device, a computer-readable storage medium and a computer program product, which at least to some extent overcome the problem of missing scene information of surveillance cameras in related technologies.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0006] According to one aspect of the present disclosure, there is provided a method for inference optimization of a model as a service, comprising: in response to call requests received from multiple users for inference of multiple fine-tuned large models, preprocessing the call requests to generate a first request set and a second request set, wherein the first request set corresponds to the inference of a basic large model, and the second request set corresponds to the inference of the fine-tuned large model; reasoning the first request set based on the shared frozen parameters of the basic large model to obtain a first batch of processing results, and reasoning the second request set based on the non-shared parameters of the fine-tuned large model to obtain a second batch of processing results; merging the first batch of processing results and the second batch of processing results to obtain a target result, and feeding back the target result to the multiple users.

[0007] In one embodiment of the present disclosure, in response to receiving call requests from multiple users to call multiple fine-tuned large model inferences, before preprocessing the call requests, it also includes: receiving information triples of fine-tuned large models submitted by users, registering and saving the information triples, the information triples including a base large model identifier, a corresponding fine-tuned large model identifier, and non-shared parameters of the fine-tuned large model.

[0008] In one embodiment of the present disclosure, the method further includes: storing the non-shared parameters of the fine-tuned large model in a large model parameter repository, wherein the large model parameter repository also includes the shared frozen parameters identified by multiple basic large models.

[0009] In one embodiment of the present disclosure, in response to receiving call requests from multiple users to call multiple fine-tuned large model inferences, the call requests are preprocessed to generate a first request set and a second request set, including: in response to the call request, extracting the fine-tuned large model identifier in the call request; querying the base large model identifier that matches the fine-tuned large model identifier; generating a shared parameter batch based on the first call request with the same base large model identifier, and generating a non-shared parameter batch based on the second call request with the same fine-tuned large model identifier; generating the first request set based on multiple shared parameter batches corresponding to multiple base large model identifiers, and generating the second request set based on multiple non-shared parameter batches corresponding to multiple fine-tuned large model identifiers.

[0010] In one embodiment of the present disclosure, the call request also includes text data input by a user, and a shared parameter batch is generated based on the call request with the same basic large model identifier, and a non-shared parameter batch is generated based on the call request with the same fine-tuning large model identifier, including: for the first call request with the same basic large model identifier, performing a dimension consistency operation on the text data in the first call request to obtain the shared parameter batch; for the second call request with the same fine-tuning large model identifier, performing a dimension consistency operation on the text data in the second call request to obtain the non-shared parameter batch.

[0011] In one embodiment of the present disclosure, before inferring the first request set based on the shared frozen parameters of the basic large model to obtain a first batch of processing results, and inferring the second request set based on the non-shared parameters of the fine-tuned large model to obtain a second batch of processing results, it also includes: scheduling the first request set to a shared parameter batch processing module to infer the first request set, and scheduling the second request set to a non-shared parameter batch processing module to infer the second request set.

[0012] In one embodiment of the present disclosure, the first request set is inferred based on the shared frozen parameters of the basic large model to obtain a first batch of processing results, and the second request set is inferred based on the non-shared parameters of the fine-tuning large model to obtain a second batch of processing results, including: reading and loading the shared frozen parameters corresponding to the basic large model identifier and the non-shared parameters corresponding to the fine-tuning large model identifier from the large model parameter repository; inputting the first request set into a processing framework loaded with the shared frozen parameters for batch processing operations to obtain the first batch of processing results; inputting the second request set into a processing framework loaded with the non-shared parameters for batch processing operations to obtain the second batch of processing results.

[0013] In one embodiment of the present disclosure, merging the first batch of processing results and the second batch of processing results to obtain a target result includes: determining a first weight of the shared freezing parameter, and a second weight of the non-shared parameter; assigning the first weight to the first batch of processing results, and assigning the second weight to the second batch of processing results, so as to merge the first batch of processing results and the second batch of processing results to obtain the target result.

[0014] According to another aspect of the present disclosure, there is provided a model-as-a-service inference optimization method, which is applied to a tenant side, and includes: sending a call request for invoking fine-tuning large model inference to a server side, the call request including a fine-tuning large model identifier; receiving a target result generated by the server side based on the call request, wherein the server side determines a matching basic large model identifier based on the fine-tuning large model identifier, and merges a first batch of inference results generated based on the basic large model identifier and a second batch of processing results generated based on the fine-tuning large model identifier to obtain the target result.

[0015] In one embodiment of the present disclosure, before sending a call request for invoking fine-tuning large model inference to a server, it also includes: submitting an information triplet of the fine-tuning large model to the server, registering and saving the information triplet, the information triplet including a basic large model identifier, a corresponding fine-tuning large model identifier, and non-shared parameters of the fine-tuning large model, wherein the first batch of processing results are obtained by reasoning based on the shared frozen parameters, and the second batch of processing results are obtained by reasoning based on the non-shared parameters.

[0016] According to another aspect of the present disclosure, there is provided an inference optimization device of a model as a service, comprising: a preprocessing module, for responding to call requests received from multiple users for inferencing multiple fine-tuned large models, preprocessing the call requests, generating a first request set and a second request set, wherein the first request set corresponds to the inference of the basic large model, and the second request set corresponds to the inference of the fine-tuned large model; an inference module, for inferring the first request set based on the shared frozen parameters of the basic large model to obtain a first batch of processing results, and inferring the second request set based on the non-shared parameters of the fine-tuned large model to obtain a second batch of processing results; a merging module, for merging the first batch of processing results and the second batch of processing results to obtain a target result, and feeding the target result back to the multiple users.

[0017] According to another aspect of the present disclosure, there is provided an inference optimization device of a model as a service, comprising: a sending module, used to send a call request for invoking fine-tuning large model inference to a server, the call request comprising a fine-tuning large model identifier; a receiving module, used to receive a target result generated by the server based on the call request, wherein the server determines a matching basic large model identifier based on the fine-tuning large model identifier, and merges a first batch of inference results generated based on the basic large model identifier and a second batch of processing results generated based on the fine-tuning large model identifier to obtain the target result.

[0018] According to another aspect of the present disclosure, there is provided an electronic device, comprising: a processor; and a memory for storing executable instructions of the processor; the processor is configured to execute the above-mentioned model-as-a-service inference optimization method by executing the executable instructions.

[0019] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned model-as-a-service reasoning optimization method is implemented.

[0020] According to another aspect of the present disclosure, a computer program product is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above-mentioned model-as-a-service reasoning optimization method is implemented.

[0021] The inference optimization scheme of Model as a Service provided by the embodiment of the present disclosure, when receiving call requests from multiple users, first enters the preprocessing step, and uses pre-set algorithms and rules to conduct in-depth analysis of each request to determine whether it is suitable for processing by the basic large model or the fine-tuning large model, and classifies them into a first request set and a second request set respectively, uses shared frozen parameters to perform inference operations for the first request set, reads and applies non-shared parameters for the second request set, inputs the requests into the fine-tuning large model for inference, and finally merges the first batch of processing results and the second batch of processing results to form a complete target result, and reduces the GPU memory resource usage of the multi-tenant large model and the inference cost by enabling different tenants to share frozen parameters for batch processing of inference tasks, and reduces the inference cost, and combines non-shared parameters to meet both general and specific needs to adapt to a variety of different application scenarios and user requests, thereby facilitating the reasonable allocation of resources and model utilization, so as to reduce the operation and maintenance costs of MaaS services.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0024] Figure 1 A schematic diagram showing a model-as-a-service inference optimization system in an embodiment of the present disclosure;

[0025] Figure 2A schematic flow chart showing a method for inference optimization of a model as a service in an embodiment of the present disclosure is shown;

[0026] Figure 3 A schematic flow chart showing another inference optimization method of a model as a service in an embodiment of the present disclosure;

[0027] Figure 4 A schematic flow chart showing another inference optimization method of a model as a service in an embodiment of the present disclosure is shown;

[0028] Figure 5 A schematic diagram showing another model-as-a-service inference optimization system in an embodiment of the present disclosure;

[0029] Figure 6 A schematic flow chart showing another inference optimization method of a model as a service in an embodiment of the present disclosure;

[0030] Figure 7 A schematic diagram of a model-as-a-service inference optimization device in an embodiment of the present disclosure is shown;

[0031] Figure 8 A schematic diagram of another model-as-a-service inference optimization device in an embodiment of the present disclosure is shown;

[0032] Fig. 9 A structural block diagram of an electronic device in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0033] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0034] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0035] Cloud computing vendors have begun to provide MaaS services, allowing users to use pre-trained large models in the cloud, train and fine-tune large models through the tools and interfaces provided by the cloud, and perform elastic large model inference, which can meet the customized application needs of large models in a low-cost manner. Large model inference is a key business of MaaS services. Due to the huge number of parameters in large models, the inference latency is long, and accelerators such as GPUs are often used for processing. However, resources such as GPU memory are limited. In cloud computing multi-tenant scenarios, traditional methods often maintain a complete customized large model for each user in the GPU system for inference, which has large GPU memory overhead and high costs.

[0036] The low-rank adaptive large model fine-tuning method (LoRA) is the mainstream large model fine-tuning solution in the industry. When fine-tuning a large model, it freezes some pre-trained model weights, thereby greatly reducing the number of training parameters for downstream tasks. The LoRA method allows large models fine-tuned by multiple tenants to share some frozen pre-trained large model weights. During inference, there is no need to maintain a complete large model for each user. Only non-shared parameters need to be maintained. Different users share some frozen parameters, which is expected to reduce GPU memory resource consumption and reduce inference costs.

[0037] In order to solve the problems of large GPU memory overhead and high cost in cloud computing multi-tenant MaaS large model inference service scenarios, the present invention proposes an inference optimization method for cloud multi-tenant large model as a service. Based on the LoRA large model fine-tuning method, multi-tenants can share the characteristics of partially frozen pre-trained large model weights. When inferring large models, different tenants share frozen parameters for batch processing of inference tasks, thereby reducing the GPU memory resource usage of multi-tenant large models and reducing inference costs.

[0038] Figure 1 1 is a schematic diagram of a computer system provided by an exemplary embodiment of the present application. The system includes: a plurality of tenant terminals 120 and a server terminal 140.

[0039] The tenant end 120 can be a mobile terminal such as a mobile phone, a game console, a tablet computer, an e-book reader, smart glasses, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a smart home device, an AR (Augmented Reality) device, a VR (Virtual Reality) device, or the tenant end 120 can also be a personal computer (PC), such as a laptop computer and a desktop computer.

[0040] Among them, the tenant end 120 may be installed with an application for providing inference optimization of model as a service.

[0041] The tenant end 120 is connected to the server end 140 via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0042] The server 140 is a server, or is composed of several servers, or is a virtualization platform, or is a cloud computing service center. The server 140 is used to provide background services for applications that provide inference optimization of models as services. Optionally, the server 140 undertakes the main computing work and the tenant 120 undertakes the secondary computing work; or, the server 140 undertakes the secondary computing work and the tenant 120 undertakes the main computing work; or, the tenant 120 and the server 140 adopt a distributed computing architecture for collaborative computing.

[0043] In some optional embodiments, the server 140 is used to store the inference optimization program information of the model as a service.

[0044] Optionally, the logistics user terminals of the application programs installed in different tenant terminals 120 are the same, or the logistics user terminals of the application programs installed on the two tenant terminals 120 are logistics user terminals of the same type of application programs on different control system platforms. Based on the different terminal platforms, the specific forms of the logistics user terminals of the application programs may also be different, for example, the logistics user terminals of the application programs may be mobile logistics user terminals, PC logistics user terminals, or global wide area network (World Wide Web, Web) logistics user terminals, etc.

[0045] Those skilled in the art will appreciate that the number of the tenant terminals 120 may be more or less. For example, the number of the tenant terminals 120 may be only one, or the number of the tenant terminals may be dozens or hundreds, or more. The embodiment of the present application does not limit the number of terminals and the device type.

[0046] Optionally, the system may also include a management device ( Figure 1 (not shown), the management device is connected to the server 140 via a communication network. Optionally, the communication network is a wired network or a wireless network.

[0047] Optionally, the above-mentioned wireless network or wired network uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, the data exchanged through the network is represented by technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above-mentioned data communication technologies.

[0048] like Figure 2 As shown, according to an embodiment of the present disclosure, a method for optimizing model-as-a-service inference includes:

[0049] Step S202, in response to receiving call requests from multiple users to call for inference of multiple fine-tuned large models, preprocess the call requests to generate a first request set and a second request set, the first request set corresponding to the inference of the basic large model, and the second request set corresponding to the inference of the fine-tuned large model.

[0050] Among them, the fine-tuned large model is a model that is further optimized and adjusted for specific tasks or data sets based on the basic large model. Through fine-tuning, the model can better adapt to the needs of specific fields or problems and improve the performance and accuracy of the model.

[0051] Preprocessing is the preliminary processing and classification of the user's call requests after receiving them. Its purpose is to organize and divide the complex and diverse call requests so that they can be processed more efficiently and targeted later. Preprocessing may include analyzing the content of the request, extracting key features, and classifying according to certain rules.

[0052] The first request set is a combination of requests that are classified as corresponding to the reasoning of the basic large model after preprocessing, and the second request set is a set of requests that correspond to the reasoning of the fine-tuned large model after preprocessing, and these requests may have specific characteristics or requirements.

[0053] Step S204, inferring the first request set based on the shared frozen parameters of the basic large model to obtain the first batch of processing results, and inferring the second request set based on the non-shared parameters of the fine-tuned large model to obtain the second batch of processing results.

[0054] Among them, shared frozen parameters are parameters that can be used by multiple requests during the inference process and remain unchanged. These parameters are usually the parts of the basic large model that have universal applicability and stability. Non-shared parameters are parameters related to fine-tuning the large model and are specific to certain requests or tasks.

[0055] Step S206, merging the first batch of processing results and the second batch of processing results to obtain a target result, and feeding back the target result to multiple users.

[0056] Among them, when merging the first batch of processing results and the second batch of processing results to obtain the target results, factors such as the weight, priority, and relevance of the results need to be considered to ensure that the merged results are accurate and reasonable.

[0057] In this embodiment, when calling requests from multiple users are received, the preprocessing step is first entered. By using pre-set algorithms and rules, each request is deeply analyzed to determine whether it is suitable for processing by the basic large model or the fine-tuning large model, and the requests are respectively classified into a first request set and a second request set. For the first request set, shared frozen parameters are used to perform reasoning operations. For the second request set, non-shared parameters are read and applied, and the requests are input into the fine-tuning large model for reasoning. Finally, the first batch of processing results and the second batch of processing results are merged to form a complete target result. By enabling different tenants to share frozen parameters for batch processing of reasoning tasks, the GPU memory resource usage of the multi-tenant large model is reduced, and the reasoning cost is reduced. Combined with non-shared parameters, general and specific requirements can be met at the same time to adapt to a variety of different application scenarios and user requests, which is conducive to the reasonable allocation of resources and model utilization, so as to reduce the operation and maintenance costs of MaaS services.

[0058] In one embodiment of the present disclosure, in response to receiving multiple users' call requests for invoking multiple fine-tuned large model inferences, before preprocessing the call requests, the method further includes:

[0059] Receive the information triplet of the fine-tuning large model submitted by the user, register and save the information triplet, the information triplet includes the basic large model identifier, the corresponding fine-tuning large model identifier and the non-shared parameters of the fine-tuning large model.

[0060] Among them, the fine-tuned large model can be a model obtained by fine-tuning the basic large model using a low-rank adaptive large model fine-tuning method.

[0061] An information triplet is a data structure used to describe the key information of a fine-tuned large model. It contains three important elements and is used to express the core information related to the fine-tuned large model.

[0062] Registration and preservation refers to formally recording and storing the data in the information triplet in a specific database or storage structure of the system so that it can be easily and accurately queried, used and managed later.

[0063] The base big model identifier BaseLLMID is an identifier used to uniquely identify the original, unfine-tuned big model. The corresponding fine-tuned big model identifier FineLLMID is a unique identifier specially set for the big model that has been specifically adjusted and optimized. It is interrelated with the base big model identifier and is used to clearly distinguish and identify different fine-tuned versions to ensure that the required specific fine-tuned model can be accurately found when used and called. The non-shared parameters of the fine-tuned big model FineLLMPara are parameters specific to the fine-tuned model that are generated during the fine-tuning process of the base big model.

[0064] The base large model identifier BaseLLMID and the fine-tuning large model identifier FineLLMID are used to classify the call requests and generate a first request set and a second request set respectively.

[0065] In this embodiment, when a user submits an information triplet of a fine-tuned large model, the parsed information triplet is stored in a specially designed database or storage area to complete the registration and save operation. Through the registration and save of the information triplet, systematic management of the fine-tuned large model is achieved, which facilitates the organization and search of a large amount of model information. In addition, clear basic large model identification and fine-tuning large model identification can ensure that the appropriate model is accurately called when processing user requests, thereby improving service quality.

[0066] In one embodiment of the present disclosure, it also includes: storing the non-shared parameters of the fine-tuned large model in a large model parameter repository, and the large model parameter repository also includes shared frozen parameters identified by multiple basic large models.

[0067] Among them, the large model parameter repository is a centralized data storage facility specifically used to store various large model-related parameters, which can support efficient reading, writing and management operations.

[0068] The shared frozen parameters identified by multiple basic large models refer to parameters corresponding to different types of basic large models, which are fixed under specific conditions and can be used by multiple related models or tasks.

[0069] In this embodiment, the shared frozen parameters identified by various basic large models have been pre-stored in the large model parameter repository. When the large model needs to be used for reasoning or calculation, the corresponding parameters are read from the repository according to specific needs. If the reasoning is based on the basic large model, the corresponding shared frozen parameters will be obtained. For the fine-tuned large model, the shared frozen parameters and its unique non-shared parameters will be obtained at the same time. The existence of shared frozen parameters enables the core functions of the basic large model to be widely used, and at the same time provides a stable foundation for the new fine-tuning model, which helps to quickly deploy and expand the application of the model in different tasks and scenarios.

[0070] like Figure 3 As shown, in one embodiment of the present disclosure, in response to receiving multiple user call requests for calling multiple fine-tuned large model inferences, the call requests are preprocessed to generate a first request set and a second request set, including:

[0071] Step S302: In response to the call request, extract the fine-tuning large model identifier in the call request.

[0072] The calling request includes the called fine-tuning large model identifier FineLLMID and the user input text data InputText. At the same time, multiple users may call multiple fine-tuning large model inferences. Each request is represented by Query 0, Query 1, ..., Query n, and each Query can include FineLLMID and InputText.

[0073] In addition, the call request also includes a user ID, that is, an ID of the tenant, so that the target result can be fed back to the corresponding user.

[0074] Step S304, querying the basic large model identifier that matches the fine-tuning large model identifier.

[0075] Matching refers to searching for a basic large model identifier that corresponds to or is associated with a given fine-tuned large model identifier in existing data or information.

[0076] Step S306, generating a shared parameter batch based on a first call request with the same basic large model identifier, and generating a non-shared parameter batch based on a second call request with the same fine-tuning large model identifier.

[0077] Among them, the call request with the same basic model identifier is the first call request, and the shared parameter batch is a group of data formed by combining the first call requests. The parameters used by these requests are shared, which means that they can share some of the same model parameters. This part of shared parameters is usually a relatively stable and common part of the basic model, which is suitable for multiple requests with the same basic model identifier, and can improve the reuse rate and calculation efficiency of parameters.

[0078] Among them, a base large model identifier BaseLLMID corresponds to a shared parameter batch, and a fine-tuning large model identifier FineLLMID corresponds to a non-shared parameter batch.

[0079] The call request with the same fine-tuned large model identifier is the second call request, and the non-shared parameter batch is generated based on the second call request. Unlike the shared parameter batch, this part of the parameters is specific to each fine-tuned large model and is not shared with other requests. They reflect the personalized and specific parameters generated after fine-tuning the basic large model, which are used to meet the unique needs of different fine-tuned large models.

[0080] Step S308: Generate a first request set based on multiple shared parameter batches corresponding to multiple basic large model identifiers, and generate a second request set based on multiple non-shared parameter batches corresponding to multiple fine-tuning large model identifiers.

[0081] In this embodiment, the request is parsed and extracted to obtain the fine-tuning large model identifier contained therein, and the basic large model identifier associated with the extracted fine-tuning large model identifier is searched based on an index, a mapping table or other fast-search data structure. According to the sameness of the basic large model identifiers, the related first call requests are combined together to form a shared parameter batch. At the same time, according to the sameness of the fine-tuning large model identifiers, the corresponding second call requests are combined into non-shared parameter batches to achieve classification and grouping of requests, ensuring that requests with the same identifier are classified into the same batch, and further integrating multiple shared parameter batches corresponding to multiple basic large model identifiers into a first request set, and integrating multiple non-shared parameter batches corresponding to multiple fine-tuning large model identifiers into a second request set. By grouping the requests into shared parameter batches and non-shared parameter batches, repeated parameter calculation and loading can be reduced, and the processing speed of requests can be accelerated.

[0082] In one embodiment of the present disclosure, the call request further includes text data input by the user, and the call request with the same basic large model identifier generates a shared parameter batch, and the call request with the same fine-tuned large model identifier generates a non-shared parameter batch, including:

[0083] For the first call request with the same basic large model identifier, a dimension consistency operation is performed on the text data in the first call request to obtain a shared parameter batch.

[0084] For a second call request with the same fine-tuning large model identifier, a dimension consistency operation is performed on the text data in the second call request to obtain a non-shared parameter batch.

[0085] Among them, dimensional consistency operation refers to the processing of text data, including adjustment and unification of the length, number of features, vector dimension and other aspects of the text data. For example, padding operation is performed on different user text input lengths in the same batch to make their dimensions the same so that they can be batch processed by the batch inference engine.

[0086] In this embodiment, by analyzing the differences in the text data on the relevant dimensions. If the text length is inconsistent, a fixed length may be selected as a standard, and the shorter texts may be padded (for example, filled with zero values ​​or other specific markers) to adjust the features or dimensions of the text data in the same batch to be consistent. The unified dimensions enable the model to process data in batches, avoiding the additional processing and computing overhead required due to inconsistent dimensions, thereby improving computing efficiency.

[0087] In one embodiment of the present disclosure, before inferring the first request set based on the shared frozen parameters of the basic large model to obtain the first batch of processing results, and inferring the second request set based on the non-shared parameters of the fine-tuned large model to obtain the second batch of processing results, the method further includes:

[0088] The first request set is scheduled to a shared parameter batch processing module to perform reasoning on the first request set, and the second request set is scheduled to a non-shared parameter batch processing module to perform reasoning on the second request set.

[0089] Here, “scheduling” refers to the process of respectively allocating and guiding the first request set and the second request set to corresponding processing modules (ie, the shared parameter batch processing module and the non-shared parameter batch processing module) according to certain rules and strategies.

[0090] In this embodiment, when the first request set and the second request set are ready for inference processing, scheduling decisions are made according to preset rules and module status information acquired in real time. Through reasonable scheduling, the resources of the two processing modules can be fully utilized.

[0091] In one embodiment of the present disclosure, reasoning on a first request set based on shared frozen parameters of a basic large model to obtain a first batch of processing results, and reasoning on a second request set based on non-shared parameters of a fine-tuned large model to obtain a second batch of processing results, including:

[0092] The shared frozen parameters corresponding to the base large model ID and the non-shared parameters corresponding to the fine-tuned large model ID are read and loaded from the large model parameter repository.

[0093] The first request set is input into a processing framework loaded with shared freezing parameters for batch processing to obtain a first batch of processing results.

[0094] The processing framework loaded with shared frozen parameters refers to the framework that has read and loaded the shared frozen parameters corresponding to the basic large model identifier from the large model parameter repository. These parameters are in a ready state in the framework, waiting for input data.

[0095] The second request set is input into the processing framework loaded with non-shared parameters for batch processing to obtain a second batch processing result.

[0096] Among them, the processing framework loaded with non-shared parameters is specifically used to process data related to fine-tuning large models.

[0097] In this embodiment, the first request set is input into a processing framework loaded with shared frozen parameters, and the second request set is input into a processing framework loaded with non-shared parameters, and the first batch of processing results and the second batch of processing results are obtained respectively. Through batch processing, multiple requests can be processed at one time, reducing repeated parameter loading and initialization operations, which is beneficial to improving the processing speed.

[0098] In one embodiment of the present disclosure, merging the first batch of processing results and the second batch of processing results to obtain the target result includes:

[0099] A first weight for the shared frozen parameter and a second weight for the unshared parameter are determined.

[0100] Among them, weight refers to a numerical value used to measure the influence of shared frozen parameters and non-shared parameters on the final processing results. The first weight is used to represent the relative importance of shared frozen parameters when merging processing results, and the second weight is used to represent the relative importance of non-shared parameters.

[0101] The first weight and the second weight may be determined according to the accuracy, reliability, and importance to a specific task of the shared frozen parameters and the non-shared parameters, or through analysis of historical data and model evaluation.

[0102] A first weight is assigned to the first batch of processing results, and a second weight is assigned to the second batch of processing results, so as to merge the first batch of processing results and the second batch of processing results to obtain a target result.

[0103] Among them, if the first weight is 0.6, the second weight is 0.4, the first batch of processing results is A, and the second batch of processing results is B, then the merged target result can be calculated as follows: target result = 0.6*A+0.4*B.

[0104] In this embodiment, after obtaining the first batch of processing results and the second batch of processing results, the two batches of processing results are weighted calculated using the predetermined first weight and second weight to obtain a large search target result. The clear weight allocation provides a certain degree of interpretability for the output of the model, which helps to understand the role of different parameters in the final result.

[0105] like Figure 4 As shown, according to another embodiment of the present disclosure, the inference optimization method of model as a service is applied to the tenant side, including:

[0106] Step S402: Send a call request for invoking fine-tuning large model inference to the server, where the call request includes a fine-tuning large model identifier.

[0107] The call request includes the fine-tuning large model identifier FineLLMID and the user input text data InputText, and also includes a user identifier, that is, an identifier of the tenant, so as to feed back the target result to the corresponding user.

[0108] Step S404, receiving the target result generated by the server based on the call request, wherein the server determines the matching basic large model identifier based on the fine-tuning large model identifier, and merges the first batch of inference results generated based on the basic large model identifier and the second batch of processing results generated based on the fine-tuning large model identifier to obtain the target result.

[0109] In this embodiment, the tenant sends a request to call the fine-tuning large model inference to the server. After receiving the call request, the server determines the corresponding basic large model identifier through an internal matching mechanism based on the fine-tuning large model identifier carried by the server, and generates the first batch of inference results based on the basic large model identifier and the second batch of processing results based on the fine-tuning large model identifier. The server merges the first batch of inference results and the second batch of processing results according to certain rules or algorithms to obtain the final target result, and feeds it back to the tenant. The server centrally manages the basic large model and various fine-tuning large models, which can meet the diverse needs of different users for using different fine-tuning large model inferences.

[0110] In one embodiment of the present disclosure, before sending a call request for invoking fine-tuning large model inference to the server, it also includes: submitting an information triplet of the fine-tuning large model to the server, registering and saving the information triplet, the information triplet includes a basic large model identifier, a corresponding fine-tuning large model identifier and non-shared parameters of the fine-tuning large model, wherein the first batch of processing results are obtained by reasoning based on the shared frozen parameters, and the second batch of processing results are obtained by reasoning based on the non-shared parameters.

[0111] like Figure 5 The inference system 500 for multi-tenant big model as a service on the cloud according to one embodiment of the present disclosure includes five key components: a multi-tenant big model management component 502, a multi-tenant request preprocessing component 504, a multi-tenant batch scheduling component 506, a multi-tenant batch inference component 508, and a multi-tenant big model parameter repository (including a big model non-shared parameter repository 5102 and a big model shared parameter repository 5104). The functions of each component are as follows:

[0112] Multi-tenant large model management component 502: includes a large model registration module 5022 and a large model information query module 5024, which are used for different tenants to register and fine-tune large model information and for the inference system to query large model information. After fine-tuning the large model based on the LoRA method, different users register the large model through this component for subsequent reasoning. During registration, the information triple of the fine-tuned large model (pre-trained basic large model identifier BaseLLMID, fine-tuned large model identifier FineLLMID, fine-tuned large model non-shared parameters FineLLMPara) is included (with Ba seLLMID represents) is submitted to the multi-tenant big model management component 502, and the multi-tenant big model management component 502 calls the big model registration module 5022 to save the above information, and saves the fine-tuning big model parameters FineLLMPara to the big model non-shared parameter repository 5102. After the registration is completed, other components of the inference system can query the multi-tenant big model related information through the big model information query module 5024 of the multi-tenant big model management component 502, and query BaseLLMID and FineLLMPara according to FineLLMID.

[0113] The multi-tenant request preprocessing component 504 includes a user request classification module 5042 and a user input preprocessing module 5044, which are used to classify and preprocess multi-tenant requests, classify the large model inference requests of multi-tenants according to the basic large model and the fine-tuning large model used by them, and obtain batches calculated using shared large model parameters and batches calculated using non-shared large model parameters. Among them, the user request classification module 5042 classifies user requests according to two classification methods. The first method classifies user requests with the same BaseLLMID into one category (this part can be calculated using the same shared large model parameters), and the second method classifies user requests with the same FineLLMID into one category (this part can be calculated using the same non-shared large model parameters). The user input preprocessing module 5044 processes user requests in the same category into batches of fixed size, and performs zero padding operations on different user input lengths in the same batch to make them have the same dimensions and can be batch processed by the batch inference engine;

[0114] The multi-tenant batch scheduling component 506 includes a shared parameter batch scheduling module 5062 and a non-shared parameter batch scheduling module 5064, which are respectively responsible for scheduling batches calculated using shared large model parameters to the shared parameter batch processing module for batch processing calculations, and scheduling batches calculated using non-shared large model parameters to the non-shared parameter batch processing module for batch processing calculations.

[0115] The multi-tenant batch reasoning component 508 includes a shared parameter batch processing module 5082, a non-shared parameter batch processing module 5084, and a batch processing result merging module 5086, which are responsible for executing batch calculations and merging and returning results. The shared parameter batch processing module 5082 processes batches calculated using shared large model parameters, the non-shared parameter batch processing module 5084 processes batches calculated using non-shared large model parameters, and the batch processing result merging module 5086 merges the results according to the LoRA frozen and non-frozen parameter calculation result merging method.

[0116] The multi-tenant large model parameter repository includes a large model non-shared parameter repository 5102 and a large model shared parameter repository 5104, wherein the large model non-shared parameter repository 5102 is used to store the non-shared parameters of the basic large model fine-tuned during the LoRA training process, and the large model shared parameter repository 5104 is used to store the shared parameters of the basic large model frozen during the LoRA fine-tuning training process.

[0117] like Figure 6 As shown, according to another embodiment of the present disclosure, a method for optimizing model-as-a-service inference includes:

[0118] Step S602: The user submits the information triplet of the registered fine-tuned large model to the large model management component.

[0119] The information triplet (BaseLLMID, FineLLMID, FineLLMPara) includes the pre-trained base large model identifier (expressed by BaseLLMID), the fine-tuned large model identifier (expressed by FineLLMID), and the fine-tuned large model non-shared parameters (expressed by FineLLMPara).

[0120] Step 604: the large model management component saves the information and saves the fine-tuned large model non-shared parameters FineLLMPara to the large model parameter repository.

[0121] When storing parameters, FineLLMPara is saved in the large model non-shared parameter repository, while in the large model shared parameter repository, a variety of pre-trained basic large model parameters are stored in advance.

[0122] Step 606: The user initiates a large model invocation request to the multi-tenant large model inference system. The request data includes an identifier of the fine-tuned large model to be invoked and user input text data.

[0123] Among them, since multiple users may call multiple fine-tuned large model inferences at the same time, each request is represented by Query 0, Query 1, ..., Query n, and each Query contains FineLLMID and InputText.

[0124] Step 608: The multi-tenant request preprocessing component preprocesses the user request to obtain batches calculated using the shared large model parameters and batches calculated using the non-shared large model parameters.

[0125] The multi-tenant request preprocessing component initiates a request to the big model information query module of the big model management component to query the BaseLLMID corresponding to the FineLLMID in each user request, and then calls the user request classification module for classification. The user requests are classified according to two classification methods. The first method classifies user requests with the same BaseLLMID into one category (this part can be calculated using the same shared big model parameters), and the second method classifies user requests with the same FineLLMID into one category (this part can be calculated using the same non-shared big model parameters). Finally, the user input preprocessing module is called to process user requests in the same category with the same BaseLLMID into batches of fixed size, and padding operations are performed on different user text input lengths in the same batch to make their dimensions the same and can be batch processed by the batch inference engine. In addition, the user requests in the same category with the same FineLLMID are also padding.

[0126] In step 610 , the multi-tenant large model inference system calls the batch scheduling component to perform batch scheduling on the two types of batches obtained through preprocessing.

[0127] Among them, the batches calculated using shared large model parameters are scheduled to the shared parameter batch processing module for batch calculation, and the batches calculated using non-shared large model parameters are scheduled to the non-shared parameter batch processing module for batch calculation. The shared parameter batch processing module reads and loads the shared frozen parameters of various BaseLLMIDs from the large model shared parameter repository, and the non-shared parameter batch processing module reads and loads the non-shared fine-tuning parameters of various FineLLMIDs from the large model non-shared parameter repository.

[0128] In step 612, the multi-tenant batch reasoning component performs batch processing according to the scheduling results, merges the results to obtain the target results, and feeds back to the user.

[0129] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0130] Refer to the following Figure 7 To describe the model-as-a-service inference optimization device 700 according to an embodiment of the present disclosure. Figure 7 The model-as-a-service inference optimization device 700 shown is merely an example and should not bring any limitation to the functionality and scope of use of the embodiments of the present disclosure.

[0131] The inference optimization device 700 of the model as a service is presented in the form of a hardware module or a software module. The components of the inference optimization device 700 of the model as a service may include but are not limited to: a preprocessing module 702, which is used to respond to the call requests received by multiple users to call multiple fine-tuned large model inferences, preprocess the call requests, generate a first request set and a second request set, the first request set corresponds to the inference of the basic large model, and the second request set corresponds to the inference of the fine-tuned large model; an inference module 704, which is used to infer the first request set based on the shared frozen parameters of the basic large model to obtain the first batch of processing results, and infer the second request set based on the non-shared parameters of the fine-tuned large model to obtain the second batch of processing results; a merging module 706, which is used to merge the first batch of processing results and the second batch of processing results to obtain the target result, and feed the target result back to multiple users.

[0132] Refer to the following Figure 8 To describe the model-as-a-service inference optimization device 800 according to an embodiment of the present disclosure. Figure 8The model-as-a-service inference optimization device 800 shown is merely an example and should not bring any limitation to the functionality and scope of use of the embodiments of the present disclosure.

[0133] The inference optimization device 800 of the model as a service is presented in the form of a hardware module or a software module. The components of the inference optimization device 800 of the model as a service may include but are not limited to: a sending module 802, used to send a call request for invoking the fine-tuning large model inference to the server, the call request including the fine-tuning large model identifier; a receiving module 804, used to receive the target result generated by the server based on the call request, wherein the server determines the matching basic large model identifier based on the fine-tuning large model identifier, and merges the first batch of inference results generated based on the basic large model identifier and the second batch of processing results generated based on the fine-tuning large model identifier to obtain the target result.

[0134] like Fig. 9 As shown, the electronic device 900 is in the form of a general computing device. The components of the electronic device 900 may include but are not limited to: at least one processing unit 910, at least one storage unit 920, and a bus 930 connecting different system components (including the storage unit 920 and the processing unit 910).

[0135] The storage unit stores a program code, which can be executed by the processing unit 910, so that the processing unit 910 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 910 can perform the following steps: Figure 2 The scheme described in .

[0136] The storage unit 920 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 9201 and / or a cache storage unit 9202 , and may further include a read-only storage unit (ROM) 9203 .

[0137] The storage unit 920 may also include a program / utility 9204 having a set (at least one) of program modules 9205, such program modules 9205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0138] Bus 930 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0139] The electronic device 900 may also communicate with one or more external devices 790 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 900, and / or may communicate with any device that enables the electronic device 900 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 950. Furthermore, the electronic device 900 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 960. As shown, the network adapter 960 communicates with other modules of the electronic device 900 via a bus 930. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0140] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or an electronic device, etc.) to execute the method according to the implementation of the present disclosure.

[0141] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on an electronic device, the program code is used to enable the electronic device to perform the steps according to various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of the present specification.

[0142] According to the program product for implementing the above method according to the embodiment of the present disclosure, it can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on an electronic device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0143] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0144] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0145] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0146] The program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the IoT terminal, as an independent software package, partially on the user computing device and partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0147] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of a module or unit described above can be further divided into multiple modules or units to be concretized. In addition, although the various steps of the method in the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that these steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired result. In addition or alternatively, some steps can be omitted, multiple steps can be combined into one step for execution, and / or one step can be decomposed into multiple steps for execution, etc.

[0148] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or an electronic device, etc.) to execute the method according to the implementation of the present disclosure.

[0149] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure herein. This application is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A method for inference optimization of a model as a service, characterized in that: Applied to the server, including: In response to receiving multiple user call requests for inference of multiple fine-tuned large models, preprocessing the call requests to generate a first request set and a second request set, wherein the first request set corresponds to the inference of the basic large model, and the second request set corresponds to the inference of the fine-tuned large model; Reasoning the first request set based on the shared frozen parameters of the basic large model to obtain a first batch of processing results, and reasoning the second request set based on the non-shared parameters of the fine-tuned large model to obtain a second batch of processing results; The first batch of processing results and the second batch of processing results are combined to obtain a target result, and the target result is fed back to the multiple users.

2. The inference optimization method of model as a service according to claim 1, characterized in that: In response to receiving multiple user call requests for invoking multiple fine-tuned large model inferences, before preprocessing the call requests, the method further includes: An information triplet of the fine-tuning large model submitted by a user is received, and the information triplet is registered and saved, wherein the information triplet includes a basic large model identifier, a corresponding fine-tuning large model identifier, and non-shared parameters of the fine-tuning large model.

3. The inference optimization method of model as a service according to claim 2, characterized in that: Also includes: The non-shared parameters of the fine-tuned large model are stored in a large model parameter repository, wherein the large model parameter repository also includes the shared frozen parameters identified by a plurality of the basic large models.

4. The inference optimization method of model as a service according to claim 2, characterized in that: In response to receiving multiple user call requests for invoking multiple fine-tuned large model inferences, preprocessing the call requests to generate a first request set and a second request set, including: In response to the call request, extracting the fine-tuning large model identifier in the call request; Querying the basic large model identifier that matches the fine-tuning large model identifier; Generate a shared parameter batch based on a first call request with the same identifier of the basic large model, and generate a non-shared parameter batch based on a second call request with the same identifier of the fine-tuning large model; The first request set is generated based on the multiple shared parameter batches corresponding to the multiple basic large model identifiers, and the second request set is generated based on the multiple non-shared parameter batches corresponding to the multiple fine-tuning large model identifiers.

5. The inference optimization method of model as a service according to claim 4, characterized in that: The call request also includes text data input by a user, generating a shared parameter batch based on the call request with the same basic large model identifier, and generating a non-shared parameter batch based on the call request with the same fine-tuning large model identifier, including: For the first call request having the same basic large model identifier, performing a dimension consistency operation on the text data in the first call request to obtain the shared parameter batch; For the second call request having the same identifier of the fine-tuning large model, a dimension consistency operation is performed on the text data in the second call request to obtain the non-shared parameter batch.

6. The inference optimization method of model as a service according to claim 1, characterized in that: Before inferring the first request set based on the shared frozen parameters of the basic large model to obtain the first batch of processing results and inferring the second request set based on the non-shared parameters of the fine-tuned large model to obtain the second batch of processing results, the method further includes: The first request set is scheduled to a shared parameter batch processing module to perform reasoning on the first request set, and the second request set is scheduled to a non-shared parameter batch processing module to perform reasoning on the second request set.

7. The inference optimization method of model as a service according to claim 3, characterized in that: Reasoning the first request set based on the shared frozen parameters of the basic large model to obtain a first batch of processing results, and reasoning the second request set based on the non-shared parameters of the fine-tuned large model to obtain a second batch of processing results, including: Reading and loading the shared frozen parameters corresponding to the basic large model identifier and the non-shared parameters corresponding to the fine-tuning large model identifier from the large model parameter repository; Inputting the first request set into a processing framework loaded with the shared freezing parameters to perform batch processing operations, and obtaining the first batch of processing results; The second request set is input into the processing framework loaded with the non-shared parameters to perform batch processing operations, so as to obtain the second batch processing results.

8. The inference optimization method of model as a service according to claim 1, characterized in that: Combining the first batch of processing results and the second batch of processing results to obtain a target result, including: determining a first weight of the shared frozen parameter and a second weight of the unshared parameter; The first weight is assigned to the first batch of processing results, and the second weight is assigned to the second batch of processing results, so as to merge the first batch of processing results and the second batch of processing results to obtain a target result.

9. A method for inference optimization of a model as a service, characterized in that: Applied to tenants, including: Sending a call request for invoking fine-tuning large model inference to the server, wherein the call request includes a fine-tuning large model identifier; Receive the target result generated by the server based on the call request, wherein the server determines a matching basic large model identifier based on the fine-tuning large model identifier, and merges a first batch of inference results generated based on the basic large model identifier and a second batch of processing results generated based on the fine-tuning large model identifier to obtain the target result.

10. The inference optimization method of model as a service according to claim 9, characterized in that: Before sending a call request to the server to fine-tune the large model inference, the following is also included: Submit an information triplet of the fine-tuned large model to the server, and register and save the information triplet, wherein the information triplet includes a basic large model identifier, a corresponding fine-tuned large model identifier, and non-shared parameters of the fine-tuned large model, wherein the first batch of processing results are obtained by reasoning based on the shared frozen parameters, and the second batch of processing results are obtained by reasoning based on the non-shared parameters.

11. A model-as-a-service inference optimization device, characterized in that: Applied to the server, including: A preprocessing module, configured to preprocess the call requests received from multiple users for invoking the inference of multiple fine-tuned large models, and generate a first request set and a second request set, wherein the first request set corresponds to the inference of the basic large model, and the second request set corresponds to the inference of the fine-tuned large model; an inference module, configured to infer the first request set based on the shared frozen parameters of the basic large model to obtain a first batch of processing results, and to infer the second request set based on the non-shared parameters of the fine-tuned large model to obtain a second batch of processing results; A merging module is used to merge the first batch of processing results and the second batch of processing results to obtain a target result, and feed back the target result to the multiple users.

12. A model-as-a-service inference optimization device, characterized in that: Applied to tenants, including: A sending module, used to send a call request for invoking the fine-tuning large model inference to the server, wherein the call request includes a fine-tuning large model identifier; A receiving module is used to receive the target result generated by the server based on the call request, wherein the server determines the matching basic large model identifier based on the fine-tuning large model identifier, and merges the first batch of inference results generated based on the basic large model identifier and the second batch of processing results generated based on the fine-tuning large model identifier to obtain the target result.

13. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the model-as-a-service reasoning optimization method described in any one of claims 1 to 8, or the model-as-a-service reasoning optimization method described in claim 9 or 10 by executing the executable instructions.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model-as-a-service reasoning optimization method described in any one of claims 1 to 10 is implemented.

15. A computer program product having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model-as-a-service reasoning optimization method described in any one of claims 1 to 10 is implemented.

Images