Large model automatic deployment method, device, equipment and medium

Abstract

The present disclosure provides a large model automatic deployment method, apparatus, equipment and medium. By executing the large model automatic deployment method, the original large model, lightweight large model, target device type of the target device and the target model architecture that can be run on the target device can be determined based on the input of the user end, and in order to adapt to target devices with different performance differences, scheduling information for pre-registered model conversion services is automatically generated. Since the model distillation service, model distillation service and model deployment service are all pre-registered, no manual intervention is required when generating directed acyclic graph task information, and the execution order between each service is constrained by the directed acyclic graph task information. After executing the directed acyclic graph task information, each service can be scheduled in sequence, and the large model can be automatically distilled, converted and other operations can be performed, and finally the converted lightweight large model can be deployed to the target device, which can improve the efficiency and quality of large model deployment.

Description

Technical Field

[0001] The present disclosure relates to the technical field of large-scale model deployment, and in particular to a large-scale model automatic deployment method, apparatus, device, and medium. Background Art

[0002] With the advancement of artificial intelligence (AI) technology, large language models (LLMs) have achieved remarkable performance in fields such as natural language processing and computer vision. Large language models, also known as large models, require a large number of parameters and complex computing requirements during deployment. Rapid deployment of these models is crucial to their promotion and application.

[0003] In related technologies, due to the limited performance of terminal devices, large models need to be adapted to varying degrees during deployment to terminal devices. However, due to the large differences in performance among different terminal devices, it is often necessary to rely on manual deployment operations of large models based on the characteristics of each terminal device, which greatly reduces the efficiency and quality of large model deployment. Summary of the Invention

[0004] The main purpose of the embodiments of the present disclosure is to propose a large-scale model automatic deployment method, device, equipment and medium, which can improve the efficiency and quality of large-scale model deployment.

[0005] To achieve the above objectives, a first aspect of an embodiment of the present disclosure provides a large model automatic deployment method, comprising:

[0006] Acquire downstream task information input by the user end, and determine the original large model, the lightweight large model, the target device type of the target device, and the target model architecture that can be run on the target device based on the downstream task information;

[0007] When at least one of the following conditions is met: the distillation device type in the process of distilling the original large model into the lightweight large model is inconsistent with the target device type, and the original model architecture of the lightweight large model is inconsistent with the target model architecture, generating scheduling information for a pre-registered model conversion service;

[0008] Constructing scheduling information for a model distillation service for distilling the pre-registered original large model into the lightweight large model, and constructing scheduling information for a model deployment service for the pre-registered target device, and constructing directed acyclic graph task information based on the sequence of all the scheduling information;

[0009] When the directed acyclic graph task information is running, based on each of the scheduling information, the model distillation service is first scheduled to distill the original large model into the lightweight large model, and then the model conversion service is scheduled to convert the distilled lightweight large model to the target device type or the target model architecture, and the model deployment service is scheduled to deploy the converted lightweight large model to the target device.

[0010] In some embodiments, constructing the directed acyclic graph task information based on the order of all the scheduling information includes:

[0011] sequentially configuring scheduling dependencies corresponding to adjacent services between each of the scheduling information of the model distillation service, the model conversion service, and the model deployment service, and constructing directed acyclic graph task information based on the order of all the scheduling information and the corresponding scheduling dependencies;

[0012] When the directed acyclic graph task information is executed, first scheduling the model distillation service to distill the original large model into the lightweight large model based on each of the scheduling information, then scheduling the model conversion service to convert the distilled lightweight large model to the target device type or the target model architecture, and then scheduling the model deployment service to deploy the converted lightweight large model to the target device, including:

[0013] When the directed acyclic graph task information is running, the model distillation service, the model conversion service, and the model deployment service are sequentially scheduled based on the respective scheduling information;

[0014] When running the model distillation service, distilling the original large model into the lightweight large model;

[0015] When the model conversion service is running, determining whether the model distillation service it depends on is completed, and after the model distillation service is completed, scheduling the model conversion service to convert the distilled lightweight large model to the target device type or the target model architecture;

[0016] When the model deployment service is running, it is determined whether the model conversion service it depends on is completed, and after the model conversion service is completed, the model deployment service is scheduled to deploy the converted lightweight large model to the target device.

[0017] In some embodiments, scheduling the model distillation service to distill the original large model into the lightweight large model, scheduling the model conversion service to convert the distilled lightweight large model to the target device type or the target model architecture, and scheduling the model deployment service to deploy the converted lightweight large model to the target device, includes:

[0018] After scheduling the model distillation service, a distillation container is created, the original large model and the lightweight large model are mounted on the distillation container, and the original large model is distilled into the lightweight large model within the distillation container;

[0019] After scheduling the model conversion service, a conversion image is built into the distillation container, and the distilled lightweight large model is converted to the target device type or the target model architecture through the conversion image;

[0020] After scheduling the model deployment service, the lightweight large model converted in the distillation container is deployed to the target device.

[0021] In some embodiments, determining the original large model, the lightweight large model, the target device type of the target device, and the target model architecture that can be run on the target device based on the downstream task information includes:

[0022] Extracting a target device and a task description to be executed in the target device from the downstream task information;

[0023] Obtaining a device description of the target device, a target device type, and a target model architecture that can be run on the target device from a preset database;

[0024] Extracting a first text feature of the task description and a second text feature of the device description, and fusing the first text feature and the second text feature to obtain a fused feature;

[0025] Based on the fusion features, a plurality of preset large models to be selected are predicted to obtain prediction results, and based on the prediction results, an original large model and a lightweight large model are selected from the plurality of large models to be selected.

[0026] In some embodiments, the step of predicting a plurality of preset large models to be selected based on the fusion features to obtain prediction results, and selecting an original large model and a lightweight large model from the plurality of large models to be selected based on the prediction results, includes:

[0027] Inputting the fusion features into a preset first prediction model, predicting a large model that meets the requirements of the task description among multiple large models to be selected, obtaining a first prediction result, and selecting an original large model from the multiple large models to be selected based on the first prediction result;

[0028] The fusion features are input into a preset second prediction model, and a large model that meets the device description requirements of multiple large models to be selected is predicted to obtain a second prediction result, and a lightweight large model is selected from the multiple large models to be selected based on the second prediction result.

[0029] In some embodiments, scheduling the model distillation service to distill the original large model into the lightweight large model includes:

[0030] Determining a distillation dataset based on the downstream task information;

[0031] The model distillation service is scheduled, and the original large model is distilled into the lightweight large model using the distillation dataset.

[0032] In some embodiments, the large model automatic deployment method further includes:

[0033] When the distillation device type in the process of distilling the original large model into the lightweight large model is consistent with the target device type, and the original model architecture of the lightweight large model is consistent with the target model architecture, constructing directed acyclic graph task information based on the scheduling information of the model distillation service and the scheduling information of the model deployment service;

[0034] When the directed acyclic graph task information is running, the model distillation service is first scheduled to distill the original large model into the lightweight large model based on each scheduling information, and the model deployment service is scheduled to deploy the distilled lightweight large model to the target device.

[0035] To achieve the above objectives, a second aspect of the embodiments of the present disclosure provides a large model automatic deployment device, comprising:

[0036] An information receiving module is used to obtain downstream task information input by a user terminal, and determine the original large model, the lightweight large model, the target device type of the target device, and the target model architecture that can be run on the target device based on the downstream task information;

[0037] a scheduling confirmation module, configured to generate scheduling information for a pre-registered model conversion service when at least one of the following conditions is met: a distillation device type in the process of distilling the original large model into the lightweight large model is inconsistent with the target device type, and an original model architecture of the lightweight large model is inconsistent with the target model architecture;

[0038] A directed acyclic graph construction module is used to construct scheduling information for a model distillation service for distilling the pre-registered original large model into the lightweight large model, and to construct scheduling information for a model deployment service for the pre-registered target device, and to construct directed acyclic graph task information based on the order of all the scheduling information;

[0039] An automatic deployment module is used to, when the directed acyclic graph task information is running, first schedule the model distillation service to distill the original large model into the lightweight large model based on each scheduling information, then schedule the model conversion service to convert the distilled lightweight large model to the target device type or the target model architecture, and schedule the model deployment service to deploy the converted lightweight large model to the target device.

[0040] To achieve the above-mentioned purpose, the third aspect of the embodiments of the present disclosure proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the large model automatic deployment method described in the first aspect embodiment.

[0041] To achieve the above-mentioned purpose, the fourth aspect of the embodiment of the present disclosure proposes a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by the processor, it implements the large model automatic deployment method described in the first aspect embodiment.

[0042] The large model automatic deployment method, device, equipment and medium proposed in the embodiments of the present disclosure, and the large model automatic deployment method can be applied in the large model automatic deployment device. By executing the large model automatic deployment method, the original large model, the lightweight large model, the target device type of the target device and the target model architecture that can be run on the target device can be determined based on the input of the user end, and in order to adapt to target devices with different performance differences, scheduling information for pre-registered model conversion services is automatically generated. Since the model distillation service, the model distillation service and the model deployment service are all pre-registered, no manual intervention is required when generating the directed acyclic graph task information, and the execution order between each service is constrained by the directed acyclic graph task information. After executing the directed acyclic graph task information, each service can be scheduled in sequence, and the large model can be automatically distilled, converted and other operations can be performed, and finally the converted lightweight large model can be deployed to the target device, which can improve the efficiency and quality of large model deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present disclosure;

[0044] Figure 2This is a flow chart of a large model automatic deployment method provided by an embodiment of the present disclosure;

[0045] Figure 3 yes Figure 2 Step S104 further includes a flow chart;

[0046] Figure 4 yes Figure 2 Another flowchart diagram further comprising step S104;

[0047] Figure 5 yes Figure 2 Step S101 further includes a flow chart;

[0048] Figure 6 yes Figure 5 Step S404 further includes a flow chart;

[0049] Figure 7 yes Figure 2 Another flowchart diagram further comprising step S104;

[0050] Figure 8 This is another flowchart of the large model automatic deployment method provided by the embodiment of the present disclosure;

[0051] Figure 9 is a schematic diagram of a large model automatic deployment system provided by an embodiment of the present disclosure;

[0052] Figure 10 It is a schematic diagram of task information for constructing a directed acyclic graph provided by an embodiment of the present disclosure;

[0053] Figure 11 is a schematic diagram of the large model automatic deployment process provided by an embodiment of the present disclosure;

[0054] Figure 12 This is a schematic diagram of the functional modules of the large model automatic deployment device provided by an embodiment of the present disclosure;

[0055] Figure 13 Schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present disclosure. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not intended to limit the present disclosure.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art in the art of the present disclosure. The terms used herein are only for the purpose of describing the embodiments of the present disclosure and are not intended to limit the present disclosure.

[0058] First, some terms used in this disclosure are analyzed:

[0059] Artificial intelligence (AI) is a new technical discipline that studies and develops theories, methods, technologies, and application systems for simulating, extending, and expanding human intelligence. A branch of computer science, AI seeks to understand the essence of intelligence and create new intelligent machines that can respond in a manner similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thinking. It also encompasses theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results.

[0060] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0061] Large language models (LLMs) are deep learning models trained using large amounts of text data. They can generate natural language text or understand the meaning of text. Large language models can be used to detect large models and handle a variety of natural language tasks, such as text classification, question answering, and conversation, and are an important path to artificial intelligence.

[0062] Object Storage Service (OBS) is an object-based storage service that provides users with massive, secure, highly reliable, and low-cost data storage capabilities. There is no need to consider capacity limitations when using it, and it provides a variety of storage types to choose from.

[0063] A Directed Acyclic Graph (DAG) is a directed graph where the edges have a direction and there are no cycles in the graph.

[0064] With the development of artificial intelligence (AI), large models have achieved remarkable performance in fields such as natural language processing and computer vision. However, large models require a large number of parameters and complex computational requirements during deployment. Therefore, rapid deployment of large models is crucial to their promotion and application.

[0065] In related technologies, due to the limited performance of terminal devices, large models need to be adapted to varying degrees during deployment to terminal devices. However, due to the large differences in performance among different terminal devices, it is often necessary to rely on manual deployment operations of large models based on the characteristics of each terminal device, which greatly reduces the efficiency and quality of large model deployment.

[0066] Based on this, the embodiments of the present disclosure provide a large model automatic deployment method, device, equipment and medium, which can improve the efficiency and quality of large model deployment.

[0067] The large model automatic deployment method in the embodiments of the present disclosure can be illustrated by the following embodiments.

[0068] The embodiments of the present disclosure can acquire and process relevant data based on artificial intelligence technology.

[0069] See also Figure 1 , Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present disclosure, wherein the implementation environment includes a terminal and a server, wherein the terminal and the server are connected via a communication network.

[0070] Exemplarily, the server can obtain downstream task information sent by the terminal, and determine the original large model, the lightweight large model, the target device type of the target device, and the target model architecture that can be run on the target device based on the downstream task information; when at least one of the distillation device type in the process of distilling the original large model to the lightweight large model is inconsistent with the target device type, and the original model architecture of the lightweight large model is inconsistent with the target model architecture, generate scheduling information for the pre-registered model conversion service; construct scheduling information for the model distillation service for distilling the pre-registered original large model to the lightweight large model, and construct scheduling information for the model deployment service for the pre-registered target device, and construct directed acyclic graph task information based on the order of all scheduling information; when the directed acyclic graph task information is running, first schedule the model distillation service to distill the original large model to the lightweight large model based on each scheduling information, then schedule the model conversion service to convert the distilled lightweight large model to the target device type or target model architecture, and schedule the model deployment service to deploy the converted lightweight large model to the target device.

[0071] The terminal can also serve as a target device and ultimately receive the converted lightweight large model sent by the server, and deploy and apply the large model.

[0072] In some embodiments, the terminal can be a smart phone, tablet computer, laptop computer, desktop computer, etc.; the server can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the large model automatic deployment method can also be deployed in the software in the server, and the software can be an application that implements the large model automatic deployment method, etc., but is not limited to the above forms.

[0073] The present disclosure can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present disclosure can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present disclosure can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0074] It should be noted that in various specific embodiments of the present disclosure, when obtaining downstream task information input by the user end for related processing, the user's permission or consent will be obtained first. Moreover, the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present disclosure needs to obtain downstream task information, the user's separate permission or consent can be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary downstream task information for the normal operation of the embodiment of the present disclosure can be obtained.

[0075] See also Figure 2 , Figure 2 This is a flow chart of the large model automatic deployment method provided by the embodiment of the present disclosure. Figure 2 The method may include but is not limited to steps S101 to S104.

[0076] Step S101, obtaining downstream task information input by the user terminal, and determining the original large model, the lightweight large model, the target device type of the target device, and the target model architecture that can be run on the target device based on the downstream task information;

[0077] Step S102: When at least one of the following conditions is met: the distillation device type during the distillation process of the original large model to the lightweight large model is inconsistent with the target device type, and the original model architecture of the lightweight large model is inconsistent with the target model architecture, generating scheduling information for a pre-registered model conversion service;

[0078] Step S103: constructing scheduling information for a model distillation service for distilling a pre-registered original large model into a lightweight large model, and constructing scheduling information for a model deployment service for a pre-registered target device, and constructing directed acyclic graph task information based on the order of all scheduling information;

[0079] Step S104, when the directed acyclic graph task information is running, based on each scheduling information, the model distillation service is first scheduled to distill the original large model into a lightweight large model, and then the model conversion service is scheduled to convert the distilled lightweight large model to the target device type or target model architecture, and the model deployment service is scheduled to deploy the converted lightweight large model to the target device.

[0080] For the above-mentioned step S101, the user end can be a terminal or an input module on the server, and the embodiment of the present disclosure does not impose specific restrictions on this. The user end can be provided with a mobile application, a web interface or a command line tool, and the user can input specific downstream task information through these tools to characterize some necessary information required for the downstream task. Exemplarily, the downstream task information can be some specific information input by the user end for selecting the large model deployment process, such as the large model id, small model id, data set id, version id, distillation component configuration, distillation device chip type, target device chip type, target device model architecture, target device address, inference algorithm and other information; or, the downstream task information is a description of the task that the target device needs to perform.

[0081] In the embodiment of the present disclosure, the original large model, the lightweight large model, the target device type of the target device, and the target model architecture that can be run on the target device can be determined based on the downstream task information. For example, if the downstream task information contains some specific information in the large model deployment process, the required large model can be directly selected as the original large model based on the input of the user end, and the required large model can be selected as a small model, that is, a lightweight large model, and the target device type of the target device and the target model architecture that can be run on the target device can be directly determined; or, if the downstream device information contains a description of the task that the target device needs to perform, the downstream device information can be parsed and analyzed to obtain the required original large model, lightweight large model, the target device type of the target device, and the target model architecture that can be run on the target device.

[0082] It should be noted that the original large model is a relatively powerful and complete pre-trained large model that can perform complex tasks. However, due to its large number of parameters, it is not suitable for deployment in some target devices with poor performance. The lightweight large model is a large model with a smaller number of parameters than the original large model. It can also be called a small model. It is suitable for deployment in some target devices with poor performance. For example, the lightweight large model can also be formed by compressing the original large model. However, due to its relatively poor performance, it needs to be adjusted before it can be deployed to the target device, including but not limited to model distillation, model conversion and other operations, and the embodiments of the present disclosure do not impose specific restrictions on this.

[0083] For the above-mentioned step S102, the embodiment of the present disclosure determines whether additional conversion of the lightweight large model is required to adapt to the characteristics and requirements of the target device, including the need to determine whether at least one of the following conditions is met: the type of distillation equipment in the process of distilling the original large model to the lightweight large model is inconsistent with the target device type, and the original model architecture of the lightweight large model is inconsistent with the target model architecture.

[0084] The distillation device type is the device type used when distilling the original large model into a lightweight large model. Different distillation devices have different computing capabilities, memory sizes, storage speeds, etc. Furthermore, the distillation device type can be the distillation device chip type; the target device type is the device type of the target device on which the lightweight large model is ultimately deployed, corresponding to the distillation device type. Furthermore, the target device type can be the target device chip type. By comparing these two device types, it can be determined whether there are significant differences that will affect the performance of the lightweight large model on the target device. If there are significant differences between the distillation device type and the target device type in terms of computing power, memory limitations, or hardware acceleration support, then the lightweight large model needs to be further converted.

[0085] The original model architecture is the architecture used by the lightweight large model during the distillation process, including the model's hierarchical structure, number and type of parameters, etc. The target model architecture is the model architecture that can be run on the target device, which is limited by the target device's hardware and software environment. By comparing the two model architectures, we can determine whether there are any incompatibilities. For example, if the target device only supports a specific type of neural network layer, and the lightweight large model contains an incompatible layer type, then the lightweight large model needs to be converted to match the architectural requirements of the target device.

[0086] Model conversion services are pre-defined and registered. They receive scheduling information and convert lightweight large models based on this information. If any of the above checks find an inconsistency, the disclosed embodiment generates scheduling information for the pre-registered model conversion service. This scheduling information instructs the model conversion service on how to perform the necessary conversion or optimization on the lightweight large model.

[0087] For the above-mentioned step S103, the embodiment of the present disclosure can construct scheduling information for the model distillation service, which is used to execute the distillation process; it can also construct scheduling information for the model conversion service so as to perform necessary conversion when the lightweight large model after distillation is incompatible with the target device or target model architecture.

[0088] Next, to ensure that each service is executed in the correct order and avoid circular dependencies, a directed acyclic graph task information is constructed. This task information is used to express the directed acyclic graph required during the service execution process. The directed acyclic graph includes multiple nodes, each node represents a service, such as model distillation service, model conversion service, model deployment service, etc. The directed edges between nodes represent the execution order and dependency relationship between services. Finally, a unique identifier is assigned to each node in the directed acyclic graph.

[0089] Therefore, by constructing these scheduling information and directed acyclic graph task information, it is possible to ensure that each step of the large model automatic deployment process is executed according to the predetermined order and conditions, thereby improving the efficiency and reliability of deployment while reducing the need for manual intervention.

[0090] For the above step S104, after obtaining the directed acyclic graph task information, the embodiment of the present disclosure can schedule each service in sequence based on the directed acyclic graph indicated by the directed acyclic graph task information, including scheduling model distillation service, model conversion service, model deployment service, etc. in sequence.

[0091] Specifically, during the process of scheduling and executing the model distillation service, the original large model and the parameters required for distillation, such as the distillation algorithm and the data set, can be passed as input to the model distillation service. The model distillation service will execute the distillation algorithm and distill the original large model into a lightweight large model. The service outputs the distilled lightweight large model, which will serve as the input for subsequent services.

[0092] Subsequently, during the scheduling and execution of the model conversion service, the distilled lightweight large model and the parameters required for conversion, such as the number of quantization bits, pruning ratio, target device type, or target model architecture, can be used as input. The model conversion service will perform necessary conversions on the distilled lightweight large model based on the input to adapt to the type and architecture of the target device, and finally output the converted lightweight large model.

[0093] Finally, the converted lightweight large model is deployed to the target device during the invocation and execution of the model deployment service. This involves uploading the model file to the device, configuring the operating environment on the device, and performing necessary testing or inference to ensure the model runs correctly. This enables automated deployment of large models, significantly improving deployment efficiency and quality. Because the entire process is automated, users only need to input downstream task information through the client during deployment, significantly reducing the need for manual intervention and lowering the error rate during deployment.

[0094] In summary, the embodiment of the present disclosure, through steps S101 to S104, executes the large model automatic deployment method, which can determine the original large model, the lightweight large model, the target device type of the target device, and the target model architecture that can be run on the target device based on the input of the user terminal, and automatically generates scheduling information for the pre-registered model conversion service in order to adapt to target devices with different performance differences. Since the model distillation service, the model distillation service and the model deployment service are all pre-registered, no manual intervention is required when generating the directed acyclic graph task information, and the execution order between each service is constrained by the directed acyclic graph task information. After executing the directed acyclic graph task information, each service can be scheduled in sequence, and the large model can be automatically distilled, converted and other operations can be performed, and finally the converted lightweight large model can be deployed to the target device, which can improve the efficiency and quality of large model deployment.

[0095] In some embodiments, the process of constructing the directed acyclic graph task information based on the order of all the scheduling information in step S103 may further include:

[0096] The scheduling dependencies corresponding to adjacent services are configured in turn for each scheduling information of the model distillation service, model conversion service, and model deployment service, and a directed acyclic graph task information is constructed based on the order of all scheduling information and the corresponding scheduling dependencies.

[0097] In the above steps, the scheduling information of the model distillation service, model conversion service and model deployment service are configured with scheduling dependencies corresponding to adjacent services, so that the execution of the latter service needs to wait until the dependent service is executed. Specifically, in the embodiment of the present disclosure, the model distillation service, model conversion service and model deployment service are sequentially used as nodes in a directed acyclic graph, wherein the model conversion service and the previous model distillation service have a scheduling dependency, so that after scheduling, the model conversion service needs to check whether the previous model distillation service has been executed, and only after the model distillation service has been executed can the model conversion service be run; in addition, the model deployment service and the previous model conversion service have a scheduling dependency, so that after scheduling, the model deployment service needs to check whether the previous model conversion service has been executed, and only after the model conversion service has been executed can the model deployment service be run.

[0098] In addition, it should be noted that when there are many nodes in the directed acyclic graph, the service on any node can also be scheduled as a dependency, so that the service on the node will only be run after any previous service is executed. That is, the embodiment of the present disclosure is not limited to relying solely on the execution of the previous service.

[0099] It should be noted that the embodiments of the present disclosure only use model distillation service, model conversion service and model deployment service as examples for illustration. On the premise of meeting the requirements of the embodiments of the present disclosure, the directed acyclic graph task can also include other services as nodes. The embodiments of the present disclosure can configure scheduling dependencies based on the services on each node to construct the directed acyclic graph task information required by the embodiments of the present disclosure.

[0100] See also Figure 3 , Figure 3 yes Figure 2 In some embodiments, step S104 may include steps S201 to S204:

[0101] Step S201: When the directed acyclic graph task information is running, the model distillation service, the model conversion service, and the model deployment service are scheduled in sequence based on the respective scheduling information;

[0102] Step S202: When running the model distillation service, the original large model is distilled into a lightweight large model;

[0103] Step S203: When the model conversion service is running, determine whether the dependent model distillation service is completed. After the model distillation service is completed, schedule the model conversion service to convert the distilled lightweight large model to the target device type or target model architecture.

[0104] Step S204: When the model deployment service is running, determine whether the model conversion service it depends on is completed, and after the model conversion service is completed, schedule the model deployment service to deploy the converted lightweight large model to the target device.

[0105] In the above steps, when the directed acyclic graph task information starts to run, the embodiment of the present disclosure will schedule and execute each service in sequence according to the order of nodes and edges defined in the indicated directed acyclic graph, including model distillation service, model conversion service and model deployment service.

[0106] Among them, in order, the model distillation service is scheduled and executed, and the model distillation service will execute the distillation algorithm to distill the original large model into a lightweight large model. Since the model conversion service depends on the execution of the model distillation service, before scheduling the model conversion service, the embodiment of the present disclosure will check whether the model distillation service it depends on has been completed. Only when the model distillation service is successfully completed, the model conversion service will be run, and the distilled lightweight large model will be converted to the target device type or target model architecture through the model conversion service to adapt to the target device. The model deployment service depends on the execution of the model conversion service. Therefore, before scheduling the model deployment service, the embodiment of the present disclosure will check whether the model conversion service it depends on has been completed. Only when the model conversion service is successfully completed, the model deployment service will be run, and the converted lightweight large model will be deployed to the target device through the model deployment service.

[0107] In summary, the embodiment of the present disclosure can automatically complete the distillation, conversion and deployment process of the large model through the execution of steps S201 to S204. By configuring the scheduling dependencies between services, the smooth operation of various services can be guaranteed, and the subsequent services can be avoided from being started before the services that must be executed are completed, thereby greatly improving the efficiency and quality of large model deployment. At the same time, since the entire process is automated, the need for manual intervention can be significantly reduced, reducing the error rate in the deployment process.

[0108] See also Figure 4 , Figure 4 yes Figure 2 In some embodiments, step S104 may further include steps S301 to S303:

[0109] Step S301: After scheduling the model distillation service, a distillation container is created, the original large model and the lightweight large model are mounted on the distillation container, and the original large model is distilled into the lightweight large model within the distillation container;

[0110] Step S302: After scheduling the model conversion service, a conversion image is built into the distillation container, and the distilled lightweight large model is converted to the target device type or target model architecture through the conversion image;

[0111] Step S303: After scheduling the model deployment service, the converted lightweight large model in the distillation container is deployed to the target device.

[0112] In the above steps, during the process of scheduling and running the model distillation service, the disclosed embodiment first creates a distillation container, which is an independent, isolated environment for executing the model distillation task. Specifically, the disclosed embodiment uses container technologies such as Docker to create and manage the distillation container. Container technologies allow applications and their dependencies to be packaged together and deployed and run as a single unit. After the distillation container is created, the code or files of the original large model and the lightweight large model are mounted into the distillation container. Within the distillation container, the original large model is then distilled into the lightweight large model.

[0113] Subsequently, upon receiving an instruction to schedule the model conversion service, the disclosed embodiment sets a conversion image within the distillation container. This conversion image is a predefined environment containing conversion tools or scripts for converting the distilled lightweight large model into a format or architecture suitable for running on the target device. Specifically, the disclosed embodiment uses technologies such as Docker images to create and manage conversion images. These images contain all the dependencies and tools required to perform the conversion task. The tools or scripts in the conversion image are then run to convert the distilled lightweight large model into a format or architecture suitable for running on the target device, ultimately obtaining the converted lightweight large model.

[0114] Finally, upon receiving instructions from the scheduling model deployment service, the disclosed embodiment deploys the converted lightweight large model within the distillation container to the target device. Therefore, the disclosed embodiment automatically completes the distillation, conversion, and deployment of large models, significantly improving the efficiency and quality of large model deployment. Furthermore, because the entire process is based on container and image technology, it ensures isolation and repeatability between different steps, reducing the error rate during deployment.

[0115] See also Figure 5 , Figure 5 yes Figure 2 In some embodiments, step S101 may include steps S401 to S404:

[0116] Step S401: extracting a target device and a description of a task to be executed in the target device from the downstream task information;

[0117] Step S402, obtaining a device description of the target device, a target device type, and a target model architecture that can be run on the target device from a preset database;

[0118] Step S403: extracting a first text feature of the task description and a second text feature of the device description, and fusing the first text feature and the second text feature to obtain a fused feature;

[0119] Step S404: predicting the preset multiple candidate large models based on the fusion features to obtain prediction results, and selecting the original large model and the lightweight large model from the multiple candidate large models based on the prediction results.

[0120] In the above steps, when the downstream task information is a description of the task that needs to be performed by the target device, the embodiment of the present disclosure can extract and analyze the content in the downstream task information. Specifically, the downstream task information describes what kind of task is expected to be performed in the target device. For example, the user can input the input data of "deploy a large model in device A to realize the intelligent question and answer scenario in the field of science and technology" through the input and output interface provided by the client. The input data can be used as downstream task information. After the embodiment of the present disclosure uses natural language processing technology to parse the downstream task information, it can identify and extract keywords or phrases about the target device, as well as text describing the task. It can be obtained that device A is the target device provided, and "deploy a large model to realize the intelligent question and answer scenario in the field of science and technology" is the task description that needs to be performed in the target device.

[0121] Next, in the embodiment of the present disclosure, after determining the target device to be deployed, the device description of the target device, the target device type, and the target model architecture that can be run on the target device can be obtained from the preset database. Among them, the preset database is a business database provided in the embodiment of the present disclosure, which is a database for pre-storing information of various devices, and the target device is one of the devices. The database contains the device description of the target device, the target device type, and the target model architecture that can be run on the target device, wherein the device description can be the hardware specifications of the device, such as CPU type, memory size, storage space, operating system information, supported software environment, etc. The target device type is the device type of the target device on which the lightweight large model is ultimately to be deployed, which corresponds to the distillation device type and can be the target device chip type; the target model architecture refers to the type of large model architecture that the device can support, such as the PyTorch model architecture or the TensorRt model architecture. In addition, the data can also contain other relevant information of the target device, such as device compatibility information, performance evaluation data, user feedback, etc.

[0122] Subsequently, the disclosed embodiment can perform text feature extraction on the task description and the device description respectively to obtain a first text feature of the task description and a second text feature of the device description. The first text feature and the second text feature may include lexical features, syntactic features, semantic features, etc. under the corresponding description. For example, keywords such as "image", "recognition", and "technology field" can be extracted from the task description, and word embedding technology (such as Word2Vec) can be used to convert these keywords into vector representations. In addition, descriptions of target devices such as "8-core CPU", "4GB RAM", and "Android system" can be obtained from a preset database, and these descriptions can also be converted into vector representations using word embedding or other text representation methods.

[0123] The disclosed embodiment further fuses the extracted first and second text features to form a fused feature vector that contains both task-related information and target device-related information. The fusion method can be feature concatenation, weighted averaging, or using more complex deep learning models such as attention mechanisms and neural network layers to fuse features and generate a fused feature vector that integrates task requirements and device characteristics.

[0124] It should be noted that the fusion of the first text feature and the second text feature can help understand the task's requirements for model performance and the computing power that the device can provide, so as to select a model that can meet the task requirements without exceeding the device performance limitations. While considering the task description or the device description alone may lead to selection bias, by fusing the two features, both task and device factors can be comprehensively considered. With the continuous development of AI technology and the emergence of new devices, the content of the task description and the device description may change, and the method of fusing features can flexibly adapt to these changes. Therefore, in the embodiment of the present disclosure, by fusing the first text feature of the task description and the second text feature of the device description, the task requirements and the performance characteristics of the target device can be comprehensively considered, so as to more accurately select the appropriate original large model and lightweight large model.

[0125] Finally, the embodiment of the present disclosure can predict multiple preset large models to be selected based on the fusion features, obtain prediction results, and select the original large model and the lightweight large model from the multiple large models to be selected based on the prediction results. Specifically, the large models to be selected are some large models pre-stored in the embodiment of the present disclosure, which can meet the deployment needs of different scenarios. The embodiment of the present disclosure can use machine learning or deep learning algorithms to predict the most suitable large model based on the fusion feature vector, including classifiers, regression models, etc., and the output of the algorithm is the score or ranking of one or more large models to be selected, and the most suitable original large model and lightweight large model are selected according to these scores or rankings. Finally, the original large model and lightweight large model that are most suitable for the target device and task are selected from the multiple large models to be selected, and these models will be used in the subsequent model distillation and deployment process.

[0126] In summary, through the above steps, the embodiment of the present disclosure can automatically select the most suitable original large model and lightweight large model based on the downstream task information input by the user, and determine the type of target device and the target model architecture that can be run, which provides important input information for the subsequent model distillation, conversion and deployment processes.

[0127] See also Figure 6 , Figure 6 yes Figure 5 In some embodiments, step S404 may include steps S501 to S502:

[0128] Step S501: Input the fusion features into a preset first prediction model, predict a large model that meets the task description requirements among multiple large models to be selected, obtain a first prediction result, and select an original large model from the multiple large models to be selected based on the first prediction result;

[0129] In step S502, the fusion features are input into a preset second prediction model, a large model that meets the device description requirements of multiple candidate large models is predicted to obtain a second prediction result, and a lightweight large model is selected from the multiple candidate large models based on the second prediction result.

[0130] In the above steps, the first prediction model is a pre-trained classifier. Its function is to predict which candidate large models can meet specific task requirements based on the input fusion features. This classifier is a deep learning model that has been trained to identify features related to the task description and classify the candidate large models based on these features. When the fusion features are input to the first prediction model, the model will output a first prediction result. The first prediction result is a probability distribution that represents the probability that each candidate large model meets the requirements of the task description, or the first prediction result is a direct classification result that indicates which candidate large model is the most suitable. Based on the first prediction result, a candidate large model with the highest probability can be selected as the original large model for the subsequent model distillation process. In addition, if the first prediction result is a classification result, the candidate large model classified as the most suitable is directly selected as the original large model.

[0131] Similarly, the second prediction model is a pre-trained classifier, whose function is to predict which candidate large models can meet specific device requirements based on the input fusion features. This classifier is a deep learning model that has been trained to identify features related to the device description and classify the candidate large models based on these features. When the fusion features are input to the second prediction model, the model will output a second prediction result. The second prediction result is a probability distribution that represents the probability that each candidate large model meets the device description requirements, or the second prediction result is a direct classification result that indicates which candidate large model is the most suitable. Based on the second prediction result, a candidate large model with the highest probability can be selected as a lightweight large model for subsequent model distillation and deployment processes. In addition, if the second prediction result is a classification result, the candidate large model classified as the most suitable is directly selected as the lightweight large model.

[0132] See also Figure 7 , Figure 7 yes Figure 2 In some embodiments, the process of scheduling the model distillation service to distill the original large model into the lightweight large model in step S104 may further include steps S601 to S602:

[0133] Step S601, determining a distillation dataset based on downstream task information;

[0134] Step S602: Schedule the model distillation service and distill the original large model into a lightweight large model through the distillation dataset.

[0135] In the above steps, the embodiment of the present disclosure can also determine the distillation data set required for distillation based on the downstream task information. For example, when the downstream task information is some specific information input by the user end for selecting the large model deployment process, the embodiment of the present disclosure can determine the required data set as the distillation data set based on the input of the user end; or, when the downstream task information is a description of the task that needs to be performed by the target device, the embodiment of the present disclosure can be analyzed based on the downstream task information to filter out a data set that meets the current task requirements from multiple alternative data sets as the distillation data set. Finally, after determining the distillation data set, the embodiment of the present disclosure can schedule the model distillation service and distill the original large model into a lightweight large model through the distillation data set.

[0136] See also Figure 8 , Figure 8 This is another flow chart of the large model automatic deployment method provided by the embodiment of the present disclosure. In some embodiments, the large model automatic deployment method may further include steps S701 to S702:

[0137] Step S701: When the distillation device type used in the process of distilling the original large model into the lightweight large model is consistent with the target device type, and the original model architecture of the lightweight large model is consistent with the target model architecture, a directed acyclic graph task information is constructed based on the scheduling information of the model distillation service and the scheduling information of the model deployment service.

[0138] Step S702: When the directed acyclic graph task information is running, the model distillation service is first scheduled based on various scheduling information to distill the original large model into a lightweight large model, and the model deployment service is scheduled to deploy the distilled lightweight large model to the target device.

[0139] In the above steps, the embodiment of the present disclosure determines whether additional conversion of the lightweight large model is required to adapt to the characteristics and requirements of the target device, including the need to determine whether at least one of the following conditions is met: the type of distillation equipment in the process of distilling the original large model to the lightweight large model is inconsistent with the type of the target device, and the original model architecture of the lightweight large model is inconsistent with the target model architecture. When the type of distillation equipment in the process of distilling the original large model to the lightweight large model is consistent with the type of the target device, and the original model architecture of the lightweight large model is consistent with the target model architecture, it indicates that the model conversion service is no longer required, and therefore the directed acyclic graph task information can be directly constructed based on the order of scheduling information of services such as the scheduling information of the model distillation service and the scheduling information of the model deployment service.

[0140] Similarly, when model conversion service is not required, the constructed directed acyclic graph task information is also used to express the directed acyclic graph required during the service execution process. The directed acyclic graph includes multiple nodes, each node represents a service, such as model distillation service, model deployment service, etc. The directed edges between nodes represent the execution order and dependency relationship between services. Finally, a unique identifier is assigned to each node in the directed acyclic graph.

[0141] Therefore, by constructing these scheduling information and directed acyclic graph task information, it is possible to ensure that each step of the large model automatic deployment process is executed according to the predetermined order and conditions, thereby improving the efficiency and reliability of deployment while reducing the need for manual intervention.

[0142] Similarly, after obtaining directed acyclic graph task information when no model conversion service is required, the embodiments of the present disclosure can sequentially schedule the various services therein based on the directed acyclic graph indicated by the directed acyclic graph task information, including sequentially scheduling the model distillation service, the model deployment service, etc., so as to first schedule the model distillation service to distill the original large model into a lightweight large model based on the respective scheduling information, and then schedule the model deployment service to deploy the distilled lightweight large model to the target device. This embodiment of the present disclosure will not be further described in detail.

[0143] The above embodiment is described below by taking as an example the case where the downstream task information is some specific information input by the user end for selecting the large model deployment process:

[0144] First, see Figure 9 , Figure 9 This is a schematic diagram of the large model automatic deployment system provided by the embodiment of the present disclosure, showing the overall structure of the large model deployment. Figure 9 As shown in the figure, the overall steps for deploying a large model include:

[0145] Step S801, environment preparation stage;

[0146] Among them, during the environment preparation stage, this embodiment can prepare the system's built-in large model, built-in small model, distillation algorithm, running image, etc. related to the large model deployment process and upload them to the system, and then prepare the large model mapping table, small model mapping table, distillation algorithm mapping table, and algorithm configuration mapping table and upload these mapping tables to the system. At the same time, the system provides pluggable management of information such as large models, algorithms, and computing platforms, as well as the association mapping function between models, algorithms, and computing platforms. Next, this embodiment can obtain the service name, service port, service address, service protocol, health status, routing, and other information of services such as the large model management service, distillation algorithm management service, large model distillation service, model conversion service, and model deployment service in the system, and then register this service information with the registration center.

[0147] And as Figure 9 As shown in the figure, the business database (MySQL) can store business data tables, mapping tables, service information tables, and process task tables; the object storage service (OBS) can store distillation algorithm files, large model files, small model files, scenario models, DAG configuration files, and log files.

[0148] Step S8021, storing the deployment data file; wherein the Web management terminal may store the deployment data file in the data storage;

[0149] Step S8022: Submit deployment information. The Web management terminal can submit deployment information to system services, including task flow services, build configuration services, and log monitoring services.

[0150] Step S8023, storing the deployment information; wherein the system service may store the deployment information in a data storage;

[0151] Step S8024, obtaining a mapping table; wherein the system service may obtain the mapping table from the data storage;

[0152] Step S8025, obtaining service information; wherein the system service can obtain service information from the registration center;

[0153] Step S8026: Construct a directed acyclic graph configuration file; wherein the system service can construct the DAG configuration file based on the acquired information;

[0154] The specific process of the above steps S8021 to S8026 can be found in Figure 10 , Figure 10 It is a schematic diagram of the task information for constructing a directed acyclic graph provided by an embodiment of the present disclosure. Figure 10 The process includes:

[0155] Step S901, uploading the distillation dataset; this step corresponds to the above step S8021;

[0156] Step S902, deployment information; this step corresponds to the above step S8022;

[0157] Step S903, storing deployment information; this step corresponds to the above step S8023;

[0158] Step S904, obtaining a mapping table; this step corresponds to the above step S8024;

[0159] Step S905, obtaining service information; this step corresponds to the above step S8025;

[0160] Step S906, building a configuration; this step corresponds to the above step S8026;

[0161] Step S907, uploading the configuration file;

[0162] according to Figure 10 In the process of building directed acyclic graph task information, that is, building a DAG configuration file, the user submits deployment information such as downstream task information, data, computing platform, etc. through the web management terminal in steps S901 and S902. The corresponding deployment data files are uploaded to the object storage service (OBS) and the business database (MySQL) respectively.

[0163] Next, in step S903, the build configuration service determines whether the deployment process requires model conversion based on the distillation device chip type and the target device chip type in the deployment information, and obtains the PyTorch model architecture obtained by distillation and the TensorRt model architecture submitted by the user. If they are inconsistent, model conversion is required. The system has built-in conversion images for most different types. In step S904, a configuration service is then constructed to retrieve the mapping table and deployment information from the business database. In step S905, service information is obtained from the registration center, that is, the registration information of the system service. If model conversion is required, according to the dependency order of the large model deployment process, the large model management service, distillation algorithm management service, model distillation service, model conversion service, and model deployment service are used as the execution order of the DAG configuration file. Otherwise, according to the dependency order of the large model deployment process, the large model management service, distillation algorithm management service, model distillation service, and model deployment service are used as the execution order of the DAG configuration file and written into the DAG configuration file DAG.json. When executing step S906, the configuration service is constructed to write the system default automation engine default parameter default_args into the directed acyclic graph configuration file, and the generated DAG.json is uploaded to the object storage service for storage through step S907.

[0164] Step S803, scheduling service;

[0165] Step S8031, service output files, service status, and service logs; wherein the service output files, service status, and service logs generated by the system can be stored as data;

[0166] The process of scheduling services can be found in Figure 11 , Figure 11 It is a schematic diagram of the large model automatic deployment process provided by the embodiment of the present disclosure. Figure 11 The process includes:

[0167] Step S1001, loading;

[0168] Step S1002, creating a task workflow;

[0169] Step S1003, scheduling;

[0170] Step S1004, creating a task;

[0171] Step S1005, updating task information and scheduling services;

[0172] Step S1006: storing in the object storage service;

[0173] Step S1007, task retry;

[0174] Step S1008, query logs and task status;

[0175] according to Figure 11 Specifically, this embodiment implements automatic deployment of large models through an automation engine. First, in step S1001, the automation engine loads the DAG configuration file, obtains the engine initialization default parameters in the directed acyclic graph configuration file, that is, the parameters in default_args, and initializes the engine. In step S1002, the scheduler creates a task workflow according to the large model deployment process sequence in the directed acyclic graph configuration and stores it in the business database. The scheduler then queries the business database for the task workflow. In step S1003, the task nodes in the task workflow are scheduled / periodically scheduled to the task queue based on the configuration, and the task node status is updated to "waiting." In step S1004, the executor retrieves the task node information from the task queue, creates a task execution unit to run the task, such as Distillation Task A, Conversion Task B, and Deployment Task C in the figure, and updates the node status to "running." In step S1005, the execution unit schedules the corresponding service based on the service information in the task information, such as the Large Model Management Service, Distillation Algorithm Management Service, Large Model Distillation Service, Model Conversion Service, and Model Deployment Service in the figure. Based on the service execution status, the node status information is updated to "success" or "failure," and the service execution information and node status are updated to the business database. Finally, in step S1006, the execution unit stores the results of the scheduled service execution in the object storage service. Furthermore, if a task node fails to execute, step S1007 is executed, and the automation engine retries the task node based on the initialization configuration and reloads it into the queue. Furthermore, by executing step S1008, the web management terminal can access log information, status, and other information in the task workflow from the business database.

[0176] For example, if the information submitted by the user on the client page includes the large model ID, small model ID, dataset ID, version ID, distillation component configuration, distillation device chip type, target device chip type, target device model architecture, target device address, inference algorithm and other information, the yolov8_x large model can be selected as the original large model, the yolov8_l small model can be used as the lightweight large model as an example, the road disease dataset can be selected as the distillation dataset, the distillation device type is the distillation device chip type, which is PowerEdgeT640 (RTX2080Ti), and the target device type is the target device chip type, which is T506s (Jetson Xavier NX).

[0177] When executing the large model automatic deployment method described in the embodiments of the present disclosure, the large model management service first manages user-uploaded models and system-built-in models, obtaining the addresses of the large and small model files based on the model ID. The large model distillation service obtains the file addresses of the large and small models, as well as the distillation configuration file, uploads the large and small models to the parallel file system, and launches the distillation container based on the distillation run image. The model address and dataset are mounted as directories within the container, and then distillation is performed. Distillation generates a PyTorch model architecture. The service then determines whether the PyTorch model architecture is consistent with the TensorRT model architecture submitted by the user, as well as the chip type of the distillation device and the chip type of the target device. If not, model conversion is performed. The model conversion service first obtains the chip type of the distillation device (using the PowerEdge T640 (RTX2080Ti) as an example) and the original model architecture, compares them with the target device chip type (using the PowerEdge T640 (RTX2080Ti) as an example) and the target model architecture submitted by the user. Using the built-in conversion image, the service obtains the distilled model address, mounts the model address and dataset as directories within the container, and converts the model into a TensorRT architecture model. Finally, the model deployment service pushes the converted TensorRT model and inference algorithm to the target device for deployment.

[0178] During the execution of the DAG configuration file, the automation causes the DAG configuration file to be loaded, the service information set of tasks is obtained, and then the set is traversed to obtain the task_id of each service information. If the task_id is not empty, it means that the current service has been created and the next one is skipped. If it is empty, it queries whether there are dependent services. If it is empty, the scheduler puts the current service information into the execution queue and randomly generates a UUID as the task_id. If it is not empty, it queries the service status of the task_id of the dependent service. If the dependent service has not ended or failed, the current task is skipped and the next task is executed. If the dependent services have all been executed, the scheduler schedules the service information to the execution queue, thus ensuring the orderly execution of the services. The executor will create services according to the execution queue.

[0179] Step S804, log monitoring; eventually, the Web management terminal can obtain relevant logs through log monitoring.

[0180] In the above multiple processes, except for the manual selection and submission of deployment downstream task information on the Web management terminal visualization page, no further manual intervention is required. Therefore, this embodiment completes scheduling services and functional services by designing an automation engine, and the system realizes efficient and diversified one-stop large-scale model deployment.

[0181] See also Figure 12 The present disclosure also provides a large model automatic deployment device that can implement the above large model automatic deployment method. The large model automatic deployment device includes:

[0182] The information receiving module 1201 is used to obtain downstream task information input by the user terminal, and determine the original large model, the lightweight large model, the target device type of the target device, and the target model architecture that can be run on the target device based on the downstream task information;

[0183] The scheduling confirmation module 1202 is configured to generate scheduling information for a pre-registered model conversion service when at least one of the following conditions is met: the distillation device type in the process of distilling the original large model into the lightweight large model is inconsistent with the target device type; and the original model architecture of the lightweight large model is inconsistent with the target model architecture.

[0184] A directed acyclic graph construction module 1203 is configured to construct scheduling information for a model distillation service for distilling a pre-registered original large model into a lightweight large model, and to construct scheduling information for a model deployment service for a pre-registered target device, and to construct directed acyclic graph task information based on the sequence of all the scheduling information;

[0185] The automatic deployment module 1204 is used to schedule the model distillation service to distill the original large model into a lightweight large model based on various scheduling information when the directed acyclic graph task information is running, and then schedule the model conversion service to convert the distilled lightweight large model to the target device type or target model architecture, and schedule the model deployment service to deploy the converted lightweight large model to the target device.

[0186] In summary, the large model automatic deployment device can determine the original large model, lightweight large model, target device type of the target device, and the target model architecture that can be run on the target device based on the input of the user end by executing the large model automatic deployment method, and automatically generate scheduling information for the pre-registered model conversion service in order to adapt to target devices with different performance differences. Since the model distillation service, model distillation service and model deployment service are all pre-registered, no manual intervention is required when generating directed acyclic graph task information, and the execution order between each service is constrained by the directed acyclic graph task information. After executing the directed acyclic graph task information, each service can be scheduled in sequence, and the large model can be automatically distilled, converted and other operations can be performed, and the converted lightweight large model can finally be deployed to the target device, which can improve the efficiency and quality of large model deployment.

[0187] The specific implementation of the large model automatic deployment device is basically the same as the specific embodiment of the large model automatic deployment method described above, and will not be repeated here. Under the premise of meeting the requirements of the embodiment of the present disclosure, the large model automatic deployment device can also be provided with other functional modules to implement the large model automatic deployment method in the above embodiment.

[0188] The present disclosure also provides an electronic device comprising a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned large-scale model automatic deployment method. The electronic device can be any intelligent terminal, including a tablet computer and an in-vehicle computer.

[0189] See also Figure 13 , Figure 13 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0190] The processor 1301 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present disclosure.

[0191] The memory 1302 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1302 can store operating devices and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1302 and is called by the processor 1301 to execute the large model automatic deployment method of the embodiments of this disclosure;

[0192] Input / output interface 1303, used to implement information input and output;

[0193] Communication interface 1304, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0194] Bus 1305 , which transmits information between various components of the device (e.g., processor 1301 , memory 1302 , input / output interface 1303 , and communication interface 1304 );

[0195] The processor 1301 , the memory 1302 , the input / output interface 1303 and the communication interface 1304 are connected to each other in communication within the device via a bus 1305 .

[0196] The embodiment of the present disclosure also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned large model automatic deployment method.

[0197] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0198] The embodiments described in the embodiments of the present disclosure are intended to more clearly illustrate the technical solutions of the embodiments of the present disclosure and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0199] Those skilled in the art will understand that the technical solutions shown in the drawings do not constitute a limitation on the embodiments of the present disclosure, and may include more or fewer steps than shown in the drawings, or a combination of certain steps, or different steps.

[0200] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0201] Those skilled in the art will appreciate that all or some of the steps, devices, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0202] The terms "first," "second," "third," "fourth," and the like (if any) in the specification of the present disclosure and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in orders other than those illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, apparatus, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0203] It should be understood that in the present disclosure, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0204] In the several embodiments provided in the present disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0205] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0206] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0207] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method of each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store programs.

[0208] The preferred embodiments of the present disclosure are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present disclosure. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present disclosure should be within the scope of the present disclosure.

Claims

1. A large model automatic deployment method, characterized in that: include: Acquire downstream task information input by the user end, and determine the original large model, the lightweight large model, the target device type of the target device, and the target model architecture that can be run on the target device based on the downstream task information; When at least one of the following conditions is met: the distillation device type in the process of distilling the original large model into the lightweight large model is inconsistent with the target device type, and the original model architecture of the lightweight large model is inconsistent with the target model architecture, generating scheduling information for a pre-registered model conversion service; Constructing scheduling information for a model distillation service for distilling the pre-registered original large model into the lightweight large model, and constructing scheduling information for a model deployment service for the pre-registered target device, and sequentially configuring scheduling dependencies corresponding to adjacent services between each of the scheduling information of the model distillation service, the model conversion service, and the model deployment service, and constructing directed acyclic graph task information based on the order of all the scheduling information and the corresponding scheduling dependencies; When the directed acyclic graph task information is running, based on each of the scheduling information, the model distillation service is first scheduled to distill the original large model into the lightweight large model, and then the model conversion service is scheduled to convert the distilled lightweight large model to the target device type or the target model architecture, and the model deployment service is scheduled to deploy the converted lightweight large model to the target device; Wherein, when the directed acyclic graph task information is running, based on each of the scheduling information, the model distillation service is first scheduled to distill the original large model into the lightweight large model, and then the model conversion service is scheduled to convert the distilled lightweight large model to the target device type or the target model architecture, and the model deployment service is scheduled to deploy the converted lightweight large model to the target device, including: When the directed acyclic graph task information is running, the model distillation service, the model conversion service, and the model deployment service are sequentially scheduled based on the respective scheduling information; When running the model distillation service, distilling the original large model into the lightweight large model; When the model conversion service is running, determining whether the model distillation service it depends on is completed, and after the model distillation service is completed, scheduling the model conversion service to convert the distilled lightweight large model to the target device type or the target model architecture; When the model deployment service is running, it is determined whether the model conversion service it depends on is completed, and after the model conversion service is completed, the model deployment service is scheduled to deploy the converted lightweight large model to the target device.

2. The large model automatic deployment method according to claim 1, characterized in that: Scheduling the model distillation service to distill the original large model into the lightweight large model, scheduling the model conversion service to convert the distilled lightweight large model to the target device type or the target model architecture, and scheduling the model deployment service to deploy the converted lightweight large model to the target device, includes: After scheduling the model distillation service, a distillation container is created, the original large model and the lightweight large model are mounted on the distillation container, and the original large model is distilled into the lightweight large model within the distillation container; After scheduling the model conversion service, a conversion image is built into the distillation container, and the distilled lightweight large model is converted to the target device type or the target model architecture through the conversion image; After scheduling the model deployment service, the lightweight large model converted in the distillation container is deployed to the target device.

3. The large model automatic deployment method according to claim 1, characterized in that: The determining, based on the downstream task information, the original large model, the lightweight large model, the target device type of the target device, and the target model architecture that can be run on the target device, includes: Extracting a target device and a task description to be executed in the target device from the downstream task information; Obtaining a device description of the target device, a target device type, and a target model architecture that can be run on the target device from a preset database; Extracting a first text feature of the task description and a second text feature of the device description, and fusing the first text feature and the second text feature to obtain a fused feature; Based on the fusion features, a plurality of preset large models to be selected are predicted to obtain prediction results, and based on the prediction results, an original large model and a lightweight large model are selected from the plurality of large models to be selected.

4. The large model automatic deployment method according to claim 3, characterized in that: The step of predicting a plurality of preset large models to be selected based on the fusion features to obtain prediction results, and selecting an original large model and a lightweight large model from the plurality of large models to be selected based on the prediction results, includes: Inputting the fusion features into a preset first prediction model, predicting a large model that meets the requirements of the task description among multiple large models to be selected, obtaining a first prediction result, and selecting an original large model from the multiple large models to be selected based on the first prediction result; The fusion features are input into a preset second prediction model, and a large model that meets the device description requirements of multiple large models to be selected is predicted to obtain a second prediction result, and a lightweight large model is selected from the multiple large models to be selected based on the second prediction result.

5. The large model automatic deployment method according to claim 1, characterized in that: Scheduling the model distillation service to distill the original large model into the lightweight large model includes: Determining a distillation dataset based on the downstream task information; The model distillation service is scheduled, and the original large model is distilled into the lightweight large model using the distillation dataset.

6. The large model automatic deployment method according to claim 1, characterized in that: The large model automatic deployment method further includes: When the distillation device type in the process of distilling the original large model into the lightweight large model is consistent with the target device type, and the original model architecture of the lightweight large model is consistent with the target model architecture, constructing directed acyclic graph task information based on the scheduling information of the model distillation service and the scheduling information of the model deployment service; When the directed acyclic graph task information is running, the model distillation service is first scheduled to distill the original large model into the lightweight large model based on each scheduling information, and the model deployment service is scheduled to deploy the distilled lightweight large model to the target device.

7. A large model automatic deployment device, characterized in that: include: An information receiving module is used to obtain downstream task information input by a user terminal, and determine the original large model, the lightweight large model, the target device type of the target device, and the target model architecture that can be run on the target device based on the downstream task information; a scheduling confirmation module, configured to generate scheduling information for a pre-registered model conversion service when at least one of the following conditions is met: a distillation device type in the process of distilling the original large model into the lightweight large model is inconsistent with the target device type, and an original model architecture of the lightweight large model is inconsistent with the target model architecture; A directed acyclic graph construction module is used to construct scheduling information for a model distillation service for distilling the pre-registered original large model into the lightweight large model, and to construct scheduling information for a model deployment service for the pre-registered target device, and to sequentially configure scheduling dependencies corresponding to adjacent services between each piece of scheduling information for the model distillation service, the model conversion service, and the model deployment service, and to construct directed acyclic graph task information based on the order of all the scheduling information and the corresponding scheduling dependencies; an automatic deployment module, configured to, when the directed acyclic graph task information is running, first schedule the model distillation service to distill the original large model into the lightweight large model based on each of the scheduling information, then schedule the model conversion service to convert the distilled lightweight large model to the target device type or the target model architecture, and then schedule the model deployment service to deploy the converted lightweight large model to the target device; Wherein, when the directed acyclic graph task information is running, based on each of the scheduling information, the model distillation service is first scheduled to distill the original large model into the lightweight large model, and then the model conversion service is scheduled to convert the distilled lightweight large model to the target device type or the target model architecture, and the model deployment service is scheduled to deploy the converted lightweight large model to the target device, including: When the directed acyclic graph task information is running, the model distillation service, the model conversion service, and the model deployment service are sequentially scheduled based on the respective scheduling information; When running the model distillation service, distilling the original large model into the lightweight large model; When the model conversion service is running, determining whether the model distillation service it depends on is completed, and after the model distillation service is completed, scheduling the model conversion service to convert the distilled lightweight large model to the target device type or the target model architecture; When the model deployment service is running, it is determined whether the model conversion service it depends on is completed, and after the model conversion service is completed, the model deployment service is scheduled to deploy the converted lightweight large model to the target device.

8. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the large model automatic deployment method according to any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the large model automatic deployment method according to any one of claims 1 to 6 is implemented.

Images