Large model deployment method and device, equipment, storage medium and program product

Abstract

The invention provides a large model deployment method and device, equipment, a storage medium and a program product, and relates to the technical field of large model deployment. The method comprises the following steps: a cloud server receives current operation scene information and current resource use information sent by a terminal; determining whether the current operation scene information and the current resource use information are adaptive to the specification of a current large model of the terminal; if not, based on the current operation scene information and the current resource use information, a multi-target constraint and optimization method is adopted to screen and obtain a target large model from the model library, and the multi-target constraint and optimization method is used for determining a constraint condition and an optimization target of large model screening; and sending the model information of the target large model to the terminal, thereby enabling the terminal to deploy the target large model according to the model information, and carrying out reasoning through the target large model. According to the method, on-demand dynamic adjustment of the large model can be realized, and the obtained target large model can better adapt to the requirements of complex and changeable scenes.

Description

[0001] The present application claims priority to the Chinese patent application No. 202510526965.2, filed on April 25, 2025, and entitled "Large Model Deployment Method, Device, Equipment, Storage Medium and Program Product", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] The present application relates to the technical field of large model deployment, and particularly relates to a large model deployment method, device, equipment, storage medium and program product. BACKGROUND

[0003] With the rapid development of artificial intelligence technology, large models rely on their powerful feature extraction and deep semantic understanding capabilities, and exhibit excellent performance in traffic flow prediction, real-time dynamic scheduling, anomaly detection and other tasks, and are increasingly widely used. The deployment of large models is a key factor affecting the landing of large models.

[0004] Currently, when deploying a large model, the deployment of the large model is usually completed by adjusting the deployment strategy of the large model, and then the large model is used for inference. However, in some scenarios, the large model deployed by the above-mentioned method cannot adapt to the needs of complex and variable scenarios. SUMMARY

[0005] The purpose of the present application is to provide a large model deployment method, device, equipment, storage medium and program product to solve the problem that the large model deployed by the current method cannot adapt to the needs of complex and variable scenarios in some scenarios.

[0006] In order to achieve the above-mentioned purpose, the technical solution adopted by the present application is as follows:

[0007] A large model deployment method applied to a cloud server, the large model deployment method comprising: receiving current running scenario information and current resource usage information sent by a terminal; determining whether the current running scenario information and the current resource usage information are suitable for the specification of a current large model of the terminal; if not, based on the current running scenario information and the current resource usage information, a multi-objective constraint and optimization method is used to screen a target large model from a model library, the multi-objective constraint and optimization method being used to determine constraint conditions and optimization objectives of large model screening; sending model information of the target large model to the terminal, so that the terminal deploys the target large model according to the model information, and performs inference through the target large model.

[0008] According to the above technical means, based on the real-time perception and adaptive optimization framework, according to the changes of the running scene information and the resource use information of the terminal, a target large model that can better meet the needs of complex and changeable scenes and can reduce unnecessary model information is obtained by using a multi-objective constraint and optimization method from the model library, and the power and energy consumption of the terminal are effectively reduced.

[0009] Further, determining whether the current running scene information and the current resource use information adapt to the specification of the current large model of the terminal comprises: if the fluctuation of the current resource use information is greater than a fluctuation threshold, and / or the current event type determined according to the current running scene information is different from the event type of the terminal at the last time, it is determined that the current running scene information and the current resource use information do not adapt to the specification of the current large model of the terminal; if the fluctuation of the current resource use information is less than or equal to the fluctuation threshold, and the current event type determined according to the current running scene information is the same as the event type of the terminal at the last time, it is determined that the current running scene information and the current resource use information adapt to the specification of the current large model of the terminal.

[0010] Further, based on the current running scene information and the current resource use information, a target large model is obtained from the model library by using a multi-objective constraint and optimization method, comprising: determining a current event type according to the current running scene information; obtaining a plurality of candidate large models from the model library by using a multi-objective constraint and optimization method, wherein the multi-objective constraint and optimization method takes the use boundary values of different resources of the terminal as constraint conditions, and takes precision, real-time performance and computing power as optimization objectives; obtaining a target large model from the plurality of candidate large models based on a preset weight corresponding to the current event type.

[0011] Further, the preset weight comprises a first weight corresponding to the precision, a second weight corresponding to the real-time performance, and a third weight corresponding to the computing power, and the target large model is obtained from the plurality of candidate large models based on the preset weight corresponding to the current event type, comprising: for each candidate large model in the plurality of candidate large models, a score corresponding to the candidate large model is obtained by weighted summation according to the candidate precision, the candidate real-time performance, the candidate computing power, the first weight, the second weight and the third weight of the candidate large model; and determining the target large model as the candidate large model corresponding to the highest score.

[0012] Further, the model information of the target large model is sent to the terminal, comprising: obtaining difference model information between the target large model and the current large model; packing and compressing the difference model information to obtain packed and compressed difference model information; and sending the packed and compressed difference model information to the terminal.

[0013] A large model deployment method applied to a terminal, the large model deployment method comprising: obtaining current running scenario information and current resource usage information; sending the current running scenario information and the current resource usage information to a cloud server; receiving model information of a target large model sent by the cloud server, the target large model being obtained from a model library by the cloud server based on the current running scenario information and the current resource usage information using a multi-objective constraint and optimization method in a case where it is determined that the current running scenario information and the current resource usage information are not suitable for a specification of a current large model of the terminal, the multi-objective constraint and optimization method being used to determine constraint conditions and optimization objectives for large model screening; deploying the target large model according to the model information; and performing reasoning by the target large model.

[0014] Further, the target large model is deployed according to the model information, comprising: deploying the target large model using a hot update and gradual switching method according to the model information.

[0015] Further, the target large model is deployed using a hot update and gradual switching method according to the model information, comprising: maintaining running of the current large model during deployment of the target large model, wherein the model information of the target large model is loaded into an idle memory area in batches according to a preset priority by a memory paging mapping method; input data of the terminal is allocated to the target large model according to a preset incremental ratio until the input data is all migrated to the large model; and a lazy resource release mechanism is used to unload the current large model.

[0016] Further, the model information is packed and compressed difference model information between the target large model and the current large model, and the target large model is deployed according to the model information, comprising: decompressing the packed and compressed difference model information to obtain decompressed difference model information; and deploying the target large model according to the decompressed difference model information.

[0017] A large model deployment apparatus applied to a cloud server, the large model deployment apparatus comprising: a receiving module configured to receive current running scenario information and current resource usage information sent by a terminal; a determining module configured to determine whether the current running scenario information and the current resource usage information are suitable for a specification of a current large model of the terminal; a obtaining module configured to, if not, obtain a target large model from a model library based on the current running scenario information and the current resource usage information using a multi-objective constraint and optimization method, the multi-objective constraint and optimization method being used to determine constraint conditions and optimization objectives for large model screening; and a sending module configured to send model information of the target large model to the terminal, so that the terminal deploys the target large model according to the model information and performs reasoning by the target large model.

[0018] Further, the determining module is specifically configured to: if the fluctuation of the current resource usage information is greater than the fluctuation threshold, and / or the current event type determined according to the current running scene information is different from the event type of the terminal at the last moment, it is determined that the current running scene information and the current resource usage information do not adapt to the specification of the current large model of the terminal; if the fluctuation of the current resource usage information is less than or equal to the fluctuation threshold, and the current event type determined according to the current running scene information is the same as the event type of the terminal at the last moment, it is determined that the current running scene information and the current resource usage information adapt to the specification of the current large model of the terminal.

[0019] Further, the obtaining module is specifically configured to: determine a current event type according to the current running scene information; use a multi-objective constraint and optimization method to screen a plurality of candidate large models from a model library, the multi-objective constraint and optimization method taking the use boundary values of different resources of the terminal as constraint conditions and taking precision, real-time performance and computing power as optimization targets; and obtain a target large model from the plurality of candidate large models based on a preset weight corresponding to the current event type.

[0020] Further, the preset weight includes a first weight corresponding to the precision, a second weight corresponding to the real-time performance, and a third weight corresponding to the computing power, and the obtaining module, when used to obtain the target large model from the plurality of candidate large models based on the preset weight corresponding to the current event type, is specifically configured to: for each candidate large model in the plurality of candidate large models, perform weighted summation on the candidate precision, the candidate real-time performance, the candidate computing power, the first weight, the second weight and the third weight corresponding to the candidate large model to obtain a score corresponding to the candidate large model; and determine the target large model as the candidate large model corresponding to the highest score.

[0021] Further, the sending module is specifically configured to: obtain difference model information between the target large model and the current large model; package and compress the difference model information to obtain packaged and compressed difference model information; and send the packaged and compressed difference model information to the terminal.

[0022] A large model deployment apparatus applied to a terminal, comprising: an obtaining module configured to obtain current running scene information and current resource usage information; a sending module configured to send the current running scene information and the current resource usage information to a cloud server; a receiving module configured to receive model information of a target large model sent by the cloud server, the target large model being obtained by the cloud server based on the current running scene information and the current resource usage information using a multi-objective constraint and optimization method from a model library in a case where it is determined that the current running scene information and the current resource usage information do not adapt to the specification of a current large model of the terminal, the multi-objective constraint and optimization method being used to determine constraint conditions and optimization targets for large model screening; a deployment module configured to deploy the target large model according to the model information; and an inference module configured to perform inference through the target large model.

[0023] Further, the deployment module is specifically configured to: according to the model information, deploy the target large model by using a hot update and a gradual switching method.

[0024] Further, when the deployment module is used to deploy the target large model by using the hot update and the gradual switching method according to the model information, the deployment module is specifically configured to: maintain the running of the current large model during the deployment of the target large model, wherein the model information of the target large model is loaded into an idle memory area in batches according to a preset priority by using a memory paging mapping method; input data of the terminal is allocated to the target large model according to a preset incremental proportion until the input data is all migrated to the large model; and the current large model is unloaded by using a lazy resource release mechanism.

[0025] Further, the model information is packaged and compressed difference model information between the target large model and the current large model, and the deployment module is specifically configured to: decompress the packaged and compressed difference model information to obtain decompressed difference model information; and deploy the target large model according to the decompressed difference model information.

[0026] A large model deployment system, comprising: a cloud server and a terminal; the cloud server is configured to execute the large model deployment method of the cloud server as described above; and the terminal is configured to execute the large model deployment method of the terminal as described above.

[0027] An electronic device, comprising: a processor, and a memory connected with the processor in communication; the memory stores computer execution instructions; and the processor executes the computer execution instructions stored in the memory to implement the large model deployment method as described above.

[0028] A computer readable storage medium, the computer readable storage medium stores computer program instructions, and the computer program instructions are executed to implement the large model deployment method as described above.

[0029] A computer program product, comprising a computer program, the computer program is executed to implement the large model deployment method as described above.

[0030] The application provides a large model deployment method, device, equipment, storage medium and program product. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0032] Figure 1 An application scenario schematic diagram provided by an embodiment of the present application;

[0033] Figure 2 A large model deployment method provided by an embodiment of the present application Figure 1 ;

[0034] Figure 3 A large model deployment method provided by an embodiment of the present application Figure 2 ;

[0035] Figure 4 A large model deployment method provided by an embodiment of the present application Figure 3 ;

[0036] Figure 4 A large model deployment method provided by an embodiment of the present application Figure 6 ;

[0037] Figure 5 A large model deployment method provided by an embodiment of the present application Figure 7 ;

[0038] Figure 8 A structural schematic diagram of a large model deployment device according to an embodiment of the present application is provided.

[0039] Figure 9 A structural schematic diagram of a large model deployment device according to another embodiment of the present application is provided.

[0040] Figure 10 A schematic diagram of a large model deployment system according to an embodiment of the present application is provided.

[0041] Figure 11 A schematic diagram of a large model deployment system according to another embodiment of the present application is provided.

[0042] Figure 1 A structural schematic diagram of an electronic device according to an embodiment of the present application is provided. DETAILED DESCRIPTION

[0043] Other advantages and effects of the present application can be easily understood by those skilled in the art from the above description of the embodiments of the present application. The present application can also be implemented or applied in other different specific embodiments, and various modifications or changes can be made to the details of the present application based on different views and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, and are not intended to limit the protection scope of the present application.

[0044] It should be noted that the diagrams provided in the following embodiments only schematically illustrate the basic concept of the present application, and only show the components related to the present application in the diagrams, rather than being drawn according to the number, shape and size of the components in actual implementation. The shape, number and proportion of each component in actual implementation can be arbitrarily changed, and the layout pattern of the components can also be more complex.

[0045] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of the relevant data need to comply with relevant laws, regulations and standards, and provide corresponding operation entrances for the user to choose authorization or refusal.

[0046] In recent years, with the rapid development and wide application of intelligent systems, the demand for big data processing in complex scenarios has shown exponential growth. Large models rely on their powerful feature extraction and deep semantic understanding capabilities, and have shown excellent performance in tasks such as traffic flow prediction, real-time dynamic scheduling, and anomaly detection. They have been increasingly widely applied and have become an indispensable technical support in the field of intelligent systems. The deployment of large models is a key factor affecting the landing of large models.

[0047] Currently, when deploying large models, the deployment strategy of the large model is usually adjusted to complete the deployment of the large model, and then the large model is used for inference. However, in some scenarios, the large model deployed by the above method cannot adapt to the needs of complex and variable scenarios. Specifically, in practical applications, there are complex and variable application scenarios such as urban traffic, intelligent manufacturing, and smart city management. The complex and variable scenario refers to an application environment that is highly dynamic and has conflicting multi-dimensional demands. Its core characteristics are: key parameters such as network bandwidth and task load in the scene will have large fluctuations; real-time, high-precision, and low-energy consumption conflictive goals need to be met, and the demand priority will be dynamically adjusted with the change of the scene; in addition, the terminal is limited by the significant difference and continuous fluctuation of computing power and network resources, and needs to maximize the efficiency of the large model under resource-limited conditions through dynamic adaptation. The existing large model deployment method usually only adjusts the deployment strategy to improve the utilization efficiency of computing resources, which is a static deployment of the large model. When the environment and demand change, it cannot adaptively adjust the large model in a timely manner, making it difficult to adapt to the complex and dynamic environment caused by changes in demand in complex and variable scenarios.

[0048] The inventors found that the current large model deployment method for complex and variable scenarios has significant limitations, especially in how to effectively combine the computing power of the terminal and the scene demand to adjust the specifications of the large model in a dynamic environment. In practical applications, such as the field of intelligent systems such as smart traffic, there is an urgent need for a dynamic optimization deployment method that can flexibly adjust the specifications of the large model according to the device computing power, communication delay fluctuation, and real-time scene demand. This deployment method should be able to save computing resources to the maximum extent while ensuring scene demand, and improve the adaptability and robustness of the large model deployment system in complex and variable environments.

[0049] Based on the above problems, the application provides a large model deployment method, which flexibly selects different specifications of large models from a model library by using a multi-objective constraint and optimization method according to changes in running scene information and resource use information of a terminal through a cloud server, the large model can preferentially meet the core scene demand, the large model is sent to the terminal for deployment and used for reasoning, thereby realizing on-demand dynamic adjustment of the large model, the obtained target large model can better adapt to the demand of a complex and changeable scene, and unnecessary model information can be reduced, thereby effectively reducing the computing power and energy consumption input of the terminal. In addition, the application introduces an instant perception and feedback mechanism for environmental and resource data, so that the large model deployment system can respond to changes in running scene information and resource use information in a timely manner, and when the large model switching is triggered, the hot update and gradual switching method is used to effectively reduce the task interruption time caused by large model switching, and the continuity of the terminal task is ensured.

[0050] Hereinafter, first, the application scenario of the scheme provided by the application is exemplarily described.

[0051] Figure 1 The application scenario schematic diagram provided by an embodiment of the application is shown in FIG. 1. Figure 1 As shown in the figure, the application scenario can include a cloud server cluster 11 and a terminal 12; wherein the cloud server cluster 11 includes a plurality of cloud servers 111 and a storage 112, and the terminal 12 can be a vehicle-mounted intelligent device (such as a smart cockpit of a vehicle), a mobile phone, a tablet computer, a notebook computer, a desktop computer, etc. The cloud server 111 receives the current running scene information and the current resource use information sent by the terminal 12; the cloud server 111 determines whether the current running scene information and the current resource use information are not suitable for the specification of the current large model of the terminal 12, and in the case of determining that the current running scene information and the current resource use information are not suitable for the specification of the current large model of the terminal 12, the cloud server 111 selects a target large model with the most suitable specification for the terminal 12 from the model library according to the large model deployment method provided by the embodiment of the application, and sends the target large model to the terminal 12. Wherein the model library provides a plurality of specifications of trained large models. The terminal 12 deploys the target large model and performs reasoning through the target large model. Wherein the cloud server 111 obtains related data from the storage 112, and stores the generated data in the storage 112. In addition, the cloud server 111 and the terminal 12 communicate through a wireless network or a wired network.

[0052] It should be noted that, Figure 1 is only a schematic diagram of an application scenario provided by the embodiment of the application, the embodiment of the application does not limit the devices included in Figure 1 , nor does it limit the positional relationship between the devices in Figure 2 .

[0053] The technical solutions of the present application will be described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments.

[0054] Figure 1 The large model deployment method provided for the embodiments of the present application is shown in the figure Figure 2 . The large model deployment method can be executed by software and / or hardware devices, for example, the hardware device can be a large model deployment device, which can be an electronic device or a processing chip in an electronic device. As Figure 3 shown, the method of the embodiments of the present application includes:

[0055] S201, the terminal acquires current running scene information and current resource usage information.

[0056] In the embodiments of the present application, the terminal is an intelligent computing node covering various hardware conditions and functional roles, for example, a signal controller capable of autonomous perception and a terminal with basic analysis capability. The terminal can run data acquisition and large model inference tasks, such as continuously obtaining rich and multi-dimensional current running scene information (data) through built-in cameras, radars, laser range finders, various sensors, etc., to provide timely and reliable information basis for subsequent selection and deployment decisions of large models.

[0057] For example, the terminal can acquire current running scene information and current resource usage information of the terminal in real time. Specifically, the terminal includes a scene and device perception module, which includes an information acquisition unit and a resource monitoring unit. The information acquisition unit is used to acquire the current running scene information of the terminal, such as environmental characteristics and scene complexity, to provide a basis for the cloud server to judge the current running scene demand. Taking the terminal as a vehicle-mounted intelligent device as an example, the current running scene information acquired by the terminal can include environmental characteristics around the vehicle, such as traffic density, crowd density, and sudden events. The resource monitoring unit is used to dynamically evaluate the current resource usage information of the terminal, such as the utilization rate of computing power, the storage capacity, the available network bandwidth, etc., so that the subsequent cloud server decision process can accurately match to obtain a large model more suitable for the terminal. The current resource usage information acquired by the terminal can include CPU usage, storage occupancy, storage usage, available bandwidth, network throughput, etc.

[0058] S202, the terminal sends the current running scene information and the current resource usage information to the cloud server.

[0059] Correspondingly, the cloud server receives the current running scene information and the current resource usage information sent by the terminal.

[0060]

[0060] Exemplarily, the terminal can periodically send the current running scenario information and the current resource usage information to the cloud server.

[0061] S203, the cloud server determines whether the current running scenario information and the current resource usage information adapt to the specification of the current large model of the terminal.

[0062] In this step, after receiving the current running scenario information and the current resource usage information sent by the terminal, the cloud server can determine whether the running scenario demand changes according to the current running scenario information, and determine whether the resources of the terminal fluctuate according to the current resource usage information. For example, taking the terminal as a vehicle-mounted intelligent device, the cloud server can determine whether the running scenario demand of the terminal changes according to the current running scenario information sent by the terminal, judging the different demand degrees of the prediction accuracy, faster response speed, or specific task ability (such as anomaly detection, trajectory tracking, event recognition, etc.) of the current task of the terminal through the current large model of the terminal. Among them, the current task of the terminal is, for example, high-precision three-dimensional scene reconstruction through the current large model of the terminal. If the running scenario demand changes and / or the resources of the terminal fluctuate, it can be determined that the current running scenario information and the current resource usage information do not adapt to the specification of the current large model of the terminal. It can be understood that large models of different specifications have different focuses in terms of parameter size, calculation complexity, accuracy and target function. For how the cloud server specifically determines whether the current running scenario information and the current resource usage information adapt to the specification of the current large model of the terminal, reference can be made to subsequent embodiments.

[0063] S204, if the current running scenario information and the current resource usage information do not adapt to the specification of the current large model of the terminal, the cloud server obtains a target large model from the model library based on the current running scenario information and the current resource usage information, and a multi-objective constraint and optimization method is used to determine the constraint conditions and optimization targets of large model screening.

[0064] In this step, the multi-objective constraint and optimization method is used to determine the constraint conditions and optimization targets of large model screening, which can be referred to in subsequent embodiments. The model library pre-stores a plurality of multi-specification large models that have been fully trained and optimized, and these large models of different specifications have different focuses in terms of parameter size, calculation complexity, accuracy and target function. Specific large models may, for example, include high-specification large models, medium-specification large models, and lightweight large models after being trimmed and quantized. The model library can support hierarchical management and continuous iterative update, for example, high-specification large models can be stored in the cloud server, and small models with key functions can be stored in the terminal, so as to be quickly called in emergency scenarios (such as scenarios with too high network delay). The above multi-level and extensible model library (model resource pool) lays a foundation for dynamically selecting and distributing the optimal large model.

[0065] Exemplarily, in a case where it is determined that the current running scene information and the current resource usage information do not adapt to the specification of the current large model of the terminal, the cloud server can analyze the current running scene information and the current resource usage information, determine the demand focus (such as improving prediction accuracy or reducing delay, etc.) of the large model, adopt a multi-objective constraint and optimization method, comprehensively consider the demand focus such as accuracy requirement, delay index, energy consumption limit and bandwidth constraint, and select the most suitable large model for the terminal under the multi-constraint conditions and optimization targets of accuracy, delay, energy consumption, bandwidth, etc. from the model library, so as to obtain the target large model.

[0066] It can be understood that the obtained target large model is most suitable for the current running scene information and the current resource usage information of the terminal in terms of parameter scale, model architecture and performance index. Compared with a single-dimensional screening strategy, the embodiment of the application introduces a multi-objective constraint and optimization method, and considers network bandwidth, delay, energy consumption and scene complexity in comprehensive consideration, which can make the selection of the large model not only meet the accuracy and delay requirements, but also preferentially select the large model after compression and quantization when the bandwidth is limited or the resources are scarce, so as to maximize the resource utilization. For example, in the case of high real-time demand, a small and responsive large model can be selected; in the case of low energy consumption requirement, a compact large model after parameter quantization and pruning can be selected, which better adapts to the current running scene information and the current resource usage information of the terminal. Among them, pruning can eliminate redundant parameters of the large model, reduce the size of the large model, and quantization can convert floating-point type parameters to low bit rate representation, so as to maintain relatively satisfactory performance under the constraints of bandwidth, computing power and storage. By introducing model pruning and quantization technology, it is helpful to deeply compress the model structure of the large model, simplify the parameters of the large model and reduce redundant calculation paths, and reduce storage and computing load without significantly losing prediction accuracy.

[0067] S205, the cloud server sends the model information of the target large model to the terminal.

[0068] Correspondingly, the terminal receives the model information of the target large model sent by the cloud server.

[0069] In this step, after the cloud server obtains the target large model from the model library, the cloud server can obtain the model of the target large model, and send the model information of the target large model to the terminal according to the network condition and the transmission protocol characteristics. The model information can include the model parameter and the model structure information of the target large model. For example, the cloud server can obtain the difference model information between the target large model and the current large model, and send the difference model information to the terminal after being packaged and compressed according to the network condition and the transmission protocol characteristics, to realize the bandwidth adaptive transmission strategy and ensure that the difference model information of the large model can be smoothly delivered in a congested or unstable network environment. For how the cloud server sends the model information of the target large model to the terminal, refer to the subsequent embodiments.

[0070] S206, the terminal deploys the target large model according to the model information of the target large model.

[0071] In this step, after receiving the model information of the target large model sent by the cloud server, the terminal automatically initializes according to the model information of the target large model, and completes the deployment of the target large model. Optionally, in order to reduce the pause and performance flash of the large model replacement, the terminal can use the hot update and gradual switching method to deploy the target large model, so that the target large model can smoothly take over the task of the terminal and avoid sudden interruption of the decision process. For how the terminal deploys the target large model by using the hot update and gradual switching method, refer to the subsequent embodiments.

[0072] S207, the terminal performs reasoning through the target large model.

[0073] In this step, after the terminal completes the deployment of the target large model, the terminal can perform reasoning and analysis through the target large model. For example, the terminal can perform scene analysis, prediction and decision through the large model, and the result can be used for upper-layer business decision such as anomaly detection and path planning. For example, when the terminal is a vehicle-mounted intelligent device, the terminal can perform anomaly detection, traffic prediction and path planning through the target large model, and obtain real-time traffic prediction, anomaly detection or path decision result.

[0074] It can be understood that in the embodiment of the application, after the terminal initially deploys the large model, based on the above steps S201 to S207, the cloud server continuously detects and analyzes the dynamic changes of the scene demand and the resources according to the current running scene information and the current resource use information sent by the terminal, that is, closely follows the dynamic changes of the scene demand and the resources, and when it is found that the current large model of the terminal is no longer adapted to the actual demand of the terminal, the large model selection and deployment cycle is triggered to ensure that the terminal is continuously in a high adaptation state with the large model, so that the terminal maintains the optimal working state. Through the above steps S201 to S207, the embodiment of the application can realize dynamic allocation and iterative optimization of the large model, and ensure that no matter how the scene demand changes, the large model consistent with the scene demand and the resource condition can be provided for the terminal in time. Compared with the single static large model deployment scheme in the prior art, the embodiment of the application can effectively reduce the power consumption required for the original inference effect, such as reducing the power consumption by 25%.

[0075] The large model deployment method provided by the embodiment of the application receives the current running scene information and the current resource use information of the terminal sent by the terminal through the cloud server, determines whether the current running scene information and the current resource use information adapt to the specification of the current large model of the terminal, if not, the cloud server obtains a target large model from the model library based on the current running scene information and the current resource use information by using a multi-objective constraint and optimization method, the multi-objective constraint and optimization method is used to determine the constraint condition and the optimization target of the large model screening, and sends the model information of the target large model to the terminal; the terminal deploys the target large model according to the model information of the target large model, and performs inference through the target large model. The embodiment of the application is based on a real-time perception and adaptive optimization framework, and according to the changes of the running scene information and the resource use information of the terminal, a target large model that can better meet the core scene demand is obtained from the model library by using a multi-objective constraint and optimization method, the on-demand dynamic adjustment of the large model is realized, the obtained target large model can better adapt to the demand of a complex and changeable scene, and unnecessary model information can be reduced, thereby effectively reducing the power and energy consumption input of the terminal.

[0076] On the basis of the above embodiment, Figure 2 The large model deployment method provided by the embodiment of the application is shown Figure 3 , which is applied to a cloud server. As shown in Figure 4 , the method of the embodiment of the application comprises:

[0077] S301, receiving the current running scene information and the current resource use information sent by the terminal.

[0078] Exemplarily, the terminal can acquire current running scene information and current resource usage information of the terminal in real time, and send the current running scene information and the current resource usage information to the cloud server. Taking the terminal as a vehicle-mounted intelligent device as an example, the current running scene information acquired by the terminal can include, for example, environmental characteristics around the vehicle, such as traffic flow density, crowd density, and sudden events, and the like; and the current resource usage information acquired by the terminal can include, for example, CPU usage, storage occupancy, storage usage, available bandwidth, network throughput, and the like. Correspondingly, the cloud receives the current running scene information and the current resource usage information sent by the terminal.

[0079] S302, determining whether the current running scene information and the current resource usage information are suitable for the specification of the current large model of the terminal.

[0080] Exemplarily, large models of different specifications have different focuses in terms of parameter scale, calculation complexity, accuracy, and target functions. After receiving the current running scene information and the current resource usage information sent by the terminal, the cloud server can determine whether the running scene demand has changed according to the current running scene information, and determine whether the resources of the terminal have fluctuated according to the current resource usage information. For example, taking the terminal as a vehicle-mounted intelligent device as an example, the cloud server can determine whether the running scene demand of the terminal has changed according to the current running scene information sent by the terminal, judging the different demand degrees of the prediction accuracy, faster response speed, or specific task ability (such as anomaly detection, trajectory tracking, event recognition, etc.) of the current task of the terminal to the current large model of the terminal, and then determining whether the running scene demand of the terminal has changed. The current task of the terminal is, for example, high-precision three-dimensional scene reconstruction by the current large model of the terminal. If the running scene demand has changed and / or the resources of the terminal have fluctuated, it can be determined that the current running scene information and the current resource usage information are not suitable for the specification of the current large model of the terminal.

[0081] S303, if not, based on the current running scene information and the current resource usage information, a target large model is obtained by screening from the model library using a multi-objective constraint and optimization method, and the multi-objective constraint and optimization method is used to determine the constraint conditions and optimization targets of the large model screening.

[0082] Exemplarily, in the case where it is determined that the current running scene information and the current resource usage information are not suitable for the specification of the current large model of the terminal, the cloud server can analyze the current running scene information and the current resource usage information, determine the demand focus of the large model (such as improving prediction accuracy or reducing delay, etc.), use a multi-objective constraint and optimization method, and comprehensively consider the demand focus of accuracy requirement, delay index, energy consumption limit, and bandwidth constraint, to screen the most suitable large model for the terminal under multiple constraint conditions and optimization targets of accuracy, delay, energy consumption, bandwidth, and the like, from the model library, thereby obtaining a target large model.

[0083] S304, send the model information of the target large model to the terminal, so that the terminal deploys the target large model according to the model information, and performs reasoning through the target large model.

[0084] In this step, after the cloud server obtains the target large model by screening from the model library, the model information of the target large model can be obtained, and the model information of the target large model is sent to the terminal according to the network condition and the transmission protocol characteristics, so that the terminal deploys the target large model according to the model information, and performs reasoning through the target large model. The model information of the target large model can include the model parameter and the model structure information of the target large model.

[0085] The large model deployment method provided by the embodiment of the application comprises the following steps:

[0086] Figure 3 The large model deployment method provided by the embodiment of the application is applied to a cloud server. Figure 3 The large model deployment method provided by the embodiment of the application is applied to a cloud server. Figure 3 The method provided by the embodiment of the application can comprise the following steps:

[0087] S401, receiving the current running scene information and the current resource usage information sent by the terminal.

[0088] The specific description of this step can be referred to the related description of S301 in the embodiment shown in Figure 3 The specific description of this step can be referred to the related description of S301 in the embodiment shown in

[0089] In the embodiment of the application, Figure 5 The S302 step in the embodiment can further comprise the following two steps of S402 and S403:

[0090] S402, if the fluctuation of the current resource usage information is greater than the fluctuation threshold value, and / or the current event type determined according to the current running scene information is different from the event type of the terminal at the last moment, it is determined that the current running scene information and the current resource usage information are not suitable for the specification of the current large model of the terminal, and the step S404 is continued to be executed.

[0091] Exemplarily, the fluctuation threshold value corresponding to the resource usage information of the terminal can be set as needed, and the present embodiment does not limit this; the event type is obtained by classifying according to the running scene information of the terminal by using a preset classification algorithm. The cloud server continuously monitors the fluctuation of the resource usage information of the terminal according to the current resource usage information fed back by the terminal, and can extract key event features from optical images by means of an image recognition algorithm, and combines multi-modal sensor data to enhance the judgment accuracy, and finally judges the event type by using a preset classification algorithm. The cloud server determines that the current running scene information and the current resource usage information are not suitable for the specification of the current large model of the terminal in the case that the fluctuation of the current resource usage information is greater than the fluctuation threshold value, or the current event type determined according to the current running scene information is different from the event type of the terminal at the last moment, that is, the resource fluctuation or the scene demand change is detected.

[0092] S403, if the fluctuation of the current resource usage information is less than or equal to the fluctuation threshold value, and the current event type determined according to the current running scene information is the same as the event type of the terminal at the last moment, it is determined that the current running scene information and the current resource usage information are suitable for the specification of the current large model of the terminal, and the process is ended.

[0093] In this step, if the fluctuation of the current resource usage information is less than or equal to the fluctuation threshold value, and the current event type determined according to the current running scene information is the same as the event type of the terminal at the last moment, it is indicated that a new large model does not need to be adapted for the terminal.

[0094] It should be noted that the present embodiment does not limit the order of execution of S402 and S403.

[0095] In the present embodiment, Figure 4 The step S303 can further include the following three steps S404 to S406:

[0096] S404, determining the current event type according to the current running scene information.

[0097] In this step, the current event type can be determined by using a preset classification algorithm according to the current running scene information.

[0098] S405, a plurality of candidate large models are obtained from the model library by using a multi-objective constraint and optimization method, and the multi-objective constraint and optimization method takes the use boundary value of different resources of the terminal as a constraint condition, and takes the precision, real-time performance and computing power as optimization targets.

[0099] Exemplarily, the available computing power, memory surplus, network bandwidth and upper limit of energy consumption of the terminal can be taken as constraint conditions (i.e., hard constraints), the precision, real-time performance and computing power are taken as optimization targets, and a plurality of candidate large models (i.e., a model subset) that meet the constraint conditions and have balanced targets are obtained from the model library by using a multi-objective constraint and optimization method.

[0100] S406, obtaining a target large model from the plurality of candidate large models based on a preset weight corresponding to the current event type.

[0101] In this step, different event types correspond to different preset weights, and the preset weights are used to represent different emphases on the precision, real-time performance and computing power. For example, high-speed road driving emphasizes high real-time performance, and fine analysis task emphasizes high precision.

[0102] Further, optionally, the preset weight includes a first weight corresponding to the precision, a second weight corresponding to the real-time performance, and a third weight corresponding to the computing power, and obtaining the target large model from the plurality of candidate large models based on the preset weight corresponding to the current event type can include: for each candidate large model in the plurality of candidate large models, performing weighted summation on the candidate precision, the candidate real-time performance, the candidate computing power, the first weight, the second weight and the third weight corresponding to the candidate large model to obtain a score corresponding to the candidate large model; and determining the target large model as the candidate large model corresponding to the highest score.

[0103] Exemplarily, the preset weights corresponding to the plurality of candidate large models are the same, but the candidate precision, the candidate real-time performance and the candidate computing power of each candidate large model are different. Therefore, for each candidate large model, the product of the candidate precision and the first weight, the product of the candidate real-time performance and the second weight, and the product of the candidate computing power and the third weight corresponding to the candidate large model can be summed, i.e., the candidate precision, the candidate real-time performance and the candidate computing power corresponding to the candidate large model are weighted and summed to obtain a score corresponding to the candidate large model. The scores corresponding to the plurality of candidate large models are sorted, so that the target large model can be determined as the candidate large model corresponding to the highest score.

[0104] In the embodiment of the application, Figure 5 The S304 step can further include the following S407 to S409 steps:

[0105] S407, obtaining difference model information between the target large model and the current large model.

[0106] In this step, the target large model can be compared with the current large model to obtain difference model information therebetween, such as difference model parameters and difference model structure information.

[0107] In S408, the difference model information is packed and compressed to obtain packed and compressed difference model information.

[0108] For example, in order to safely and efficiently send the difference model information to the terminal, the cloud server can pack and compress the difference model information according to network conditions and transmission protocol characteristics to obtain packed and compressed difference model information, so as to realize a bandwidth adaptive transmission strategy and ensure that the difference model information can still be smoothly issued in a congested or unstable network environment. In particular, in the case of limited bandwidth, a high compression rate transmission strategy is preferentially enabled.

[0109] In S409, the packed and compressed difference model information is sent to the terminal, so that the terminal deploys the target large model according to the difference model information and performs inference through the target large model.

[0110] In this step, after obtaining the packed and compressed difference model information, a difference parameter transmission mode is started to send the packed and compressed difference model information to the terminal, and an adaptive fragmentation transmission protocol can be used to ensure stable transmission of the difference model information when the network is congested.

[0111] It can be understood that through adaptive transmission compression, model pruning and quantization technology, the embodiments of the present application can still maintain acceptable performance under harsh resource conditions.

[0112] The large model deployment method provided by the embodiment of the application comprises the following steps: receiving current running scene information and current resource usage information sent by a terminal; determining that the current running scene information and the current resource usage information are not suitable for the specification of a current large model of the terminal if the fluctuation of the current resource usage information is greater than a fluctuation threshold value and / or the current event type determined according to the current running scene information is different from the event type of the terminal at a previous moment; determining the current event type according to the current running scene information; screening a plurality of candidate large models from a model library by using a multi-objective constraint and optimization method, wherein the multi-objective constraint and optimization method takes the usage boundary values of different resources of the terminal as constraint conditions and takes precision, real-time performance and computing power as optimization objectives; obtaining a target large model from the plurality of candidate large models based on preset weights corresponding to the current event type, so as to realize dynamic on-demand acquisition of a large model; the target large model obtained can better adapt to the demand of a complex and changeable scene and can reduce unnecessary model information, thereby effectively reducing the computing power and energy consumption input of the terminal; then, difference model information between the target large model and the current large model is obtained, the difference model information is packed and compressed to obtain packed and compressed difference model information, and the packed and compressed difference model information is sent to the terminal, so that the terminal deploys the target large model according to the difference model information and performs reasoning by using the target large model. In the method, only the difference model information is sent to the terminal, so that a bandwidth adaptive transmission strategy can be realized, and it is ensured that the difference model information can still be stably transmitted under a congested or unstable network environment, thereby ensuring that the terminal can obtain and deploy the target large model in time and perform reasoning by using the target large model.

[0113] On the basis of the above embodiment, Figure 6 The large model deployment method provided by the embodiment of the application is shown in the figure Figure 5 , and is applied to a terminal. As shown in the figure Figure 6 , the method of the embodiment of the application comprises the following steps:

[0114] S501, current running scene information and current resource usage information are obtained.

[0115] Exemplarily, the terminal can obtain the current running scene information and the current resource usage information of the terminal in real time. Specifically, the terminal can obtain the current running scene information such as the environmental characteristics and the scene complexity of the terminal, so as to provide a basis for the cloud server to judge the current running scene demand. Taking the terminal as a vehicle-mounted intelligent device as an example, the current running scene information obtained by the terminal may, for example, include the environmental characteristics around the vehicle, such as traffic density, crowd density and sudden events. The terminal can obtain the current resource usage information such as the CPU usage rate, the storage occupancy, the storage usage, the available bandwidth and the network throughput of the terminal, so that the cloud server can accurately match and obtain a more suitable large model in the subsequent decision-making process.

[0116] S502, send the current running scenario information and the current resource usage information to a cloud server.

[0117] In this step, after obtaining the current running scenario information and the current resource usage information, the terminal sends the current running scenario information and the current resource usage information to the cloud server.

[0118] S503, receive model information of a target large model sent by the cloud server, the target large model being obtained from a model library by the cloud server based on the current running scenario information and the current resource usage information in a case where the cloud server determines that the current running scenario information and the current resource usage information are not suitable for a specification of a current large model of the terminal, and the cloud server determines a constraint condition and an optimization target of large model screening by using a multi-objective constraint and optimization method.

[0119] In this step, how the cloud server specifically obtains the target large model can be referred to the above embodiments, which will not be described here. Illustratively, after obtaining the target large model, the cloud server sends the model information of the target large model to the terminal. Correspondingly, the terminal receives the model information of the target large model sent by the cloud server.

[0120] S504, deploy the target large model according to the model information.

[0121] In this step, after receiving the model information of the target large model sent by the cloud server, the terminal automatically deploys the target large model according to the model information. For example, the terminal can deploy the target large model by using a hot update and a gradual switching method to reduce the pause and performance flash of large model replacement and ensure the continuity of the tasks running in the terminal. How to specifically deploy the target large model by using the hot update and the gradual switching method can be referred to subsequent embodiments.

[0122] S505, perform reasoning by using the target large model.

[0123] In this step, after deploying the target large model, the terminal can perform reasoning and analysis by using the target large model. Illustratively, the terminal can perform scene analysis, prediction and decision making by using the target large model, and the results can be used for upper-layer business decision making such as anomaly detection and path planning. Specifically, taking the terminal as a vehicle-mounted intelligent device as an example, the terminal can perform anomaly detection, traffic flow prediction and path planning by using the target large model, and obtain real-time traffic prediction, anomaly detection or path decision making results. It can be understood that the embodiments of the present application can effectively reduce the calculation power consumption required by the original reasoning effect, for example, the calculation power consumption can be reduced by 25%.

[0124] The large model deployment method provided by the embodiment of the application comprises the following steps: obtaining current running scene information and current resource usage information, and sending the current running scene information and the current resource usage information to a cloud server; receiving model information of a target large model sent by the cloud server, the target large model being obtained by the cloud server from a model library based on the current running scene information and the current resource usage information by using a multi-objective constraint and optimization method in a case where it is determined that the current running scene information and the current resource usage information are not suitable for a specification of a current large model of the terminal, the multi-objective constraint and optimization method being used to determine constraint conditions and optimization targets for large model screening; and deploying the target large model according to the model information, and performing reasoning by using the target large model, so as to realize on-demand dynamic adjustment of the large model, wherein the target large model is a large model that preferentially meets core scene requirements and is obtained by using the multi-objective constraint and optimization method, the obtained target large model can better adapt to requirements of complex and changeable scenes, and can reduce unnecessary model information, and effectively reduce computing power and energy consumption input of the terminal.

[0125] Figure 5 The large model deployment method provided by the embodiment of the application is shown in the figure Figure 5 , and is applied to a terminal. On the basis of the above embodiment, the large model deployment method is further described in the embodiment of the application. As shown in the figure Figure 5 , the method of the embodiment of the application can comprise the following steps:

[0126] S601, obtaining current running scene information and current resource usage information.

[0127] The specific description of this step can be referred to the related description of S501 in the embodiment shown in the figure Figure 5 , which will not be described here again.

[0128] S602, sending the current running scene information and the current resource usage information to a cloud server.

[0129] The specific description of this step can be referred to the related description of S502 in the embodiment shown in the figure Figure 7 , which will not be described here again.

[0130] S603, receiving model information of a target large model sent by the cloud server, the target large model being obtained by the cloud server from a model library based on the current running scene information and the current resource usage information by using a multi-objective constraint and optimization method in a case where it is determined that the current running scene information and the current resource usage information are not suitable for a specification of a current large model of the terminal, the multi-objective constraint and optimization method being used to determine constraint conditions and optimization targets for large model screening.

[0131] The specific description of this step can be referred to the related description of S503 in the embodiment shown in the figure Figure 7 , which will not be described here again.

[0132] In the embodiments of the present application, Figure 8 The step S504 can further include a step S604 as follows:

[0133] The step S604 includes deploying the target large model by using a hot update and a gradual switching method according to the model information.

[0134] For example, in order to reduce the pause and performance flash during the large model replacement, improve the smoothness of the large model switching process, and reduce the interruption and resource waste caused by the large model switching, the target large model can be deployed by using a hot update and a gradual switching method according to the model information of the target large model, that is, the target large model is initialized and loaded gradually, so that the target large model can quickly enter an executable state. When the target large model is deployed by using the hot update and the gradual switching method, the following steps are mainly included: (1) parallel running, that is, during the deployment and initialization of the target large model, the current large model of the terminal continues to run to ensure that the task is not interrupted; (2) task migration, that is, once the target large model reaches an executable state, the inference and data processing responsibilities are gradually transferred from the current large model of the terminal to the target large model to ensure continuous and stable output results; (3) resource release, that is, after the task migration is completed, the current large model of the terminal is unloaded to release the memory and computing power resources, thereby reducing the redundant expenditure. The process can optimize the memory allocation and data reading method according to the hardware characteristics of the terminal to shorten the loading time delay of the target large model.

[0135] Further, the target large model can be deployed by using a hot update and a gradual switching method according to the model information, which can include maintaining the running of the current large model during the deployment of the target large model, wherein the model information of the target large model is loaded into the idle memory area in batches according to a preset priority by using a memory paging mapping method; the input data of the terminal is allocated to the target large model according to a preset incremental proportion until all the input data are migrated to the large model; and the current large model is unloaded by using a lazy resource release mechanism.

[0136] Exemplarily, the terminal can enable an asynchronous parallel loading mode, and target large model information is loaded into an idle memory area according to a preset priority while the current large model continues to run, the idle memory area being an independent memory area, so that resource competition with the current large model can be avoided. Through a memory paging mapping technology, progressive migration is performed, and a task allocation ratio is dynamically adjusted based on a time window. For example, if a preset incremental ratio is 10%, input data can be allocated to the target large model according to the 10% incremental ratio, and the current large model still processes the remaining data, until the target large model completely takes over. After the migration is completed, a lazy resource release mechanism is started, first, the video memory and thread resources are released in batches according to the computing node load, so as to avoid system jitter caused by instantaneous resource recovery, and finally, the current large model of the terminal is unloaded, and a running log is updated. Through the hot updating and progressive switching method of the embodiment of the application, while the target large model is introduced, part of the calculation of the current large model of the terminal is continued, until the target large model is deployed and takes over all tasks of the current large model, so that the continuity of the tasks can be effectively guaranteed. For example, when the large model switching is triggered, through the hot updating and progressive switching method, the task interruption time caused by the large model switching can be ensured to be less than 0.3 seconds.

[0137] Optionally, the model information is packaged and compressed difference model information between the target large model and the current large model, and deploying the target large model according to the model information can include: decompressing the packaged and compressed difference model information to obtain decompressed difference model information; and deploying the target large model according to the decompressed difference model information.

[0138] Exemplarily, the model information is packaged and compressed difference model information between the target large model and the current large model, in order to safely and efficiently send the difference model information to the terminal, the cloud server can package and compress the difference model information according to network conditions and transmission protocol characteristics to obtain packaged and compressed difference model information, and send the packaged and compressed difference model information to the terminal. Correspondingly, the terminal receives the packaged and compressed difference model information, decompresses the packaged and compressed difference model information to obtain decompressed difference model information, and then can deploy the target large model by using the hot updating and progressive switching method according to the decompressed difference model information.

[0139] S605, reasoning is performed by the target large model.

[0140] The specific description of this step can be referred to Figure 8 The related description of S505 in the embodiment shown in the figure is not repeated here.

[0141] It can be understood that the embodiment of the application constructs a perception-decision closed loop by acquiring current running scene information and current resource usage information in real time, selects the most suitable large model specification from the model library based on a multi-objective constraint and optimization method, and deploys it to the terminal to perform tasks; when the scene demand changes or the resource condition fluctuates, a hot update mechanism is triggered in time, and the parallel running of new and old models and the smooth migration of tasks are realized through a gradual switching path, which can ensure business continuity while completing the redeployment of large models and realizing the dynamic adjustment of large models on demand.

[0142] The large model deployment method provided by the embodiment of the application acquires current running scene information and current resource usage information, sends the current running scene information and the current resource usage information to a cloud server, receives model information of a target large model sent by the cloud server, and deploys the target large model by using a hot update and a gradual switching method according to the model information and performs reasoning through the target large model. The target large model is obtained by the cloud server from a model library based on the current running scene information and the current resource usage information by using a multi-objective constraint and optimization method in a case where it is determined that the current running scene information and the current resource usage information are not suitable for a specification of a current large model of the terminal. The target large model can better adapt to the demand of a complex and changeable scene, and can reduce unnecessary model information, effectively reducing the computing power and energy consumption input of the terminal. By using the hot update and the gradual switching method, the system resilience and continuity of the embodiment of the application in a complex and changeable scene can be ensured, the task interruption time caused by large model switching can be effectively reduced, the continuity of the task can be ensured, and the smoothness of the large model switching process can be improved.

[0143] To sum up, the embodiment of the application constructs a whole-process adaptation framework from scene perception, model selection, dynamic transmission, flexible deployment, seamless switching and pruning and quantization optimization. In a complex scene with limited resources and changing conditions, the framework can flexibly deploy a large model that is most suitable for the current resources and scene by analyzing the scene demand in real time and combining the available resources, can reduce unnecessary large model parameters, and can effectively reduce the system computing power and energy consumption input. Compared with a single static model deployment scheme, the computing power required to achieve the same reasoning effect is reduced by 25%. In addition, relying on the hot update and the gradual switching method, the task interruption time caused by large model switching is less than 0.3 seconds, which can ensure the continuity of the task.

[0144] The following is an apparatus embodiment of the application, which can be used to perform the method embodiment of the application. For details not disclosed in the apparatus embodiment of the application, please refer to the method embodiment of the application.

[0145] Figure 9 The structure diagram of the large model deployment apparatus provided by an embodiment of the application is applied to a cloud server. As shown in Figure 9As shown, the large model deployment apparatus 700 of the embodiment of the present application comprises: a receiving module 701 configured to receive current running scene information and current resource usage information sent by a terminal; a determining module 702 configured to determine whether the current running scene information and the current resource usage information are suitable for the specification of a current large model of the terminal; an obtaining module 703 configured to, if not, obtain a target large model from a model library based on the current running scene information and the current resource usage information by using a multi-objective constraint and optimization method, the multi-objective constraint and optimization method being used to determine constraint conditions and optimization objectives of large model screening; and a sending module 704 configured to send model information of the target large model to the terminal, so that the terminal deploys the target large model according to the model information and performs reasoning through the target large model.

[0146] Further, the determining module 702 can be specifically configured to: if fluctuation of the current resource usage information is greater than a fluctuation threshold value, and / or a current event type determined according to the current running scene information is different from an event type of the terminal at a previous moment, determine that the current running scene information and the current resource usage information are not suitable for the specification of the current large model of the terminal; and if fluctuation of the current resource usage information is less than or equal to the fluctuation threshold value, and the current event type determined according to the current running scene information is the same as the event type of the terminal at the previous moment, determine that the current running scene information and the current resource usage information are suitable for the specification of the current large model of the terminal.

[0147] Further, the obtaining module 703 can be specifically configured to: determine a current event type according to the current running scene information; obtain a plurality of candidate large models from the model library by using the multi-objective constraint and optimization method, the multi-objective constraint and optimization method taking usage boundary values of different resources of the terminal as constraint conditions and taking precision, real-time performance and computing power as optimization objectives; and obtain the target large model from the plurality of candidate large models based on a preset weight corresponding to the current event type.

[0148] Further, the preset weight comprises a first weight corresponding to the precision, a second weight corresponding to the real-time performance, and a third weight corresponding to the computing power. When used to obtain the target large model from the plurality of candidate large models based on the preset weight corresponding to the current event type, the obtaining module 703 can be specifically configured to: for each candidate large model in the plurality of candidate large models, perform weighted summation on a candidate precision, a candidate real-time performance, a candidate computing power, the first weight, the second weight and the third weight corresponding to the candidate large model to obtain a score corresponding to the candidate large model; and determine the target large model as a candidate large model corresponding to the highest score.

[0149] Further, the sending module 704 can be specifically configured to: obtain difference model information between the target large model and the current large model; package and compress the difference model information to obtain packaged and compressed difference model information; and send the packaged and compressed difference model information to the terminal.

[0150] The device of the embodiment of the present application can be used to implement the scheme of the cloud server in any of the above method embodiments, and has similar implementation principles and technical effects, which will not be described here.

[0151] Figure 7 The structure schematic diagram of the large model deployment device provided for another embodiment of the present application is applied to a terminal. As shown in the figure, Figure 2 The large model deployment device 800 of the embodiment of the present application includes: an acquisition module 801, configured to acquire current running scenario information and current resource usage information; a sending module 802, configured to send the current running scenario information and the current resource usage information to a cloud server; a receiving module 803, configured to receive model information of a target large model sent by the cloud server, the target large model being obtained from a model library by the cloud server based on the current running scenario information and the current resource usage information by using a multi-objective constraint and optimization method in a case where it is determined that the current running scenario information and the current resource usage information are not suitable for a specification of a current large model of the terminal, the multi-objective constraint and optimization method being used to determine constraint conditions and optimization targets for large model screening; a deployment module 804, configured to deploy the target large model according to the model information; and an inference module 805, configured to perform inference by using the target large model.

[0152] Further, the deployment module 804 can be specifically configured to deploy the target large model by using a hot update and gradual switching method according to the model information.

[0153] Further, when the deployment module 804 is configured to deploy the target large model by using a hot update and gradual switching method according to the model information, it can be specifically configured to maintain running of the current large model during deployment of the target large model, wherein the model information of the target large model is loaded into an idle memory area in batches according to a preset priority by using a memory paging mapping method; input data of the terminal is allocated to the target large model according to a preset incremental ratio until the input data is all migrated to the large model; and a lazy resource release mechanism is used to unload the current large model.

[0154] Further, the model information is difference model information obtained by packing and compressing between the target large model and the current large model, and the deployment module 804 can be specifically configured to decompress the difference model information obtained by packing and compressing to obtain decompressed difference model information, and deploy the target large model according to the decompressed difference model information.

[0155] The device of the embodiment of the present application can be used to implement the scheme of the terminal in any of the above method embodiments, and has similar implementation principles and technical effects, which will not be described here.

[0156] Based on the above embodiments, Figure 3 The schematic diagram of the large model deployment system provided for an embodiment of the present application is as follows, Figure 4As shown, the large-scale deployment system 900 includes a cloud server 901 and a terminal 902, which are communicatively connected. The cloud server 901 can employ... Figure 8 The structure of the device embodiment, correspondingly, can perform... Figure 2 , Figure 5 or Figure 6 The technical solutions in the embodiments are similar in principle and technical effect, and will not be described again here. Terminal 902 can adopt... Figure 10 The structure of the device embodiment, correspondingly, can perform... Figure 10 , Figure 11 or Figure 11 The technical solutions in the embodiments are similar in principle and effect, and will not be described again here.

[0157] Based on the above embodiments, ​ A schematic diagram of a large model deployment system provided in another embodiment of the present invention, such as... ​ As shown, the large model deployment system 1000 can include a model resource layer, a scene perception and decision-making layer, a dynamic deployment execution layer, and a terminal inference layer. The model resource layer includes a model storage and management unit, used to store and manage large models of various specifications through a model library. The scene perception and decision-making layer includes an information acquisition unit, a resource monitoring unit, a scene requirement analysis unit, and a model matching unit. These four units work closely together to dynamically select large models based on real-time scene requirements and device resource usage. The dynamic deployment execution layer includes a model transmission unit and a model loading unit. These units ensure the smooth deployment of the target large model to the terminal. When switching between old and new models is required, hot updates and gradual switching methods are used to achieve a smooth replacement of the large model. The terminal inference layer includes an inference unit, covering data processing and result feedback, ultimately providing necessary functional support for practical applications such as intelligent scheduling, path planning, and risk warning. Through the aforementioned large-scale model deployment system, this embodiment of the invention combines the complexity of real-world scenarios with the diversity of large-scale models, ensuring that even in real-world environments where road conditions, scenario complexity, and network conditions are constantly fluctuating, it can still quickly match and deploy large-scale model solutions that meet target indicators, thereby achieving a balance between resource optimization and efficient decision-making.

[0158] ​ This is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present invention. ​ As shown, the electronic device 1100 may include at least one processor 1101 and a memory 1102.

[0159] The memory 1102 is used to store programs. Specifically, the program may include program code, which includes computer-executable instructions.

[0160] The memory 1102 can include a high-speed random access memory (RAM) and can also include a non-volatile memory such as at least one disk memory.

[0161] The processor 1101 is configured to execute the computer-executable instructions stored in the memory 1102 to implement the large model deployment method described in the foregoing method embodiments. The processor 1101 can be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. Specifically, when implementing the large model deployment method described in the foregoing method embodiments, the electronic device can be a cloud server or a terminal.

[0162] Optionally, the electronic device 1100 can further include a communication interface 1103. In a specific implementation, if the communication interface 1103, the memory 1102, and the processor 1101 are independently implemented, the communication interface 1103, the memory 1102, and the processor 1101 can be connected to each other through a bus and complete communication between them. The bus can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.

[0163] Optionally, in a specific implementation, if the communication interface 1103, the memory 1102, and the processor 1101 are integrated on a chip, the communication interface 1103, the memory 1102, and the processor 1101 can complete communication through an internal interface.

[0164] The present application also provides a computer-readable storage medium, which stores computer program instructions, and when a processor executes the computer program instructions, the large model deployment method described above is implemented.

[0165] The present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the large model deployment method described above is implemented.

[0166] The computer readable storage medium described above can be implemented by any type of volatile or nonvolatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read only memory (EEPROM), erasable programmable read only memory (EPROM), programmable read only memory (PROM), read only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0167] An exemplary readable storage medium is coupled to the processor, thereby enabling the processor to read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a part of the processor. The processor and the readable storage medium can be located in a special integrated circuit. Of course, the processor and the readable storage medium can also exist as discrete components in a large model deployment device.

[0168] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The foregoing program can be stored in a computer readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the foregoing storage medium includes ROM, RAM, magnetic disk or optical disk and various storage medium that can store program codes.

[0169] Finally, it should be pointed out that the above examples are only preferred examples for fully illustrating the present application, and the protection scope of the present application is not limited thereto. Equivalent substitutions or transformations made by those skilled in the art on the basis of the present application are within the protection scope of the present application.

Claims

1. A large model deployment method, characterized in that, The large model deployment method applied to a cloud server comprises: receiving current running scene information and current resource usage information sent by a terminal; determining whether the current running scene information and the current resource usage information are suitable for the specification of a current large model of the terminal; if not, using a multi-objective constraint and optimization method to screen a target large model from a model library based on the current running scene information and the current resource usage information, the multi-objective constraint and optimization method being used to determine constraint conditions and optimization objectives of large model screening; sending model information of the target large model to the terminal, so that the terminal deploys the target large model according to the model information and performs reasoning through the target large model.

2. The large model deployment method of claim 1, wherein, The determination of whether the current running scene information and the current resource usage information are suitable for the specification of the current large model of the terminal comprises: if the fluctuation of the current resource usage information is greater than a fluctuation threshold value, and / or the current event type determined according to the current running scene information is different from the event type of the terminal at the last time point, it is determined that the current running scene information and the current resource usage information are not suitable for the specification of the current large model of the terminal; if the fluctuation of the current resource usage information is less than or equal to the fluctuation threshold value, and the current event type determined according to the current running scene information is the same as the event type of the terminal at the last time point, it is determined that the current running scene information and the current resource usage information are suitable for the specification of the current large model of the terminal.

3. The large model deployment method of claim 2, wherein, The screening of the target large model from the model library based on the current running scene information and the current resource usage information using the multi-objective constraint and optimization method comprises: determining a current event type according to the current running scene information; screening a plurality of candidate large models from the model library using the multi-objective constraint and optimization method, the multi-objective constraint and optimization method taking the usage boundary values of different resources of the terminal as constraint conditions and taking precision, real-time performance and computing power as optimization objectives; obtaining the target large model from the plurality of candidate large models based on preset weights corresponding to the current event type.

4. The large model deployment method of claim 3, wherein, The preset weights comprise a first weight corresponding to the precision, a second weight corresponding to the real-time performance and a third weight corresponding to the computing power, and the obtaining of the target large model from the plurality of candidate large models based on the preset weights corresponding to the current event type comprises: for each candidate large model in the plurality of candidate large models, performing weighted summation on the candidate precision, the candidate real-time performance, the candidate computing power, the first weight, the second weight and the third weight corresponding to the candidate large model to obtain a score corresponding to the candidate large model; determining the target large model as the candidate large model corresponding to the highest score.

5. The large model deployment method of any one of claims 1-4, wherein, The sending of the model information of the target large model to the terminal comprises: obtaining difference model information between the target large model and the current large model; packing and compressing the difference model information to obtain packed and compressed difference model information; sending the packed and compressed difference model information to the terminal.

6. A large model deployment method, characterized by, The large model deployment method applied to a terminal comprises the following steps: obtaining current running scene information and current resource usage information; sending the current running scene information and the current resource usage information to a cloud server; receiving model information of a target large model sent by the cloud server, the target large model being obtained from a model library by a multi-objective constraint and optimization method based on the current running scene information and the current resource usage information in a case where the cloud server determines that the current running scene information and the current resource usage information are not suitable for a specification of a current large model of the terminal, the multi-objective constraint and optimization method being used to determine constraint conditions and optimization objectives for large model screening; deploying the target large model according to the model information; and 7. The large model deployment method of claim 6, wherein, performing inference by the target large model. The step of deploying the target large model according to the model information comprises the following steps:

8. The large model deployment method of claim 7, wherein, deploying the target large model by using a hot update and gradual switching method according to the model information. The step of deploying the target large model by using the hot update and gradual switching method according to the model information comprises the following steps: maintaining running of the current large model during deployment of the target large model, wherein the model information of the target large model is loaded into an idle memory area in batches according to a preset priority by using a memory paging mapping method; allocating input data of the terminal to the target large model according to a preset incremental ratio until the input data is migrated to the large model; and 9. The large model deployment method of any one of claims 6-8, wherein, uninstalling the current large model by using a lazy resource release mechanism. The model information is difference model information obtained by packing and compressing between the target large model and the current large model, and the step of deploying the target large model according to the model information comprises the following steps: decompressing the packed and compressed difference model information to obtain decompressed difference model information; and 10.A large model deployment apparatus, characterized by comprising: deploying the target large model according to the decompressed difference model information. The large model deployment apparatus applied to a cloud server comprises the following components: a receiving module configured to receive current running scene information and current resource usage information sent by a terminal; a determining module configured to determine whether the current running scene information and the current resource usage information are suitable for a specification of a current large model of the terminal; an obtaining module configured to, if not, obtain a target large model from a model library by using a multi-objective constraint and optimization method based on the current running scene information and the current resource usage information, the multi-objective constraint and optimization method being used to determine constraint conditions and optimization objectives for large model screening; 11.A large model deployment apparatus, characterized by comprising: a sending module configured to send model information of the target large model to the terminal, so that the terminal deploys the target large model according to the model information and performs inference by the target large model. The large model deployment apparatus applied to a terminal comprises the following components: an obtaining module configured to obtain current running scene information and current resource usage information; a sending module configured to send the current running scene information and the current resource usage information to a cloud server; receive model information of a target large model sent by the cloud server, the target large model being obtained by the cloud server from a model library based on the current running scenario information and the current resource usage information by using a multi-objective constraint and optimization method in a case that the cloud server determines that the current running scenario information and the current resource usage information do not match a specification of a current large model of the terminal, the multi-objective constraint and optimization method being used to determine a constraint condition and an optimization target of large model screening; deploy the target large model according to the model information; perform reasoning by using the target large model.

12. A large model deployment system, characterized by, Comprise: a cloud server and a terminal; the cloud server is configured to perform the large model deployment method in any one of claims 1 to 5; the terminal is configured to perform the large model deployment method in any one of claims 6 to 9.

13. An electronic device, comprising: Comprise: a processor, and a memory connected to the processor in communication; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the large model deployment method in any one of claims 1 to 9.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, and the computer program instructions are executed to implement the large model deployment method in any one of claims 1 to 9.

15. A computer program product comprising a computer program, characterized in that, The computer program is executed to implement the large model deployment method in any one of claims 1 to 9.

Images