End-side large model resource optimization method and system, and storage medium

Abstract

The application provides an end-side large model resource optimization method and system, and a storage medium. The method comprises the following steps: performing resource optimization trigger detection on an end-side large model; if the resource optimization trigger detection is qualified, obtaining current resource optimization event information, determining a target resource level according to the current resource optimization event information; comparing the current resource level with the target resource level to obtain a resource comparison result, and performing resource optimization processing on the end-side large model according to the resource comparison result. According to the embodiment of the application, the resource optimization trigger detection is performed on the end-side large model to determine whether the resource optimization operation needs to be performed on the end-side large model. The target resource level can be effectively determined by obtaining the current resource optimization event information. The resource optimization processing can be automatically performed on the end-side large model based on the resource comparison result, without the need to perform quantization compression on the end-side large model or to reduce the performance of the end-side large model, thereby improving the user experience.

Description

Technical Field

[0001] This invention relates to the field of large model resource optimization technology, and in particular to an edge-side large model resource optimization method, system, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, edge-side large-scale models have moved from conceptual exploration to large-scale application. Through lightweight technologies such as quantization and distillation, the core inference capabilities of large models are deployed to terminal hardware such as mobile phones, in-vehicle devices, and smart home devices. Relying on local computing power to complete data processing and intelligent interaction, they can operate offline without continuous cloud support, possessing core advantages such as localization, low latency, and high privacy protection. However, fully loading an edge-side large-scale model requires significant memory, while terminal devices have limited available memory. Each call necessitates reloading model weights from storage; therefore, the optimization of edge-side large-scale model resources is receiving increasing attention.

[0003] In the current use of large edge models, resource optimization of large edge models is achieved by quantizing and compressing them. However, this results in limited capabilities of the large edge models and a reduced user experience. Summary of the Invention

[0004] The purpose of this invention is to provide a method, system, and storage medium for optimizing large-scale edge models, in order to solve the problem of limited large-scale edge model capabilities in the prior art.

[0005] This invention is implemented as follows: a method for optimizing large-scale end-side model resources, the method comprising:

[0006] Resource optimization trigger detection is performed on the large model on the endpoint.

[0007] If the resource optimization trigger detection is qualified, the current resource optimization event information is obtained, and the target resource level is determined based on the current resource optimization event information;

[0008] The current resource level is compared with the target resource level to obtain the resource comparison result, and the resource optimization process is performed on the end-side large model based on the resource comparison result.

[0009] Preferably, resource optimization trigger detection is performed on the large model on the edge side, including:

[0010] Broadcast monitoring is performed on the process manager corresponding to the large end-side model to obtain activity broadcast monitoring information, and the foreground application package name change information is determined based on the activity broadcast monitoring information;

[0011] When the front-end application package name in the front-end application package name change information changes, the resource optimization trigger detection is deemed qualified.

[0012] Broadcast monitoring is performed on the battery manager corresponding to the large end-side model to obtain battery broadcast monitoring information, and the change value of battery percentage is determined based on the battery broadcast monitoring information;

[0013] If the change in battery percentage is greater than the battery change threshold, the resource optimization trigger detection is deemed to be qualified.

[0014] The large-scale model on the endpoint is monitored for memory warning broadcasts to obtain memory warning monitoring information, and the currently available memory is determined based on the memory warning monitoring information.

[0015] If the current available memory is less than the memory threshold, then the resource optimization trigger detection is deemed qualified.

[0016] Obtain the current time, and if the current time is within a preset time range, determine that the resource optimization trigger detection is qualified;

[0017] The current network status is obtained, and if the current network status meets the preset network conditions, the resource optimization trigger detection is deemed qualified.

[0018] Preferably, obtain current resource optimization event information, including:

[0019] Get the current remaining battery level from the battery manager, and get the current available memory and the current application package name from the process manager;

[0020] Obtain the historical hierarchical usage information corresponding to the large model on the edge, obtain the current network type according to the network manager, and obtain the temperature information according to the heat manager;

[0021] The current resource optimization event information is generated based on the current remaining battery power, the current available memory, the current application package name, the historical tier usage information, the current network type, and the temperature information.

[0022] Preferably, determining the target resource level based on the current resource optimization event information includes:

[0023] If the current remaining power is less than the power threshold, then the target resource level is determined to be the basic resource level;

[0024] If the currently available memory is less than the memory threshold, then the target resource level is determined to be the basic resource level;

[0025] The application type is determined based on the current application package name, and when the application type is a development tool, the target resource level is determined as a professional resource level.

[0026] When the application type is an office application and a multi-turn dialogue is detected, the target resource level is determined as an enhanced resource level.

[0027] When the application type is a chat application, the target resource level is determined as an enhanced resource level;

[0028] The historical resource level corresponding to the current time is determined based on the historical resource level usage information, and the historical resource level is determined as the target resource level.

[0029] Preferably, resource optimization processing is performed on the large-scale end-side model based on the resource comparison results, including:

[0030] The key-value cache of the large model on the terminal side is serialized to a temporary storage space, and the current dialogue context message sequence is saved, and the model configuration parameters are recorded.

[0031] Obtain the weight files of the target resource level and the current resource level to obtain the target weight file and the current weight file, and determine the resource loading increment based on the target weight file and the current weight file;

[0032] Create a shared memory file and incrementally map the resource load to the memory in the shared memory file;

[0033] Load the key-value cache into the temporary storage space, restore the current dialogue context message sequence, and apply the model configuration parameters;

[0034] Release the memory mapping of the current resource level, reclaim memory resources, and update memory usage statistics;

[0035] Send a resource optimization completion signal, update the hierarchical status, and record the switchover log.

[0036] Preferably, a switchover log is recorded, including:

[0037] The switching log is obtained by recording the switching time, the current resource level, the target resource level, the switching reason, the triggering event type, the switching time, memory changes, and power consumption changes.

[0038] Preferably, the current resource level is compared with the target resource level to obtain a resource comparison result, including:

[0039] If the current resource level is the same as the target resource level, then maintain the current resource state and record the decision log;

[0040] If the current resource level is different from the target resource level, then resource optimization is required.

[0041] Another objective of this invention is to provide an edge-side large model resource optimization system, the system comprising:

[0042] The trigger detection module is used to perform resource optimization trigger detection on large models on the edge side;

[0043] The resource level determination module is used to obtain the current resource optimization event information and determine the target resource level based on the current resource optimization event information if the resource optimization trigger detection is qualified.

[0044] The resource optimization module is used to compare the current resource level with the target resource level to obtain the resource comparison result, and to perform resource optimization processing on the end-side large model based on the resource comparison result.

[0045] Preferably, the trigger detection module is further used for:

[0046] Broadcast monitoring is performed on the process manager corresponding to the large end-side model to obtain activity broadcast monitoring information, and the foreground application package name change information is determined based on the activity broadcast monitoring information;

[0047] When the front-end application package name in the front-end application package name change information changes, the resource optimization trigger detection is deemed qualified.

[0048] Broadcast monitoring is performed on the battery manager corresponding to the large end-side model to obtain battery broadcast monitoring information, and the change value of battery percentage is determined based on the battery broadcast monitoring information;

[0049] If the change in battery percentage is greater than the battery change threshold, the resource optimization trigger detection is deemed to be qualified.

[0050] The large-scale model on the endpoint is monitored for memory warning broadcasts to obtain memory warning monitoring information, and the currently available memory is determined based on the memory warning monitoring information.

[0051] If the current available memory is less than the memory threshold, then the resource optimization trigger detection is deemed qualified.

[0052] Obtain the current time, and if the current time is within a preset time range, determine that the resource optimization trigger detection is qualified;

[0053] The current network status is obtained, and if the current network status meets the preset network conditions, the resource optimization trigger detection is deemed qualified.

[0054] This invention improves the accuracy of resource optimization for large edge models by detecting resource optimization triggers on the edge model to determine whether resource optimization operations need to be performed. By acquiring current resource optimization event information, the target resource level can be effectively determined. By comparing the current resource level with the target resource level, a resource comparison result can be effectively generated. Based on the resource comparison result, resource optimization processing can be automatically performed on the edge model without quantization compression or performance degradation, thereby improving the user experience. Attached Figure Description

[0055] Figure 1 This is a flowchart of the end-side large model resource optimization method provided in the first embodiment of the present invention;

[0056] Figure 2 This is a schematic diagram of the structure of the end-side large model resource optimization system provided in the second embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram of the structure of the terminal device provided in the third embodiment of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0059] To illustrate the technical solution described in this invention, specific embodiments are described below.

[0060] Example 1

[0061] Please see Figure 1 This is a flowchart of the edge-side large model resource optimization method provided in the first embodiment of the present invention. This edge-side large model resource optimization method can be applied to any device or system. The edge-side large model resource optimization method includes the following steps:

[0062] Step S10: Perform resource optimization trigger detection on the large model on the edge side;

[0063] Specifically, an application context is created within the terminal corresponding to the large-scale model on the device side, the decision engine is initialized, the system broadcast receiver is registered, the base layer of the large-scale model on the device side is loaded into memory, the weight file of the base layer is read from storage and loaded into the shared memory area through memory mapping, the capabilities of the base layer are activated, the base layer provides basic dialogue, simple question and answer, and command recognition capabilities, the inference engine is initialized, the decision engine enters event-driven mode, and listens for broadcast and user demand events to realize resource optimization trigger detection of the large-scale model on the device side.

[0064] Optionally, resource optimization trigger detection can be performed on the large model on the edge, including:

[0065] Broadcast monitoring is performed on the process manager corresponding to the large client-side model to obtain activity broadcast monitoring information, and the foreground application package name change information is determined based on the activity broadcast monitoring information; wherein, application switching event detection is performed on the large client-side model, the broadcast of the process manager is listened to, the foreground application package name change is detected, and the application type (chat / office / development / system) is identified.

[0066] When the front-end application package name in the front-end application package name change information changes, the resource optimization trigger detection is deemed qualified; wherein, when the front-end application package name in the front-end application package name change information changes, it is determined that the current front-end application has changed, and thus, the resource optimization process of the peer-side large model is triggered.

[0067] Broadcast monitoring is performed on the battery manager corresponding to the large-scale model on the edge to obtain battery broadcast monitoring information, and the change value of battery percentage is determined based on the battery broadcast monitoring information; wherein, battery change event detection is performed on the large-scale model on the edge, listening to the broadcast of the battery manager and detecting the change of battery percentage;

[0068] If the change in the percentage of battery power is greater than the battery power change threshold, the resource optimization trigger detection is deemed qualified; wherein, when the change exceeds the battery power change threshold, the resource optimization process of the large model on the other end is triggered.

[0069] The large-scale client-side model is monitored for memory warning broadcasts to obtain memory warning monitoring information, and the currently available memory is determined based on the memory warning monitoring information; wherein, the large-scale client-side model is monitored for memory shortage events and memory warning broadcasts are listened for.

[0070] If the current available memory is less than the memory threshold, the resource optimization trigger detection is deemed qualified; wherein, when the available memory is lower than the memory threshold, the resource optimization process of the large model on the peer side is triggered.

[0071] The current time is obtained, and if the current time is within a preset time range, the resource optimization trigger detection is deemed qualified; wherein, time mode event detection is performed on the large model on the end side, and the working time / night mode switching is determined by the time. The working time is usually the daytime period, and the night mode is the late night period.

[0072] The current network status is obtained, and if the current network status meets the preset network conditions, the resource optimization trigger detection is deemed qualified. Among them, network status event detection is performed on the large model on the end side, and the broadcast of the network manager is listened to, which affects the prefetching strategy. The preset network conditions can be set according to the requirements, and can be set to detect the switching between WiFi and mobile networks.

[0073] Step S20: If the resource optimization trigger detection is qualified, obtain the current resource optimization event information and determine the target resource level based on the current resource optimization event information;

[0074] Optionally, obtain current resource optimization event information, including:

[0075] The remaining battery level is obtained from the battery manager, and the available memory and application package name are obtained from the process manager. The remaining battery level, available memory, and application package name are used to determine whether the large model on the other end needs to be switched at different levels. The memory and resources required for different levels are different.

[0076] The historical hierarchical usage information corresponding to the large model on the edge is obtained, the current network type is obtained according to the network manager, and the temperature information is obtained according to the heat manager; wherein, the historical hierarchical usage information includes user usage habit statistics, including the usage frequency of each level, time period preference, etc.

[0077] The current resource optimization event information is generated based on the current remaining battery power, the current available memory, the current application package name, the historical tier usage information, the current network type, and the temperature information.

[0078] Further, the target resource level is determined based on the current resource optimization event information, including:

[0079] If the current remaining power is less than the power threshold, the target resource level is determined to be the basic resource level. The power threshold can be set according to the requirements. If the current remaining power is less than the power threshold, an emergency downgrade is performed. By determining the target resource level to be the basic resource level, the resource utilization of the large model on the end side in the low power mode can be realized.

[0080] If the current available memory is less than the memory threshold, the target resource level is determined to be the basic resource level. The memory threshold can be set according to the requirements. If the current available memory is less than the memory threshold, an emergency degradation is performed. By determining the target resource level to be the basic resource level, the resource utilization of the large model on the edge in low memory mode can be realized.

[0081] The application type is determined based on the current application package name, and when the application type is a development tool, the target resource level is determined as a professional resource level. Specifically, when the currently switched application type is detected to be a development tool, the code generation capability is loaded into the client-side large model by determining the target resource level as a professional resource level, so as to meet the resource utilization of the development tool, ensure the performance of the client-side large model, and improve the user experience.

[0082] When the application type is an office application and a multi-turn dialogue is detected, the target resource level is determined as an enhanced resource level. Specifically, when the currently switched application type is detected as an office application and a multi-turn dialogue is detected, the target resource level is determined as an enhanced resource level to meet the resource usage of the large-scale client-side model during multi-turn dialogues in office applications, thus ensuring the user experience.

[0083] When the application type is a chat application, the target resource level is determined as an enhanced resource level; wherein, when it is detected that the currently switched application type is a chat application, the target resource level is determined as an enhanced resource level to ensure the use of large model resources on the client side during real-time chat.

[0084] Based on the historical resource level usage information, the historical resource level corresponding to the current time is determined, and the historical resource level is determined as the target resource level. During working hours, an adaptive enhancement layer strategy is adopted, and during night mode, a basic layer power saving strategy is adopted.

[0085] Default strategy: Under no special conditions, the user's historically high-frequency tier is used as the target resource tier.

[0086] Step S30: Compare the current resource level with the target resource level to obtain the resource comparison result, and perform resource optimization processing on the end-side large model based on the resource comparison result;

[0087] The process involves comparing the current resource level with the target resource level to obtain a resource comparison result. This includes: if the current resource level is the same as the target resource level, maintaining the current resource state and recording a decision log; if the current resource level is different from the target resource level, determining that resource optimization is needed. Preferably, when the time interval since the last switch is detected to be less than a time threshold, maintaining the current resource state avoids frequent switching (e.g., not switching repeatedly within 5 minutes).

[0088] Optionally, resource optimization processing is performed on the large-scale end-side model based on the resource comparison results, including:

[0089] The key-value cache of the terminal-side large model is serialized to a temporary storage space, and the current dialogue context message sequence is saved, and the model configuration parameters are recorded; wherein, the current state of the terminal-side large model is saved, and the key-value cache of the terminal-side large model is serialized to a temporary storage space.

[0090] Obtain the weight files of the target resource level and the current resource level to obtain the target weight file and the current weight file, and determine the resource loading increment based on the target weight file and the current weight file; wherein, calculate the difference weight between the target resource level and the current resource level, read the target level weight file, compare it with the current level weight, and calculate the incremental part that needs to be loaded;

[0091] Create a shared memory file and map the resource loading increment to the memory in the shared memory file; wherein, memory mapping loading is performed, a shared memory file is created, and the resource loading increment is mapped to the memory in the shared memory file to achieve zero-copy access;

[0092] Load the key-value cache into the temporary storage space, restore the current dialogue context message sequence, and apply the model configuration parameters; wherein, restoring the dialogue state involves loading the key-value cache from the temporary storage, restoring the dialogue context message sequence, and applying the model configuration parameters.

[0093] Release the memory mapping of the current resource level, reclaim memory resources, and update memory usage statistics; among these, the original layer resources are released.

[0094] Send a resource optimization completion signal, update the hierarchical status, and record the switchover log.

[0095] Furthermore, a switching log is recorded, including: the switching time, the current resource level, the target resource level, the switching reason, and the trigger event type, as well as the switching time, memory changes, and power consumption changes, to obtain the switching log; wherein, user usage habit statistics are updated, including the usage frequency of each level, time period distribution, and application type distribution, and the switching time, memory changes, and power consumption changes are recorded for subsequent optimization analysis, the switching result is returned to the caller, and services continue to be provided, waiting for the next trigger event.

[0096] Preferably, this embodiment also includes:

[0097] Scheduled task triggering: Configure a periodic task through the job manager to trigger a prefetch check every 30 minutes;

[0098] Check prefetching conditions: Determine if the device is charging, connected to a WiFi network, or in an idle state;

[0099] Predict the next level of demand: Analyze user habits, current time period, and scheduled application plans to predict the most likely level of demand;

[0100] Background download of target layer: Downloads the target layer weight file from the cloud or local cache, supporting incremental download and resume download;

[0101] Preload into cache: Preload the downloaded target layer weights into the memory cache to prepare for memory mapping and warm up the model;

[0102] Update prefetch status: Record prefetch time, target level, prefetch result, and notify the decision engine that prefetching is complete.

[0103] The system in this embodiment includes a four-layer architecture:

[0104] 1. User application layer (chat applications, office applications, development tools, system assistants);

[0105] 2. Unified API interface layer (encapsulating capabilities such as dialogue / translation / writing / code / multimodal), dynamically layered loading decision engine (decision factor collection module, layered strategy calculation module);

[0106] 3. Model management layer (model hierarchical storage structure, layer switching optimization module);

[0107] 4. Android system resource layer (MemoryFile, SharedMemory, WorkManager, BatteryManager).

[0108] In this embodiment, the problem of excessive memory consumption of large models on the client side is effectively solved by significantly reducing resident memory usage. Cold start latency is significantly reduced, layer switching latency is extremely low and almost imperceptible to the user. The survival time of the model in the background is greatly increased, ensuring service continuity. Power consumption is significantly reduced in low power mode, intelligent prefetching avoids power consumption from mobile networks, and the dialogue context remains continuous during layer switching without the user's awareness. The application type is automatically identified and the corresponding capabilities are preloaded. Through dynamic layered architecture, incremental loading reduces a large number of I / O operations, and zero-copy memory mapping avoids data copying overhead.

[0109] In this embodiment, resource optimization trigger detection is performed on the large-scale model on the edge to determine whether resource optimization operations need to be performed on the large-scale model on the edge, which improves the accuracy of resource optimization on the large-scale model on the edge. By obtaining the current resource optimization event information, the target resource level can be effectively determined. By comparing the current resource level with the target resource level, a resource comparison result can be effectively generated. Based on the resource comparison result, resource optimization processing of the large-scale model on the edge can be automatically performed without quantization compression of the large-scale model on the edge, without reducing the performance of the large-scale model on the edge, thereby improving the user experience.

[0110] Example 2

[0111] Please see Figure 2 This is a schematic diagram of the structure of the end-side large model resource optimization system 100 provided in the second embodiment of the present invention, including:

[0112] Trigger detection module 10 is used to perform resource optimization trigger detection on the large model on the end side.

[0113] Optionally, the trigger detection module 10 is further configured to: perform broadcast monitoring on the process manager corresponding to the terminal large model, obtain activity broadcast monitoring information, and determine the foreground application package name change information based on the activity broadcast monitoring information;

[0114] When the front-end application package name in the front-end application package name change information changes, the resource optimization trigger detection is deemed qualified.

[0115] Broadcast monitoring is performed on the battery manager corresponding to the large end-side model to obtain battery broadcast monitoring information, and the change value of battery percentage is determined based on the battery broadcast monitoring information;

[0116] If the change in battery percentage is greater than the battery change threshold, the resource optimization trigger detection is deemed to be qualified.

[0117] The large-scale model on the endpoint is monitored for memory warning broadcasts to obtain memory warning monitoring information, and the currently available memory is determined based on the memory warning monitoring information.

[0118] If the current available memory is less than the memory threshold, then the resource optimization trigger detection is deemed qualified.

[0119] Obtain the current time, and if the current time is within a preset time range, determine that the resource optimization trigger detection is qualified;

[0120] The current network status is obtained, and if the current network status meets the preset network conditions, the resource optimization trigger detection is deemed qualified.

[0121] The resource level determination module 11 is used to obtain the current resource optimization event information and determine the target resource level based on the current resource optimization event information if the resource optimization trigger detection is qualified.

[0122] Optionally, the resource level determination module 11 is also used to: obtain the current remaining power according to the battery manager, and obtain the current available memory and the current application package name according to the process manager;

[0123] Obtain the historical hierarchical usage information corresponding to the large model on the edge, obtain the current network type according to the network manager, and obtain the temperature information according to the heat manager;

[0124] The current resource optimization event information is generated based on the current remaining battery power, the current available memory, the current application package name, the historical tier usage information, the current network type, and the temperature information.

[0125] Furthermore, the resource level determination module 11 is also used to: if the current remaining power is less than the power threshold, then determine the target resource level as the basic resource level;

[0126] If the currently available memory is less than the memory threshold, then the target resource level is determined to be the basic resource level;

[0127] The application type is determined based on the current application package name, and when the application type is a development tool, the target resource level is determined as a professional resource level.

[0128] When the application type is an office application and a multi-turn dialogue is detected, the target resource level is determined as an enhanced resource level.

[0129] When the application type is a chat application, the target resource level is determined as an enhanced resource level;

[0130] The historical resource level corresponding to the current time is determined based on the historical resource level usage information, and the historical resource level is determined as the target resource level.

[0131] The resource optimization module 12 is used to compare the current resource level with the target resource level to obtain the resource comparison result, and to perform resource optimization processing on the end-side large model based on the resource comparison result.

[0132] Optionally, the resource optimization module 12 is also used to: serialize the key-value cache of the terminal-side large model to a temporary storage space, save the current dialogue context message sequence, and record the model configuration parameters;

[0133] Obtain the weight files of the target resource level and the current resource level to obtain the target weight file and the current weight file, and determine the resource loading increment based on the target weight file and the current weight file;

[0134] Create a shared memory file and incrementally map the resource load to the memory in the shared memory file;

[0135] Load the key-value cache into the temporary storage space, restore the current dialogue context message sequence, and apply the model configuration parameters;

[0136] Release the memory mapping of the current resource level, reclaim memory resources, and update memory usage statistics;

[0137] Send a resource optimization completion signal, update the hierarchical status, and record the switchover log.

[0138] Furthermore, the resource optimization module 12 is also used to: record the switching time, the current resource level, the target resource level, the switching reason, the triggering event type, and record the switching time, memory changes, and power consumption changes to obtain the switching log.

[0139] Furthermore, the resource optimization module 12 is also used to: if the current resource level is the same as the target resource level, maintain the current resource state and record the decision log;

[0140] If the current resource level is different from the target resource level, then resource optimization is required.

[0141] In this embodiment, resource optimization trigger detection is performed on the large-scale model on the edge to determine whether resource optimization operations need to be performed on the large-scale model on the edge, which improves the accuracy of resource optimization on the large-scale model on the edge. By obtaining the current resource optimization event information, the target resource level can be effectively determined. By comparing the current resource level with the target resource level, a resource comparison result can be effectively generated. Based on the resource comparison result, resource optimization processing of the large-scale model on the edge can be automatically performed without quantization compression of the large-scale model on the edge, without reducing the performance of the large-scale model on the edge, thereby improving the user experience.

[0142] Example 3

[0143] Figure 3 This is a structural block diagram of a terminal device 2 provided in the third embodiment of this application. For example... Figure 3As shown, the terminal device 2 in this embodiment includes a processor 20, a memory 21, and a computer program 22 stored in the memory 21 and executable on the processor 20, such as a program for a large-scale resource optimization method for edge devices. When the processor 20 executes the computer program 22, it implements the steps in each embodiment of the various edge-scale resource optimization methods described above.

[0144] For example, the computer program 22 may be divided into one or more modules, which are stored in the memory 21 and executed by the processor 20 to complete this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 22 in the terminal device 2. The terminal device may include, but is not limited to, the processor 20 and the memory 21.

[0145] The processor 20 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0146] The memory 21 can be an internal storage unit of the terminal device 2, such as a hard drive or memory of the terminal device 2. The memory 21 can also be an external storage device of the terminal device 2, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the terminal device 2. Furthermore, the memory 21 can include both internal and external storage units of the terminal device 2. The memory 21 is used to store the computer program and other programs and data required by the terminal device. The memory 21 can also be used to temporarily store data that has been output or will be output.

[0147] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0148] If an integrated module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. This computer-readable storage medium can be non-volatile or volatile. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable storage medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the contents of a computer-readable storage medium may be appropriately added to or subtracted from the contents as required by the legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, a computer-readable storage medium may not include electrical carrier signals and telecommunication signals.

[0149] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for optimizing resources in a large edge model, characterized in that, The method includes: Resource optimization trigger detection is performed on the large model on the endpoint. If the resource optimization trigger detection is qualified, the current resource optimization event information is obtained, and the target resource level is determined based on the current resource optimization event information; The current resource level is compared with the target resource level to obtain the resource comparison result, and the resource optimization process is performed on the end-side large model based on the resource comparison result.

2. The end-side large model resource optimization method as described in claim 1, characterized in that, Resource optimization trigger detection is performed on the large model on the endpoint, including: Broadcast monitoring is performed on the process manager corresponding to the large end-side model to obtain activity broadcast monitoring information, and the foreground application package name change information is determined based on the activity broadcast monitoring information; When the front-end application package name in the front-end application package name change information changes, the resource optimization trigger detection is deemed qualified. Broadcast monitoring is performed on the battery manager corresponding to the large end-side model to obtain battery broadcast monitoring information, and the change value of battery percentage is determined based on the battery broadcast monitoring information; If the change in battery percentage is greater than the battery change threshold, the resource optimization trigger detection is deemed to be qualified. The large-scale model on the endpoint is monitored for memory warning broadcasts to obtain memory warning monitoring information, and the currently available memory is determined based on the memory warning monitoring information. If the current available memory is less than the memory threshold, then the resource optimization trigger detection is deemed qualified. Obtain the current time, and if the current time is within a preset time range, determine that the resource optimization trigger detection is qualified; The current network status is obtained, and if the current network status meets the preset network conditions, the resource optimization trigger detection is deemed qualified.

3. The end-side large model resource optimization method as described in claim 1, characterized in that, Obtain information about current resource optimization events, including: Get the current remaining battery level from the battery manager, and get the current available memory and the current application package name from the process manager; Obtain the historical hierarchical usage information corresponding to the large model on the edge, obtain the current network type according to the network manager, and obtain the temperature information according to the heat manager; The current resource optimization event information is generated based on the current remaining battery power, the current available memory, the current application package name, the historical tier usage information, the current network type, and the temperature information.

4. The end-side large model resource optimization method as described in claim 3, characterized in that, Determining the target resource level based on the current resource optimization event information includes: If the current remaining power is less than the power threshold, then the target resource level is determined to be the basic resource level; If the currently available memory is less than the memory threshold, then the target resource level is determined to be the basic resource level; The application type is determined based on the current application package name, and when the application type is a development tool, the target resource level is determined as a professional resource level. When the application type is an office application and a multi-turn dialogue is detected, the target resource level is determined as an enhanced resource level. When the application type is a chat application, the target resource level is determined as an enhanced resource level; The historical resource level corresponding to the current time is determined based on the historical resource level usage information, and the historical resource level is determined as the target resource level.

5. The end-side large model resource optimization method as described in claim 1, characterized in that, Based on the resource comparison results, resource optimization processing is performed on the large-scale end-side model, including: The key-value cache of the large model on the terminal side is serialized to a temporary storage space, and the current dialogue context message sequence is saved, and the model configuration parameters are recorded. Obtain the weight files of the target resource level and the current resource level to obtain the target weight file and the current weight file, and determine the resource loading increment based on the target weight file and the current weight file; Create a shared memory file and incrementally map the resource load to the memory in the shared memory file; Load the key-value cache into the temporary storage space, restore the current dialogue context message sequence, and apply the model configuration parameters; Release the memory mapping of the current resource level, reclaim memory resources, and update memory usage statistics; Send a resource optimization completion signal, update the hierarchical status, and record the switchover log.

6. The end-side large model resource optimization method as described in claim 5, characterized in that, Record the switchover log, including: The switching log is obtained by recording the switching time, the current resource level, the target resource level, the switching reason, the triggering event type, the switching time, memory changes, and power consumption changes.

7. The end-side large model resource optimization method as described in claim 1, characterized in that, The current resource level is compared with the target resource level to obtain the resource comparison result, including: If the current resource level is the same as the target resource level, then maintain the current resource state and record the decision log; If the current resource level is different from the target resource level, then resource optimization is required.

8. A large-scale end-model resource optimization system, characterized in that, The system includes: The trigger detection module is used to perform resource optimization trigger detection on large models on the edge side; The resource level determination module is used to obtain the current resource optimization event information and determine the target resource level based on the current resource optimization event information if the resource optimization trigger detection is qualified. The resource optimization module is used to compare the current resource level with the target resource level to obtain the resource comparison result, and to perform resource optimization processing on the end-side large model based on the resource comparison result.

9. The end-side large model resource optimization system as described in claim 8, characterized in that, The trigger detection module is also used for: Broadcast monitoring is performed on the process manager corresponding to the large end-side model to obtain activity broadcast monitoring information, and the foreground application package name change information is determined based on the activity broadcast monitoring information; When the front-end application package name in the front-end application package name change information changes, the resource optimization trigger detection is deemed qualified. Broadcast monitoring is performed on the battery manager corresponding to the large end-side model to obtain battery broadcast monitoring information, and the change value of battery percentage is determined based on the battery broadcast monitoring information; If the change in battery percentage is greater than the battery change threshold, the resource optimization trigger detection is deemed to be qualified. The large-scale model on the endpoint is monitored for memory warning broadcasts to obtain memory warning monitoring information, and the currently available memory is determined based on the memory warning monitoring information. If the current available memory is less than the memory threshold, then the resource optimization trigger detection is deemed qualified. Obtain the current time, and if the current time is within a preset time range, determine that the resource optimization trigger detection is qualified; The current network status is obtained, and if the current network status meets the preset network conditions, the resource optimization trigger detection is deemed qualified.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images