Large language model reasoning optimization method and device, storage medium and computer equipment

Abstract

According to the large language model reasoning optimization method and device, the storage medium and the computer equipment provided by the invention, the multi-dimensional monitoring data is acquired, and then the multi-dimensional monitoring data is input into the pre-trained environment decision network to obtain the optimal model configuration. The current resource state is evaluated according to the multi-dimensional monitoring data, and the optimal execution path is determined according to the resource state and the optimal model configuration. And finally, adjusting the target model based on the optimal model configuration, and performing model reasoning according to the optimal execution path in the reasoning process of the target model. In the process, the resource state is evaluated and the optimal model configuration is determined by collecting the multi-dimensional detection data, and then the optimal execution path is dynamically determined, so that the model structure and the calculation strategy can be dynamically adjusted according to the state during actual operation, and the response delay is effectively reduced and the resource utilization efficiency is improved while the reasoning precision is guaranteed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and particularly relates to a large language model inference optimization method and device, a storage medium and a computer device. BACKGROUND

[0002] Under the background of continuous expansion of the scale of the current large language model and continuous extension of the application scene to the edge side, how to realize efficient and stable inference on resource-constrained terminal devices has become a key challenge. The rapid growth of model parameter quantity and computing demand makes it difficult for traditional deployment methods to meet the actual needs of low delay and high response under the condition of limited computing power and memory.

[0003] The current mainstream optimization method mainly depends on parameter adjustment strategies based on rules or offline optimization. Although such methods have the advantage of easy implementation, they lack real-time perception ability of the runtime resource state and cannot dynamically adjust according to the actual resource state, resulting in problems such as high delay and insufficient resource utilization efficiency when deployed on mobile terminals and edge sides. SUMMARY

[0004] The present application aims to at least solve one of the above technical defects, particularly the technical defect that the prior art mainly depends on parameter adjustment strategies based on rules or offline optimization. Although such methods have the advantage of easy implementation, they lack real-time perception ability of the runtime resource state and cannot dynamically adjust according to the actual resource state.

[0005] In a first aspect, the present application provides a large language model inference optimization method, which comprises:

[0006] acquiring multi-dimensional monitoring data;

[0007] inputting the multi-dimensional monitoring data into a pre-trained environment decision network to obtain an optimal model configuration;

[0008] evaluating a current resource state according to the multi-dimensional monitoring data, and determining an optimal execution path according to the resource state and the optimal model configuration;

[0009] adjusting a target model based on the optimal model configuration, and performing model inference according to the optimal execution path in the inference process of the target model.

[0010] In one embodiment, the acquiring multi-dimensional monitoring data comprises:

[0011] determining a target model;

[0012] collecting hardware sensor data in a device in which the target model is deployed, and acquiring user behavior data from a preset log system;

[0013] obtain a historical input text set of the target model, and perform feature extraction on the historical input text set to obtain text features;

[0014] aggregate the hardware sensor data, the user behavior data, and the text features to obtain multi-dimensional monitoring data.

[0015] In one of the embodiments, the training process of the environment decision network comprises:

[0016] obtain multi-dimensional sample data and a corresponding optimal configuration set thereof;

[0017] input the multi-dimensional sample data into a preset initial decision network to iteratively train the initial decision network, and in the iterative training process, construct a loss function according to output data of the initial decision network and the optimal configuration set, and update the initial decision network based on the loss function;

[0018] when a preset training condition is met, fine-tune the current initial decision network using a preset adaptive target function, and determine the initial decision network after fine-tuning as the environment decision network.

[0019] In one of the embodiments, the adaptive target function takes minimizing a weighted combination of inference delay and device energy consumption as an optimization target.

[0020] In one of the embodiments, the evaluation of the current resource state according to the multi-dimensional monitoring data, and the determination of an optimal execution path according to the resource state and the optimal model configuration, comprise:

[0021] extract hardware sensor data from the multi-dimensional monitoring data, and evaluate the current resource state based on the hardware sensor data;

[0022] determine a device performance indicator according to the resource state and the optimal model configuration;

[0023] select a target processing unit that best matches the device performance indicator, and generate an optimal execution path according to the target processing unit;

[0024] when a change in the device performance indicator is detected to exceed an expectation, reselect a target processing unit that best matches the changed device performance indicator, and update the optimal execution path.

[0025] In one of the embodiments, the adjustment of the target model based on the optimal model configuration, and the model inference in the inference process of the target model according to the optimal execution path, comprise:

[0026] adjusting, according to the optimal model configuration, a parameter in the target model corresponding to the optimal model configuration;

[0027] In the inference process of the target model, the latest optimal execution path is obtained, and model inference is performed based on a processing unit corresponding to the optimal execution path.

[0028] In one of the embodiments, the method further comprises:

[0029] In the inference process of the target model, the hidden states of the previous several layers are fused to determine the semantic information of the current layer;

[0030] A gating mechanism is set to dynamically fuse the semantic information of different layers to obtain fused semantic information;

[0031] The fused semantic information is subjected to layer normalization processing to output enhanced semantic information, and until the inference is completed, the output content of the target model is determined according to the last output enhanced semantic information.

[0032] In a second aspect, the present application provides a large language model inference optimization device, the device comprising:

[0033] A data acquisition module is configured to acquire multi-dimensional monitoring data;

[0034] A configuration determination module is configured to input the multi-dimensional monitoring data into a pre-trained environment decision network to obtain an optimal model configuration;

[0035] A state evaluation module is configured to evaluate a current resource state according to the multi-dimensional monitoring data, and determine an optimal execution path according to the resource state and the optimal model configuration;

[0036] An inference optimization module is configured to adjust a target model based on the optimal model configuration, and perform model inference according to the optimal execution path in the inference process of the target model.

[0037] In a third aspect, the present application provides a storage medium, the storage medium storing computer readable instructions, the computer readable instructions being executed by one or more processors to cause the one or more processors to perform the steps of the large language model inference optimization method according to any one of the above embodiments.

[0038] In a fourth aspect, the present application provides a computer device, comprising one or more processors and a memory;

[0039] The memory stores computer readable instructions, and the one or more processors execute the computer readable instructions to perform the steps of the large language model inference optimization method according to any one of the above embodiments.

[0040] From the above technical solutions, the embodiments of the present application have the following advantages:

[0041] The large language model inference optimization method, device, storage medium and computer equipment provided by the present application acquire multi-dimensional monitoring data, then input the multi-dimensional monitoring data into a pre-trained environment decision network to obtain an optimal model configuration. Then, the current resource state is evaluated according to the multi-dimensional monitoring data, and the optimal execution path is determined according to the resource state and the optimal model configuration. Finally, the target model is adjusted based on the optimal model configuration, and model inference is performed according to the optimal execution path in the inference process of the target model. In the above process, the resource state is evaluated and the optimal model configuration is determined by collecting multi-dimensional detection data, and then the optimal execution path is dynamically determined, which can dynamically adjust the model structure and the calculation strategy according to the actual running state, thereby effectively reducing the response delay while ensuring the inference accuracy, and improving the resource utilization efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 A flowchart of a large language model inference optimization method provided by an embodiment of the present application;

[0044] Figure 2 An example diagram of a large language model inference optimization method provided by an embodiment of the present application;

[0045] Figure 3 An example diagram of semantic association of cross-layer attention understanding provided by an embodiment of the present application;

[0046] Figure 4 A structural diagram of a large language model inference optimization device provided by an embodiment of the present application;

[0047] Figure 5 An internal structure diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] With reference to the drawings and the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0049] In one of the embodiments, the present application provides a large language model inference optimization method. The following embodiments are described with the method applied to a server. It can be understood that the large language model inference optimization method can be executed by a single server or a server cluster composed of multiple servers, and the present application does not make specific limitations thereto.

[0050] As shown in Figure 1 The present application provides a large language model inference optimization method, which comprises:

[0051] S101: Obtain multi-dimensional monitoring data.

[0052] The multi-dimensional monitoring data comprises hardware sensor data, user behavior data, and text feature data.

[0053] In this step, when interacting with a large language model for question and answer, the relevant data of the large language model and its deployed device can be obtained through a monitoring platform to form multi-dimensional monitoring data. Specifically, the hardware sensor data of the deployed device can be obtained, and the user behavior data and text feature data of the large language model can be obtained. Then, the multi-dimensional monitoring data is generated according to the obtained hardware sensor data, user behavior data, and text feature data.

[0054] Further, when the hardware sensor data, user behavior data, and text feature data are obtained, the data can be preprocessed first, such as data cleaning, normalization, etc., to determine the accuracy of the data. Then, the multi-dimensional monitoring data is generated according to the preprocessed data.

[0055] S102: Input the multi-dimensional monitoring data into a pre-trained environment decision network to obtain an optimal model configuration.

[0056] The environment decision network is used to determine the optimal model configuration corresponding to the multi-dimensional monitoring data. The optimal model configuration refers to the optimal configuration of the large language model in the current environment. The optimal model configuration includes but is not limited to batch size, model layer number, attention head number, etc.

[0057] In this step, after determining the multi-dimensional monitoring data, the multi-dimensional monitoring data is input into the pre-trained environment decision network to obtain the optimal model configuration output by the environment decision network. The environment decision network can calculate the optimal model configuration suitable for the environment represented by the multi-dimensional monitoring data based on the inference mechanism embedded therein.

[0058] In one example, the inference process of the environment decision network can be represented as follows:

[0059]

[0060] In the formula, represents the model layer, represents the batch size, represents the environment decision network, represents the parameters of the environment decision network, represents the memory occupancy rate, represents the CPU / GPU temperature, represents the request frequency, represents the text complexity. Among them, the memory occupancy rate and the CPU / GPU temperature are derived from hardware sensor data, the request frequency is derived from user behavior data, and the text complexity is derived from text feature data.

[0061] S103: According to the multi-dimensional monitoring data, the current resource state is evaluated, and according to the resource state and the optimal model configuration, the optimal execution path is determined.

[0062] Among them, the resource state is used to measure the resource usage of the device deployed by the large language model or the target model. The optimal execution path refers to the processing unit that can maximize the resource utilization efficiency under the given resource state and the optimal model configuration.

[0063] In this step, the resource situation of the device where the large language model is located is determined according to the multi-dimensional monitoring data, so as to evaluate the current resource state. Then, the most suitable processing unit is matched according to the resource state and the optimal model configuration, so as to fully utilize the device resources.

[0064] Specifically, in the evaluation of the resource state, corresponding evaluation data can be obtained from the multi-dimensional monitoring data, and then the resource state can be further evaluated according to the pre-defined rules or pre-trained neural network, which is not limited in the present application. In addition, in the determination of the optimal execution path according to the resource state and the optimal model configuration, the matching determination can be performed according to the pre-constructed knowledge graph, which is constructed offline through historical performance data and is used to record the performance indicators of each processing unit under different loads. Through the resource state and the optimal model configuration, the load condition can be determined, and then the processing unit with the optimal performance under the load condition is matched in the knowledge graph to generate the optimal execution path.

[0065] S104: Adjust the target model based on the optimal model configuration, and perform model inference in the inference process of the target model according to the optimal execution path.

[0066] The target model refers to a large language model, i.e., the model for which inference optimization is performed in the present method.

[0067] In this step, when the optimal model configuration and the optimal execution path are determined, the corresponding parameters in the target model are adjusted based on the optimal model configuration, and in the inference process of the target model, the processing unit corresponding to the optimal execution path is used as the execution device for the inference task to perform model inference. It can be understood that the model configuration and the optimal execution path are adjusted according to the environment during the inference process, so that the inference process fits the current environmental resource condition and maximizes the resource utilization efficiency.

[0068] In the above embodiment, multi-dimensional monitoring data is obtained, and then the multi-dimensional monitoring data is input into a pre-trained environment decision network to obtain an optimal model configuration. Then, the current resource state is evaluated according to the multi-dimensional monitoring data, and the optimal execution path is determined according to the resource state and the optimal model configuration. Finally, the target model is adjusted based on the optimal model configuration, and model inference is performed according to the optimal execution path in the inference process of the target model. In the above process, the resource state is evaluated by collecting multi-dimensional monitoring data to determine the optimal model configuration, and then the optimal execution path is dynamically determined, which can dynamically adjust the model structure and the calculation strategy according to the actual running state, thereby effectively reducing the response delay while ensuring the inference accuracy and improving the resource utilization efficiency.

[0069] In one embodiment, the multi-dimensional monitoring data is obtained, including:

[0070] S1: Determine the target model.

[0071] S2: Collect hardware sensor data in the device where the target model is deployed, and obtain user behavior data from a pre-set log system.

[0072] S3: Obtain a historical input text set of the target model and perform feature extraction on the historical input text set to obtain text features.

[0073] S4: Aggregate hardware sensor data, user behavior data, and text features to obtain multi-dimensional monitoring data.

[0074] The hardware sensor data refers to data collected by hardware sensors arranged in the device where the target model is deployed, including but not limited to memory usage, CPU / GPU temperature, battery level, etc., for reflecting the device running state and environmental conditions. The user behavior data refers to data generated during user interaction with the target model, including but not limited to request frequency, input text length, historical response time, etc. The text features refer to structured representations that can represent text content, structure, or semantic information, including but not limited to sentence complexity, entity number, semantic dependency depth, etc. The historical input text set refers to a collection of texts received by the target model in the past period of time.

[0075] In this embodiment, after determining the target model, hardware sensor data in the device where the target model is deployed needs to be collected first. Specifically, the data monitored by each hardware sensor can be obtained through the detection platform. Then, log data is obtained from a pre-set log system, and the log data is analyzed to extract information related to user behavior to generate user behavior data. Next, feature extraction can be performed on the historical input text set of the target model to obtain text features that can represent the semantics and structure of past input texts. Finally, hardware sensor data, user behavior data, and text features are aggregated to obtain multi-dimensional monitoring data.

[0076] In one embodiment, the training process of the environment decision network includes:

[0077] S1: Obtain multi-dimensional sample data and its corresponding optimal configuration set.

[0078] S2: Input the multi-dimensional sample data into a pre-set initial decision network to iteratively train the initial decision network. During the iterative training process, a loss function is constructed based on the output data of the initial decision network and the optimal configuration set, and the initial decision network is updated based on the loss function.

[0079] S3: When the pre-set training condition is met, the current initial decision network is fine-tuned using a pre-set adaptive target function, and the fine-tuned initial decision network is determined as the environment decision network.

[0080] The multi-dimensional sample data refers to a collection of past multi-dimensional monitoring data, and the optimal configuration set refers to a collection of optimal model configurations corresponding to each multi-dimensional monitoring data in the multi-dimensional sample data.

[0081] In this embodiment, when the multi-dimensional sample data and its corresponding optimal configuration set are obtained, the multi-dimensional sample data is input into a preset initial decision network. The initial decision network is a model based on machine learning, which can receive input data and output decision results. In the iterative training process, a loss function is constructed according to the output data of the initial decision network and the optimal configuration set. The loss function is used to measure the difference between the model output and the real optimal configuration. By minimizing the loss function, the parameters of the initial decision network can be optimized.

[0082] In each iteration, the initial decision network is updated based on the loss function. Specifically, the gradient of the loss function with respect to the network parameters can be calculated, and the parameters of the initial decision network can be adjusted to reduce the loss value. This process is repeated until a preset training condition is met, such as reaching a certain number of iterations or the loss value being lower than a certain threshold. When the preset training condition is met, the current initial decision network is fine-tuned using a preset adaptive target function. The adaptive target function can optimize the inference delay and device energy consumption of the decision network, further improving the performance of the network. The initial decision network after fine-tuning is finally determined as the environment decision network for actual decision tasks.

[0083] Specifically, the preset training condition refers to the condition for stopping training. The preset training condition can be set to a value of the loss function less than a preset value, or the absolute value of the difference between the loss functions of two iterations less than a preset value. The present application does not make specific limitations.

[0084] In one example, the Adam optimizer can be used for end-to-end training, and the loss function can use the mean square error loss. The loss function can be represented as:

[0085]

[0086] In the formula, represents the loss function, represents the number of samples in the multi-dimensional sample data, represents the i-th multi-dimensional monitoring data in the multi-dimensional sample data, represents the optimal model configuration corresponding to the i-th multi-dimensional monitoring data output by the environment decision network, is the optimal model configuration corresponding to the i-th multi-dimensional monitoring data in the optimal configuration set.

[0087] In one embodiment, the adaptive target function aims to minimize the weighted combination of inference delay and device energy consumption. In one example, the adaptive target function can be represented as follows:

[0088]

[0089] wherein, , denotes the weight, denotes the inference delay, denotes the device energy consumption. denotes the parameter when minimizing the adaptive target function. In addition, and the sum is 1, , is the maximum delay limit.

[0090] In one embodiment, the current resource state is evaluated according to multi-dimensional monitoring data, and the optimal execution path is determined according to the resource state and the optimal model configuration, including:

[0091] S1: Extracting hardware sensor data from multi-dimensional monitoring data, and evaluating the current resource state based on the hardware sensor data.

[0092] S2: Determining the device performance index according to the resource state and the optimal model configuration.

[0093] S3: Selecting the target processing unit that best matches the device performance index, and generating the optimal execution path according to the target processing unit.

[0094] S4: When detecting that the change of the device performance index exceeds the expectation, reselecting the target processing unit that best matches the changed device performance index, and updating the optimal execution path.

[0095] Wherein, the device performance index refers to a series of parameters used to measure the efficiency and quality of device operation. The target processing unit refers to a computing resource or processing module in the device for executing specific tasks, such as GPU, CPU, NPU, etc.

[0096] In this embodiment, hardware sensor data is extracted from multi-dimensional monitoring data, and then the current device resource state can be evaluated based on the hardware sensor data through pre-set evaluation rules or pre-trained deep learning models. Then the pre-constructed knowledge graph can be obtained, the device performance index is matched in the knowledge graph, and the processing unit represented by the closest path is determined as the target processing unit, and the path is determined as the optimal execution path. It should be noted that the device referred to in this application refers to a device that deploys a large language model or a target model.

[0097] During the inference process of the target model, when a change in device performance metrics is detected that exceeds expectations, the target processing unit that best matches the changed device performance metrics can be reselected, and the optimal execution path can be updated accordingly. Furthermore, after updating the optimal execution path, a path switch can be triggered, and model inference can be performed according to the updated optimal execution path.

[0098] In one example, device performance metrics can be obtained from one or more combinations of metrics such as memory usage and temperature. When device performance metrics consist of multiple metrics, to determine whether a change exceeds expectations, the average change can be calculated, or an expected change value can be set for each metric, and then each metric can be evaluated individually. This application does not impose specific limitations in this regard.

[0099] It is understandable that selecting the target processing unit that best matches the device's performance metrics ensures that the task is executed on the most suitable resources, improving the task's efficiency and quality. When device performance changes, promptly reselecting the target processing unit and updating the execution path ensures the device's dynamic adaptability and avoids model inference failures or delays due to degraded device performance.

[0100] In one embodiment, the target model is adjusted based on the optimal model configuration, and model inference is performed according to the optimal execution path during the inference process of the target model, including:

[0101] S1: Adjust the parameters in the target model that correspond to the optimal model configuration based on the optimal model configuration.

[0102] S2: During the reasoning process of the target model, obtain the latest optimal execution path and perform model reasoning based on the processing unit corresponding to the optimal execution path.

[0103] In this embodiment, the parameters in the target model corresponding to the configurations included in the optimal model configuration are adjusted sequentially. Then, during the inference process of the target model, the latest optimal execution path is obtained, and model inference is performed based on the processing unit corresponding to the latest optimal execution path. Specifically, the optimal execution path is determined based on the current resource status and model configuration. Selecting the most suitable processing unit to perform model inference can optimize resource utilization and reduce latency. For example, if the target model requires high computing power, a high-performance GPU may be selected. If the target model requires fast response, a low-latency CPU may be selected.

[0104] In one embodiment, such as Figure 2 As shown, Figure 2 This is an example diagram illustrating a large language model inference optimization method provided in an embodiment of this application. Figure 2In this embodiment, the application can further determine whether the resource is limited after obtaining the multi-dimensional monitoring data. If the resource is limited, the optimal model configuration is determined based on the environment decision network. If the resource is not limited, the current model configuration can be used to perform model inference.

[0105] In one of the embodiments, the large language model inference optimization method further includes:

[0106] S1: In the inference process of the target model, the hidden states of the previous several layers are fused to determine the semantic information of the current layer.

[0107] S2: A gating mechanism is set to dynamically fuse the semantic information of different layers to obtain fused semantic information.

[0108] S3: The fused semantic information is subjected to layer normalization processing to output enhanced semantic information, and the output content of the target model is determined according to the last output enhanced semantic information when the inference is completed.

[0109] In this embodiment, the semantic information of different layers is understood and fused through cross-layer attention, and the enhanced semantic information is finally output. This can effectively capture long-range semantic associations, avoid semantic ambiguity or semantic loss, and improve the inference accuracy of the model.

[0110] In one example, the above semantic determination, fusion and enhancement process can be represented as the following processes:

[0111] (1) Cross-layer global attention calculation

[0112] In each layer of the Transformer structure, a cross-layer attention mechanism is introduced. In addition to the current layer Query (Q), Key (K), and Value (V), the hidden states of the previous several layers are also fused, and the calculation form is as follows:

[0113]

[0114] In the formula, is the hidden state of the previous layer or multiple layers, which is used to enhance the current semantic representation, represents the dimension size of the Key vector.

[0115] (2) Gating semantic state fusion

[0116] A gating mechanism is designed to dynamically fuse the semantic information of different layers. Specifically, the semantic information of the current layer can be fused with the semantic information of the previous layer or multiple layers to avoid irrelevant information interference. The expression of this gating mechanism can be represented as:

[0117]

[0118]

[0119] In the formula, express, This represents the Sigmoid function. , Indicates trainable parameters, Indicates the hidden state of the current layer. This represents the final hidden state after fusion via the gating mechanism. This indicates element-wise multiplication.

[0120] (3) Enhance semantic representation output

[0121] The fused semantic states are then subjected to layer normalization to output enhanced word vectors. This process can be represented as follows:

[0122]

[0123] In the formula, Representation layer normalization, Represents enhanced semantic information. This indicates the fusion of semantic information.

[0124] In one example, such as Figure 3 As shown, Figure 3 This diagram illustrates the semantic association example of cross-layer attention understanding provided in this application embodiment. It shows the semantic association captured by the model through a cross-layer attention mechanism when a user queries "Can diabetic patients consume honey?". The red nodes (diabetes, honey) are the core entities of the query, and the green nodes (glycemic index, glycemic index) are key medical concepts strongly associated with these core entities. The values ​​on the connecting lines represent attention weights; higher weights indicate stronger semantic associations. This demonstrates that the model successfully understands the medical connection between "honey" and "diabetes" established through "glycemic index".

[0125] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0126] The following describes a large language model inference optimization device provided by the embodiments of the present application. The large language model inference optimization device described below can correspond to the large language model inference optimization method described above.

[0127] As shown in Figure 4 The present application provides a large language model inference optimization device 200, which comprises:

[0128] A data acquisition module 201 is configured to acquire multi-dimensional monitoring data.

[0129] A configuration determination module 202 is configured to input the multi-dimensional monitoring data into a pre-trained environment decision network to obtain an optimal model configuration.

[0130] A state evaluation module 203 is configured to evaluate a current resource state according to the multi-dimensional monitoring data, and determine an optimal execution path according to the resource state and the optimal model configuration.

[0131] An inference optimization module 204 is configured to adjust a target model based on the optimal model configuration, and perform model inference according to the optimal execution path in the inference process of the target model.

[0132] In the above embodiments, multi-dimensional monitoring data is acquired, and then the multi-dimensional monitoring data is input into a pre-trained environment decision network to obtain an optimal model configuration. Then, a current resource state is evaluated according to the multi-dimensional monitoring data, and an optimal execution path is determined according to the resource state and the optimal model configuration. Finally, a target model is adjusted based on the optimal model configuration, and model inference is performed according to the optimal execution path in the inference process of the target model. In the above process, the resource state is evaluated and the optimal model configuration is determined by collecting multi-dimensional monitoring data, and then the optimal execution path is dynamically determined. In this way, the model structure and the calculation strategy can be dynamically adjusted according to the actual running state, so as to effectively reduce the response delay while ensuring the inference accuracy, and improve the resource utilization efficiency.

[0133] In one of the embodiments, the data acquisition module comprises:

[0134] A model determination sub-module is configured to determine a target model.

[0135] A data acquisition sub-module is configured to collect hardware sensor data in a device where the target model is deployed, and acquire user behavior data from a preset log system.

[0136] A feature extraction sub-module is configured to acquire a historical input text set of the target model, and perform feature extraction on the historical input text set to obtain text features.

[0137] The data aggregation sub-module is configured to aggregate the hardware sensor data, the user behavior data, and the text features to obtain multi-dimensional monitoring data.

[0138] In one of the embodiments, the configuration determination module comprises:

[0139] The configuration acquisition sub-module is configured to acquire the multi-dimensional sample data and the corresponding optimal configuration set.

[0140] The model training sub-module is configured to input the multi-dimensional sample data into a preset initial decision network to iteratively train the initial decision network, and to update the initial decision network based on a loss function constructed according to the output data of the initial decision network and the optimal configuration set during the iterative training.

[0141] The model fine-tuning sub-module is configured to fine-tune the current initial decision network using a preset adaptive target function when a preset training condition is met, and to determine the initial decision network after the fine-tuning as the environment decision network.

[0142] In one of the embodiments, the adaptive target function takes minimizing a weighted combination of inference delay and device energy consumption as an optimization target.

[0143] In one of the embodiments, the state evaluation module comprises:

[0144] The state evaluation sub-module is configured to extract the hardware sensor data from the multi-dimensional monitoring data, and to evaluate the current resource state based on the hardware sensor data.

[0145] The index determination sub-module is configured to determine a device performance index according to the resource state and the optimal model configuration.

[0146] The path generation sub-module is configured to select a target processing unit that best matches the device performance index, and to generate an optimal execution path according to the target processing unit.

[0147] The path update sub-module is configured to reselect a target processing unit that best matches a changed device performance index when detecting that the change of the device performance index exceeds an expectation, and to update the optimal execution path.

[0148] In one of the embodiments, the inference optimization module comprises:

[0149] The parameter determination sub-module is configured to adjust parameters in a target model corresponding to the optimal model configuration according to the optimal model configuration.

[0150] The model inference sub-module is configured to acquire the latest optimal execution path during inference of the target model, and to perform model inference based on a processing unit corresponding to the optimal execution path.

[0151] In one of the embodiments, the large language model inference optimization apparatus further comprises:

[0152] a semantic determination module configured to fuse hidden states of previous layers to determine semantic information of a current layer in an inference process of the target model;

[0153] a semantic fusion module configured to set a gating mechanism to dynamically fuse semantic information of different layers to obtain fused semantic information;

[0154] a semantic enhancement module configured to perform layer normalization processing on the fused semantic information to output enhanced semantic information, and determine output content of the target model according to the last output enhanced semantic information when the inference is completed.

[0155] The division of each module in the large language model inference optimization apparatus is only used for illustration, and in other embodiments, the large language model inference optimization apparatus can be divided into different modules as needed to complete all or part of the functions of the large language model inference optimization apparatus. Each module in the large language model inference optimization apparatus can be implemented in whole or in part by software, hardware, and combinations thereof. Each module described above can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0156] In one embodiment, the present application also provides a storage medium having computer readable instructions stored therein, and the computer readable instructions, when executed by one or more processors, cause the one or more processors to perform the steps of the large language model inference optimization method according to any one of the above embodiments.

[0157] In one embodiment, the present application also provides a computer device having computer readable instructions stored therein, and the computer readable instructions, when executed by one or more processors, cause the one or more processors to perform the steps of the large language model inference optimization method according to any one of the above embodiments.

[0158] As shown in Figure 5 , Figure 5 Fig. 2 is a schematic diagram of an internal structure of a computer device according to an embodiment of the present application. The computer device 300 can be provided as a server. Referring to Figure 5The computer device 300 includes a processing component 302, further including one or more processors, and a memory resource represented by the memory 301, for storing instructions executable by the processing component 302, such as an application program. The application program stored in the memory 301 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 302 is configured to execute the instructions to perform the large language model inference optimization method of any of the embodiments described above.

[0159] The computer device 300 can also include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 can operate based on an operating system stored in the memory 301, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.

[0160] Those skilled in the art can understand that Figure 5 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0161] Finally, it should be noted that the terms "first" and "second", and the like, herein do not denote any order, quantity, combination or important / primary / secondary status, but are used to distinguish one element from another, and do not denote or imply any actual relationship or sequence among such elements. Also, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element. The terms "a" and "an" and "the" and similar referents in the context of this disclosure are to be construed to be inclusive of both singular and plural, unless the context clearly dictates otherwise. It will be further understood that the terms "comprises", "comprising", "has", "having", "includes" and / or "including", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. The terms "and / or" and "or" as used herein refer to and encompass any and all possible combinations of one or more of the associated items, including but not limited to any one of the associated items alone, any two of the associated items, all of the associated items, and any permutation and combination thereof.

[0162] The various embodiments described in this specification are presented by way of example, and are not intended to limit the scope of the disclosure. Each embodiment is presented in a way that emphasizes its particular features, and the embodiments can be combined according to the needs of the user.

[0163] The above description of disclosed embodiments provides examples, and is not intended to limit the scope, applicability, or configuration of the disclosure. Various modifications can occur to those skilled in the art to which the disclosure relates, and such modifications are intended to be included within the scope of the disclosure. It is intended that the disclosure be construed as including all such modifications and equivalents. Accordingly, the application is not to be restricted, except in accordance with the following claims and their equivalents.

Claims

1. A large language model inference optimization method, characterized in that, The method comprises: acquiring multi-dimensional monitoring data; inputting the multi-dimensional monitoring data into a pre-trained environment decision network to obtain an optimal model configuration; evaluating a current resource state according to the multi-dimensional monitoring data, and determining an optimal execution path according to the resource state and the optimal model configuration; adjusting a target model based on the optimal model configuration, and performing model inference according to the optimal execution path in an inference process of the target model.

2. The large language model inference optimization method of claim 1, wherein, The acquisition of the multi-dimensional monitoring data comprises: determining a target model; collecting hardware sensor data in a device in which the target model is deployed, and acquiring user behavior data from a preset log system; acquiring a historical input text set of the target model, and performing feature extraction on the historical input text set to obtain text features; summarizing the hardware sensor data, the user behavior data and the text features to obtain multi-dimensional monitoring data.

3. The large language model inference optimization method of claim 1, wherein, The training process of the environment decision network comprises: acquiring multi-dimensional sample data and a corresponding optimal configuration set; inputting the multi-dimensional sample data into a preset initial decision network to iteratively train the initial decision network, and in the iterative training process, constructing a loss function according to output data of the initial decision network and the optimal configuration set, and updating the initial decision network based on the loss function; when a preset training condition is met, fine-tuning the current initial decision network by using a preset adaptive target function, and determining the initial decision network after fine-tuning as the environment decision network.

4. The large language model inference optimization method of claim 3, wherein, The adaptive target function takes minimizing a weighted combination of inference delay and device energy consumption as an optimization target.

5. The large language model inference optimization method of claim 1, wherein, The evaluation of the current resource state according to the multi-dimensional monitoring data, and the determination of the optimal execution path according to the resource state and the optimal model configuration, comprise: extracting hardware sensor data from the multi-dimensional monitoring data, and evaluating a current resource state based on the hardware sensor data; determining a device performance indicator according to the resource state and the optimal model configuration; selecting a target processing unit that best matches the device performance indicator, and generating an optimal execution path according to the target processing unit; when detecting that a change in the device performance indicator exceeds an expectation, reselecting a target processing unit that best matches the changed device performance indicator, and updating the optimal execution path.

6. The large language model inference optimization method of claim 1, wherein, The adjustment of the target model based on the optimal model configuration, and the performance of model inference according to the optimal execution path in the inference process of the target model, comprise: adjusting parameters in the target model corresponding to the optimal model configuration according to the optimal model configuration; in the inference process of the target model, acquiring a latest optimal execution path, and performing model inference based on a processing unit corresponding to the optimal execution path.

7. The large language model inference optimization method according to any one of claims 1 to 6, characterized in that, The method further comprises: in the inference process of the target model, fusing hidden states of previous layers to determine semantic information of a current layer; setting a gating mechanism to dynamically fuse semantic information of different layers to obtain fused semantic information; The fusion semantic information is subjected to layer normalization processing, and enhanced semantic information is output until the output content of the target model is determined according to the last output enhanced semantic information when the inference is completed.

8. A large language model inference optimization apparatus, characterized by, The device comprises: a data acquisition module configured to acquire multi-dimensional monitoring data; a configuration determination module configured to input the multi-dimensional monitoring data into a pre-trained environment decision network to obtain an optimal model configuration; a state evaluation module configured to evaluate a current resource state according to the multi-dimensional monitoring data, and determine an optimal execution path according to the resource state and the optimal model configuration; an inference optimization module configured to adjust a target model based on the optimal model configuration, and perform model inference according to the optimal execution path in the inference process of the target model.

9. A storage medium characterized by: The storage medium stores computer readable instructions, and the computer readable instructions are executed by one or more processors to cause the one or more processors to perform the steps of the large language model inference optimization method according to any one of claims 1 to 7.

10. A computer device, comprising: comprise: one or more processors, and a memory; the memory stores computer readable instructions, and the computer readable instructions are executed by the one or more processors to perform the steps of the large language model inference optimization method according to any one of claims 1 to 7.

Images