Large language model reasoning optimization method and device, equipment and storage medium

Abstract

The invention discloses a big language model reasoning optimization method and device, equipment and a storage medium, and relates to the technical field of machine learning, and the method comprises the steps: carrying out the structural pruning of an initial big language model, so as to obtain a to-be-adjusted big language model; wherein the to-be-adjusted large language model is a pre-trained model, and model quantization is completed in the model training stage; obtaining target to-be-reasoned data, judging whether the target to-be-reasoned data is smaller than a preset length threshold value or not, and if the target to-be-reasoned data is not smaller than the preset length threshold value, segmenting the target to-be-reasoned data into data fragments with target length; feature information of the target to-be-reasoned data is obtained, an initial context window in the to-be-adjusted large language model is automatically adjusted according to the feature information to obtain a corresponding target large language model, and the target large language model is utilized to conduct reasoning on the data fragments. The model is quantized, and the data is fragmented, so that the efficiency and expandability of model reasoning are improved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a large language model reasoning optimization method, device, equipment and storage medium. Background Art

[0002] LLM (Large Language Model) is a deep learning model trained with large amounts of text data that can generate natural language text or understand the meaning of language text. These models are trained on large data sets to learn the patterns and structures of natural language, thus being able to simulate the human language cognition and generation process to a certain extent.

[0003] With the development of deep learning, large-scale language models have attracted widespread attention due to their powerful natural language processing capabilities. However, in practical applications, the inference stage of these models has extremely high demand for computing resources, especially in application scenarios with high real-time requirements, such as online customer service, smart assistants, etc. Existing methods are insufficient in scalability and efficiency, and cannot fully utilize modern hardware resources such as GPUs (Graphics Processing Units) and TPUs (Tensor Processing Units) for efficient parallel computing. Therefore, how to use large language models for efficient and scalable data reasoning has become a technical problem that needs to be solved. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a large language model reasoning optimization method, device, equipment and storage medium, which can quantize the model and slice the data, thereby improving the efficiency and scalability of model reasoning. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a large language model reasoning optimization method, comprising:

[0006] Performing structured pruning on a preset initial large language model to obtain a corresponding large language model to be adjusted; wherein the large language model to be adjusted is a pre-trained model, and model quantization has been completed in the model training stage;

[0007] Obtain target data to be inferred, determine whether the data length of the target data to be inferred is less than a preset length threshold, and if the data length of the target data to be inferred is not less than the preset length threshold, divide the target data to be inferred into data segments of target length;

[0008] Acquire feature information of the target data to be inferred, automatically adjust the initial context window in the large language model to be adjusted according to the feature information to obtain a corresponding target large language model, and use the target large language model to infer the data segment.

[0009] Optionally, the performing structured pruning on the preset initial large language model includes:

[0010] Evaluating the importance of each neuron in the initial large language model so as to select a target neuron from each of the neurons; wherein the target neuron is a neuron whose importance is less than a preset importance threshold;

[0011] The target neuron and a connection structure corresponding to the target neuron are removed to perform structured pruning on the preset initial large language model.

[0012] Optionally, the using the target large language model to reason about the data segment includes:

[0013] A target computing device is selected according to the feature information of the target data to be inferred, and the data segment corresponding to the target data to be inferred is inferred based on the target computing device and the target large language model.

[0014] Optionally, the using the target large language model to reason about the data segment includes:

[0015] Design a target accuracy selection algorithm, and use the target accuracy selection algorithm to analyze the feature information of the target data to be inferred, so as to obtain a target accuracy pattern corresponding to the target data to be inferred;

[0016] The target large language model and the target accuracy mode are used to infer the data segment corresponding to the target data to be inferred.

[0017] Optionally, the using the target large language model to reason about the data segment includes:

[0018] Monitor the load status of each node in the inference server corresponding to the target large language model, and analyze the load status of each node using a preset load balancing algorithm to obtain a target task allocation strategy;

[0019] The target large language model and the target task allocation strategy are used to reason about the data segment corresponding to the target data to be inferred.

[0020] Optionally, the large language model reasoning optimization method further includes:

[0021] If the target data to be inferred is a continuous session, data inference is performed on the current target data to be inferred according to the cached data corresponding to the target inference result; wherein the target inference result is the inference result obtained by the target large language model when it last inferred the target data to be inferred.

[0022] Optionally, the large language model inference optimization method also includes: saving the target data to be inferred and the model inference state corresponding to the target data to be inferred in real time, so as to perform data inference on the target data to be inferred using the same model inference state when the user session is reconnected.

[0023] In a second aspect, the present application provides a large language model reasoning optimization device, comprising:

[0024] A model pruning module, used for performing structured pruning on a preset initial large language model to obtain a corresponding large language model to be adjusted; wherein the large language model to be adjusted is a pre-trained model, and model quantization has been completed in the model training stage;

[0025] A data segmentation module is used to obtain target data to be inferred, determine whether the data length of the target data to be inferred is less than a preset length threshold, and if the data length of the target data to be inferred is not less than the preset length threshold, segment the target data to be inferred into data segments of target length;

[0026] A data inference module is used to obtain feature information of the target data to be inferred, automatically adjust the initial context window in the large language model to be adjusted according to the feature information to obtain the corresponding target large language model, and use the target large language model to infer the data segment.

[0027] In a third aspect, the present application provides an electronic device, including:

[0028] Memory, used to store computer programs;

[0029] A processor is used to execute the computer program to implement the aforementioned large language model reasoning optimization method.

[0030] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the aforementioned large language model reasoning optimization method.

[0031] The present application first performs structured pruning on the preset initial large language model to obtain the corresponding large language model to be adjusted; wherein the large language model to be adjusted is a pre-trained model, and the model quantization has been completed in the model training stage, and then the target data to be inferred is obtained, and it is determined whether the data length of the target data to be inferred is less than the preset length threshold. If the data length of the target data to be inferred is not less than the preset length threshold, the target data to be inferred is divided into data segments of the target length, and finally the feature information of the target data to be inferred is obtained, and the initial context window in the large language model to be adjusted is automatically adjusted according to the feature information to obtain the corresponding target large language model, and the target large language model is used to infer the data segment. It can be seen that the large language model in the present application has completed the model quantization in the model training stage, so the large language model in the present application can be deployed on a hardware device that supports low-precision data types, ensuring the scalability of the data reasoning process; by dividing the data to be inferred into data segments and processing the data segments in sequence, the speed of model reasoning is improved; by adjusting the context window according to the characteristics of the data to be inferred, the overhead of computing resources is reduced on the basis of ensuring the quality of model reasoning, thereby improving the efficiency of model reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0033] Figure 1 A flow chart of a large language model reasoning optimization method disclosed in this application;

[0034] Figure 2 A schematic diagram of a specific large language model reasoning optimization system framework disclosed in this application;

[0035] Figure 3 A flowchart of a specific large language model reasoning optimization method disclosed in this application;

[0036] Figure 4 This is a schematic diagram of the structure of a large language model reasoning optimization device disclosed in this application;

[0037] Figure 5 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] The current model reasoning method has extremely high requirements for computing resources in the data reasoning stage, especially in scenarios with high real-time requirements, and cannot fully utilize modern hardware resources such as GPUs, TPUs, etc. for efficient parallel computing. To this end, this application provides a model reasoning optimization method that improves the efficiency and scalability of model reasoning by quantizing the model and sharding the data.

[0040] See also Figure 1 As shown, the embodiment of the present invention discloses a large language model reasoning optimization method, including:

[0041] Step S11: perform structured pruning on the preset initial large language model to obtain a corresponding large language model to be adjusted; wherein the large language model to be adjusted is a pre-trained model, and model quantization has been completed in the model training stage.

[0042] The system architecture in this embodiment is as follows Figure 2As shown, firstly, the model A and the model B are quantized and pruned respectively. When a data inference request is received, the corresponding model is used to infer the data according to the load balancing principle. That is, the present embodiment can use different large language models to perform model inference on the inference data at the same time. It can be understood that the present embodiment can also use different models that have been quantized and structured pruned to perform model inference at the same time. In order to improve the efficiency of model inference, the model needs to be structured pruned first to reduce unnecessary calculation amount. The above process of structured pruning of the model can specifically include: pruning each of the initial large language models The importance of neurons is evaluated so as to select target neurons from each neuron; wherein, the target neuron is a neuron whose importance is less than a preset importance threshold; the target neuron and the connection structure corresponding to the target neuron are removed to perform structured pruning on the preset initial large language model; that is, the model parameters are structured pruned and sparsely processed, and by removing unimportant connections or neurons, that is, neurons whose importance is less than a preset importance threshold and the corresponding leveling, the model size can be greatly reduced without affecting the reasoning accuracy, which helps to deploy large models in resource-constrained environments, such as edge devices or mobile terminals. It can be understood that in this embodiment, when performing model structured pruning, the reasoning accuracy of the model is also taken into account. In a specific implementation, a model reasoning accuracy threshold can be preset, and the reasoning accuracy of the model can be measured in real time when the model is pruned. When the reasoning accuracy of the model is less than the preset accuracy threshold, the structured pruning of the model is stopped.

[0043] In this embodiment, the large language model to be adjusted is a pre-trained model, and the model quantization has been completed in the model training stage. Specifically, in addition to the conventional floating-point representation, a set of quantized parameter copies will also be maintained at the same time. This allows the model to be deployed directly on hardware that supports low-precision data types such as BF16 (Brain Floating Point 16-bit, a floating-point format) or INT8, thereby further improving the reasoning speed and reducing power consumption. By introducing quantization operations during the training process, the model can be directly deployed in a low-precision environment, further improving the reasoning speed and reducing resource consumption, and increasing the scope of application of the model and improving scalability; by performing structured pruning on the model, the output accuracy and data processing efficiency of the model can be balanced, while ensuring the output accuracy and improving the data processing efficiency of the model.

[0044] Step S12, obtaining the target data to be inferred, and determining whether the data length of the target data to be inferred is less than a preset length threshold; if the data length of the target data to be inferred is not less than the preset length threshold, dividing the target data to be inferred into data segments of target length.

[0045] In this embodiment, after obtaining the data to be inferred, it is necessary to determine whether the data length of the data to be inferred is less than a preset length threshold. In a specific implementation, if the data length of the data to be inferred is greater than the preset length threshold, the data to be inferred is divided into data segments of target length. That is, according to the sliding window cache mechanism, for the long text input scenario, this embodiment introduces segmented cache technology and sliding window strategy, which effectively reduces the extra burden caused by repeated calculations. Specifically, in order to cope with the computational redundancy problem caused by long text input, this embodiment designs a sliding window cache mechanism. When a long text input is received, the system divides it into Small segments of fixed length are processed one by one, and whenever a new segment is encountered, only the newly added part needs to be calculated, without having to recalculate the entire sequence from the beginning; in another specific implementation, if the data length of the data to be inferred is not greater than the preset length threshold, there is no need to segment the data to be inferred, and the large language model is directly used to process the data to be inferred; in addition, the data to be inferred in this embodiment can be a medical diagnosis report, and the medical diagnosis report is analyzed by using a large language model to obtain the corresponding treatment prescription for the patient; it can also be work order information in the production process, and the work order title corresponding to the work order is obtained by reasoning the work order information. By segmenting the data to be inferred, whenever a new segment is encountered, only the newly added part needs to be calculated, without having to recalculate the entire sequence from the beginning, thereby improving the efficiency of model reasoning.

[0046] Step S13: Acquire feature information of the target data to be inferred, automatically adjust the initial context window in the large language model to be adjusted according to the feature information to obtain a corresponding target large language model, and use the target large language model to infer the data segment.

[0047] In this embodiment, it is necessary to obtain the feature information of the data to be inferred, and automatically adjust the context window of the large language model to be adjusted according to the feature information. Specifically, the adaptive context length adjustment automatically adjusts the size of the context window according to the content characteristics of the input text, that is, the above-mentioned feature information, to balance the reasoning speed and accuracy. Considering that different tasks have different requirements for context length, the system can dynamically adjust the size of the context window according to factors such as the semantic complexity of the input text and the coherence of the topic. For example, a shorter context can be used when processing simple question-answer pairs; and the context range can be appropriately extended when generating a coherent article summary. This can ensure the quality of reasoning without causing excessive computational overhead.

[0048] In addition, in this embodiment, when performing model reasoning, multi-precision collaboration is utilized. The model combines multiple floating-point formats such as FP16 (Half-precision floating-point, i.e., half-precision floating-point numbers) / FP32 / BF16 (Brain Floating Point16-bit, i.e., a floating-point number format), and uses lower-precision data types as much as possible to speed up calculations and save memory while ensuring the accuracy of the reasoning results; in the reasoning process, this embodiment adopts a mixed precision strategy. Specifically, FP32 is used to maintain numerical stability for operations before the activation function; and for relatively stable steps such as weight updates, FP16 is switched to speed up the calculation. In addition, this embodiment also introduces technologies such as Loss Scaling (i.e., loss scaling technology) to prevent the gradient disappearance or explosion problems that may be caused by low-precision operations.

[0049] In this embodiment, the above-mentioned process of reasoning the data fragments using the target large language model can specifically include: selecting the target computing device according to the characteristic information of the target data to be inferred, and reasoning the data fragments corresponding to the target data to be inferred based on the target computing device and the target large language model; that is, the target large language model in this embodiment is a model compatible with a variety of computing devices, such as CPU, GPU, TPU and other devices, and the model can select the optimal computing platform according to the characteristics of the task, thereby maximizing the utilization rate; considering the differences in the demand for computing resources of different tasks, specifically, this embodiment designs an inference architecture compatible with various types of computing devices, and the system can select the optimal computing platform according to the characteristics of different tasks to maximize resource utilization. For example, for matrix multiplication intensive tasks, priority is given to GPU; and for tasks dominated by control flow, it is more suitable for CPU execution. In addition, this embodiment can use the precision mode corresponding to the data to be inferred to perform model reasoning operations, and this embodiment can also select appropriate nodes for model reasoning operations according to the status of different nodes in the model.

[0050] It should be noted that the present embodiment can also perform model inference operations according to the context of the conversation with the user, that is, if the target data to be inferred is a continuous conversation, data inference is performed on the current target data to be inferred according to the cached data corresponding to the target inference result; wherein the target inference result is the inference result obtained by the target large language model when inferring the target data to be inferred last time, that is, for continuously input conversations or document fragments, only the newly added parts are inferred, and the cached results are used to avoid repeated calculations. When processing continuous input (such as conversations or document fragments), the present embodiment only infers the newly added parts, and the cached results are used to avoid repeated calculations. In addition, the large language model in this embodiment also has a real-time data saving and recovery mechanism, which can save the target data to be inferred and the model reasoning state corresponding to the target data to be inferred in real time, so that the target data to be inferred can be inferred using the same model reasoning state when the user session is reconnected; that is, when the user session is reconnected after the interruption, the previous reasoning state can be quickly restored to continue the unfinished task, that is, when the large language model in this embodiment is reconnected after the user session is interrupted, the previous reasoning state can be quickly restored to continue the unfinished task, which is particularly suitable for use in mobile devices or other unstable network environments. By selecting the most appropriate floating point format according to different operation requirements, the calculation is greatly accelerated and memory is saved while ensuring the accuracy of the reasoning result, thereby improving the overall reasoning speed; through the incremental reasoning and cache reuse mechanism, only the newly added part is inferred, avoiding unnecessary repeated calculations and significantly reducing the waste of computing resources; through the historical state saving and recovery function, it is ensured that the previous reasoning state can be quickly restored after the user session is interrupted, and the unfinished task can be continued, providing a smoother and more convenient user experience.

[0051] It can be seen that this embodiment segments the inference data so that whenever a new segment is encountered, only the newly added part needs to be calculated, without having to recalculate the entire sequence from scratch, thereby improving the efficiency of model inference; the sliding window cache mechanism and adaptive context length adjustment reduce repeated calculations during long text input and shorten the inference time, especially when processing continuous conversations or document segments; model compression and quantization techniques (such as structured pruning, sparsification processing, and quantization-aware training) not only reduce the model size and computational complexity, but also enable large models to be deployed in resource-constrained environments, such as edge devices or mobile terminals, thereby expanding application scenarios.

[0052] Based on the above embodiments, it can be seen that the present application discloses a large language model reasoning optimization process. In order to make the technical solution of the present application more complete, the present application will elaborate on the details of data reasoning in the large language model reasoning optimization process. Figure 3As shown, the embodiment of the present application provides a specific large language model reasoning optimization process, including:

[0053] Step S21: perform structured pruning on the preset initial large language model to obtain a corresponding large language model to be adjusted; wherein the large language model to be adjusted is a pre-trained model, and model quantization has been completed in the model training stage.

[0054] Step S22, obtain the target data to be inferred, and determine whether the data length of the target data to be inferred is less than a preset length threshold. If the data length of the target data to be inferred is not less than the preset length threshold, divide the target data to be inferred into data segments of target length.

[0055] Step S23: Acquire feature information of the target data to be inferred, and automatically adjust the initial context window in the large language model to be adjusted according to the feature information to obtain a corresponding target large language model.

[0056] Step S24, obtaining the precision mode corresponding to the target data to be inferred, and determining the target node from the target large language model, and performing data inference on the data segment corresponding to the target data to be inferred using the target node in the target large language model and the precision mode.

[0057] In this embodiment, the data segment can be inferred according to the precision mode corresponding to the target data to be inferred. Specifically, a target precision selection algorithm is designed, and the target precision selection algorithm is used to analyze the feature information of the target data to be inferred to obtain the target precision mode corresponding to the target data to be inferred; the target large language model and the target precision mode are used to infer the data segment corresponding to the target data to be inferred; that is, based on the characteristics of the input text, such as semantic complexity (one of the above-mentioned feature information), the most appropriate precision mode is dynamically selected to ensure the best performance under different tasks. In order to enable the system to automatically select the most appropriate precision mode according to the characteristics of the input text, this embodiment designs an intelligent precision selection algorithm, namely the above-mentioned target precision selection algorithm. The algorithm analyzes the semantic complexity, vocabulary diversity and other characteristics of the input text, and recommends the most suitable precision configuration accordingly. For example, for more complex scientific literature, a high-precision mode may be preferred; while for daily conversations, a lower precision can be selected to obtain a faster response speed.

[0058] In addition, the present embodiment can also select a target node from the large language model and use the target node to complete the model reasoning task; specifically, monitor the load status of each node in the reasoning server corresponding to the target large language model, and use a preset load balancing algorithm to analyze the load status of each node to obtain a target task allocation strategy; use the target large language model and the target task allocation strategy to infer the data fragments corresponding to the target data to be inferred, that is, the present embodiment monitors each computing node, that is, the workload of different models, and dynamically adjusts the task allocation to ensure that all nodes can work efficiently, avoiding the situation where some nodes are overloaded and other nodes are idle; in order to ensure that all computing nodes can work efficiently, the present embodiment implements an adaptive load balancing algorithm based on a prediction model, and the algorithm analyzes the status of each node in real time, and makes the optimal task allocation decision based on this, that is, the above-mentioned target task allocation strategy; specifically, the present embodiment collects performance indicators (such as CPU / GPU utilization, memory occupancy, etc.) after each round of iteration, and estimates the performance of the next few rounds through a machine learning algorithm, and then plans the next task distribution plan in advance. Through the intelligent precision selection algorithm, the optimal precision configuration is automatically selected according to the characteristics of the input text, ensuring the optimal performance under different tasks and improving the model reasoning speed; through the adaptive load balancing algorithm based on the predictive model, task allocation is dynamically adjusted to ensure that all nodes work efficiently and maximize resource utilization, which not only improves the stability and reliability of the system, but also enhances the ability to cope with complex tasks.

[0059] Among them, the specific process of the above steps S21 to S23 can refer to the corresponding content disclosed in the above embodiments, and will not be repeated here.

[0060] It can be seen that the large language model in this application has completed model quantization in the model training stage, so the large language model in this application can be deployed on hardware devices that support low-precision data types, ensuring the scalability of the data reasoning process; by dividing the data to be inferred into data segments and processing the data segments in sequence, the speed of model reasoning is improved; by adjusting the context window according to the characteristics of the data to be inferred, the overhead of computing resources is reduced while ensuring the quality of model reasoning, thereby improving the efficiency of model reasoning.

[0061] See also Figure 4 As shown, the embodiment of the present invention discloses a large language model reasoning optimization device, including:

[0062] The model pruning module 11 is used to perform structured pruning on the preset initial large language model to obtain the corresponding large language model to be adjusted; wherein the large language model to be adjusted is a pre-trained model, and the model quantization has been completed in the model training stage;

[0063] The data segmentation module 12 is used to obtain target data to be inferred, determine whether the data length of the target data to be inferred is less than a preset length threshold, and if the data length of the target data to be inferred is not less than the preset length threshold, segment the target data to be inferred into data segments of target length;

[0064] The data inference module 13 is used to obtain feature information of the target data to be inferred, automatically adjust the initial context window in the large language model to be adjusted according to the feature information to obtain the corresponding target large language model, and use the target large language model to infer the data segment.

[0065] It can be seen that the large language model in this application has completed model quantization in the model training stage, so the large language model in this application can be deployed on hardware devices that support low-precision data types, ensuring the scalability of the data reasoning process; by dividing the data to be inferred into data segments and processing the data segments in sequence, the speed of model reasoning is improved; by adjusting the context window according to the characteristics of the data to be inferred, the overhead of computing resources is reduced while ensuring the quality of model reasoning, thereby improving the efficiency of model reasoning.

[0066] In some specific embodiments, the model pruning module 11 may specifically include:

[0067] A neuron screening unit, used for evaluating the importance of each neuron in the initial large language model, so as to screen out a target neuron from each of the neurons; wherein the target neuron is a neuron whose importance is less than a preset importance threshold;

[0068] A model pruning unit is used to remove the target neuron and the connection structure corresponding to the target neuron to perform structured pruning on the preset initial large language model.

[0069] In some specific embodiments, the data reasoning module 13 may specifically include:

[0070] The first data inference unit is used to select a target computing device according to the feature information of the target data to be inferred, and to infer the data segment corresponding to the target data to be inferred based on the target computing device and the target large language model.

[0071] In some specific embodiments, the data reasoning module 13 may specifically include:

[0072] An accuracy mode acquisition unit, used to design a target accuracy selection algorithm, and use the target accuracy selection algorithm to analyze the feature information of the target data to be inferred, so as to obtain a target accuracy mode corresponding to the target data to be inferred;

[0073] The second data inference unit is used to use the target large language model and the target accuracy mode to infer the data segment corresponding to the target data to be inferred.

[0074] In some specific embodiments, the data reasoning module 13 may specifically include:

[0075] A node monitoring unit, used to monitor the load status of each node in the inference server corresponding to the target large language model, and analyze the load status of each node using a preset load balancing algorithm to obtain a target task allocation strategy;

[0076] The third data inference unit is used to use the target large language model and the target task allocation strategy to infer the data segment corresponding to the target data to be inferred.

[0077] In some specific embodiments, the large language model reasoning optimization device further includes:

[0078] The second data inference module is used to perform data inference on the current target data to be inferred according to the cached data corresponding to the target inference result if the target data to be inferred is a continuous session; wherein the target inference result is the inference result obtained by the target large language model when inferring the target data to be inferred last time.

[0079] In some specific embodiments, the large language model reasoning optimization device further includes:

[0080] The data saving module is used to save the target data to be inferred and the model inference state corresponding to the target data to be inferred in real time, so as to use the same model inference state to perform data inference on the target data to be inferred when the user session is reconnected.

[0081] Furthermore, the present application also discloses an electronic device. Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0082] Figure 5A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the large language model reasoning optimization method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0083] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0084] In addition, the memory 22 as a carrier for storing resources may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be temporary storage or permanent storage.

[0085] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the large language model reasoning optimization method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0086] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned large language model reasoning optimization method is implemented. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0087] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0088] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0089] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0090] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0091] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A large language model reasoning optimization method, characterized in that: include: Performing structured pruning on a preset initial large language model to obtain a corresponding large language model to be adjusted; wherein the large language model to be adjusted is a pre-trained model, and model quantization has been completed in the model training stage; Obtain target data to be inferred, determine whether the data length of the target data to be inferred is less than a preset length threshold, and if the data length of the target data to be inferred is not less than the preset length threshold, divide the target data to be inferred into data segments of target length; Acquire feature information of the target data to be inferred, automatically adjust the initial context window in the large language model to be adjusted according to the feature information to obtain a corresponding target large language model, and use the target large language model to infer the data segment.

2. The large language model reasoning optimization method according to claim 1, characterized in that: The structural pruning of the preset initial large language model includes: Evaluating the importance of each neuron in the initial large language model so as to select a target neuron from each of the neurons; wherein the target neuron is a neuron whose importance is less than a preset importance threshold; The target neuron and a connection structure corresponding to the target neuron are removed to perform structured pruning on the preset initial large language model.

3. The large language model reasoning optimization method according to claim 1, characterized in that: The using the target large language model to reason about the data segment includes: A target computing device is selected according to the feature information of the target data to be inferred, and the data segment corresponding to the target data to be inferred is inferred based on the target computing device and the target large language model.

4. The large language model reasoning optimization method according to claim 1, characterized in that: The using the target large language model to reason about the data segment includes: Design a target accuracy selection algorithm, and use the target accuracy selection algorithm to analyze the feature information of the target data to be inferred, so as to obtain a target accuracy pattern corresponding to the target data to be inferred; The target large language model and the target accuracy mode are used to infer the data segment corresponding to the target data to be inferred.

5. The large language model reasoning optimization method according to claim 1, characterized in that: The using the target large language model to reason about the data segment includes: Monitor the load status of each node in the inference server corresponding to the target large language model, and analyze the load status of each node using a preset load balancing algorithm to obtain a target task allocation strategy; The target large language model and the target task allocation strategy are used to reason about the data segment corresponding to the target data to be inferred.

6. The large language model reasoning optimization method according to claim 1, characterized in that: Also includes: If the target data to be inferred is a continuous session, data inference is performed on the current target data to be inferred according to the cached data corresponding to the target inference result; The target inference result is the inference result obtained by the target large language model when it last inferred the target data to be inferred.

7. The large language model reasoning optimization method according to any one of claims 1 to 6, characterized in that: Also includes: The target data to be inferred and the model inference state corresponding to the target data to be inferred are saved in real time, so that the same model inference state can be used to perform data inference on the target data to be inferred when the user session is reconnected.

8. A large language model reasoning optimization device, characterized in that: include: A model pruning module, used for performing structured pruning on a preset initial large language model to obtain a corresponding large language model to be adjusted; wherein the large language model to be adjusted is a pre-trained model, and model quantization has been completed in the model training stage; A data segmentation module is used to obtain target data to be inferred, determine whether the data length of the target data to be inferred is less than a preset length threshold, and if the data length of the target data to be inferred is not less than the preset length threshold, segment the target data to be inferred into data segments of target length; A data inference module is used to obtain feature information of the target data to be inferred, automatically adjust the initial context window in the large language model to be adjusted according to the feature information to obtain the corresponding target large language model, and use the target large language model to infer the data segment.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the large language model reasoning optimization method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the large language model reasoning optimization method as described in any one of claims 1 to 7.

Images