Large model reasoning optimization method and system based on lightweight gating mechanism

Abstract

The invention discloses a large model reasoning optimization method and system based on a lightweight gating mechanism, and aims to improve the efficiency of a large-scale pre-training model in a reasoning task and remarkably reduce the consumption of computing resources. According to the method, firstly, a pre-training model is initialized, a dynamic routing mechanism is introduced in the reasoning process, and key layer output in the reasoning process is effectively screened in combination with a lightweight gating and threshold judgment mechanism. The output of each layer is dynamically evaluated, a gating output value is generated by using a gating LSTM unit, and the importance score of the generation layer is activated through a sigmoid function. And if the score is lower than a preset threshold value, skipping the calculation of the layer to reduce unnecessary calculation overhead. The optimization strategy can adaptively judge the importance of each layer of output, and on the premise of ensuring the output quality of the model, redundant calculation is reduced, and the reasoning speed is increased. The method is suitable for various task scenes including natural language processing, image generation and the like, and has wide application value.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and in particular to a large model reasoning optimization method and system based on a lightweight gating mechanism, aiming to improve the reasoning speed of large-scale pre-trained language models and reduce computing resource consumption. Background Art

[0002] Current large-scale language models (such as GPT, BERT, etc.) have shown excellent performance in many tasks, but due to their large number of layers, the model reasoning process requires very high computing resources, resulting in low model reasoning efficiency in practical applications, affecting the response speed. Although existing large model reasoning optimization technologies such as pruning and quantization can partially improve the reasoning speed, there are still problems such as large amount of calculation, long reasoning time, and performance degradation. Therefore, how to optimize the reasoning speed without significantly affecting the quality of model generation has become a hot topic in current research. Summary of the invention

[0003] The purpose of the present invention is to address the deficiencies of the prior art and provide a large model reasoning optimization method and system based on a lightweight gating mechanism; it is particularly suitable for using a dynamic routing mechanism to improve reasoning efficiency and speed during the reasoning process, and a method for optimizing the large model reasoning process by selectively skipping certain network layers, effectively reducing the amount of calculation, improving reasoning speed and resource utilization, while maintaining model performance and accuracy.

[0004] To achieve the above object, the present invention provides a large model reasoning optimization method based on a lightweight gating mechanism, comprising the following steps:

[0005] (1) Use a large-scale dataset to pre-train a large model to generate a pre-trained model suitable for reasoning tasks; input the data into the pre-trained model;

[0006] (2) Use a lightweight gating mechanism to quantitatively evaluate the importance score of each layer output of the pre-trained model to obtain the confidence or importance score of each layer output;

[0007] (3) Dynamically decide whether to skip the calculation of each layer based on the confidence or importance score of the output of each layer;

[0008] (4) Through the gated LSTM unit, the output of each layer is gated and evaluated to screen out the key output features; if the output importance score is lower than the set threshold, the layer and subsequent calculations are skipped;

[0009] (5) Perform reasoning tasks based on the screened key output features to improve the reasoning speed.

[0010] Furthermore, the pre-training model is implemented through a causal language model of a Transformer structure, including BERT, GPT, and QWen model structures.

[0011] Furthermore, the step (3) includes: defining specific layers processed by the dynamic routing mechanism, and selecting the layers to be dynamically routed through the selected_layers parameter; the dynamic routing mechanism allows the user or developer to select specific processing layers at different levels of the model; through the dynamic routing mechanism, the model selectively skips certain layers according to the requirements of the task during the inference process, or retains specific layers to ensure the accuracy of the output.

[0012] Furthermore, in step (4), the output of the layer is linearly transformed to adjust the dimension to adapt to the input requirements of the LSTM; specifically, the following steps are included:

[0013] A. Use linear transformation to adjust the dimension of each layer output to make it consistent with the LSTM input size;

[0014] B. Linear transformation is implemented in the form of matrix multiplication, that is, the layer output is transformed through a weight matrix; the parameters of the linear layer are automatically updated through the training process to ensure its adaptability to the input.

[0015] Furthermore, in step (4), the gating mechanism adopts a lightweight neural network structure, and uses a lightweight gated LSTM to decide whether to skip the calculation of the current layer; this LSTM generates a gating value based on the input features, whose value is between 0 and 1, and is processed by the torch.sigmoid() function; if the gating value is less than the set threshold, the calculation of the layer is skipped and None is returned, otherwise the calculation of the layer continues.

[0016] Furthermore, in the layer-by-layer forward propagation process, the output of each layer is processed by the dynamic routing dynamic_routing function; if the output of a layer is judged to be of low importance, that is, the gate value is less than the set threshold, the calculation of this layer will be skipped and the output of the previous layer will continue to be used; the threshold is dynamically adjusted according to the complexity of the reasoning task.

[0017] Furthermore, when the reasoning task is a generative text task, the diversity and fluency of the generated text are evaluated based on the screening results of each layer.

[0018] To achieve the above object, the present invention also provides a large model reasoning optimization system based on a lightweight gating mechanism, the system comprising:

[0019] Pre-training module: used to pre-train large models using large-scale data sets;

[0020] Importance evaluation module: used to embed a lightweight gating mechanism to quantitatively evaluate the importance of each layer’s output;

[0021] Dynamic layer skipping module: used to dynamically skip the calculation of certain layers based on the confidence or importance score of the output;

[0022] Gating module: evaluates the output of each layer through a lightweight gating mechanism to determine whether to skip the calculation of the layer;

[0023] Reasoning execution module: performs reasoning tasks based on the filtered important output information.

[0024] To achieve the above-mentioned purpose, the present invention also provides an electronic device, including a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned large model reasoning optimization method based on the lightweight gating mechanism.

[0025] To achieve the above objectives, the present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the above-mentioned large model reasoning optimization method based on the lightweight gating mechanism is implemented.

[0026] The beneficial effect of the present invention is that the present invention can significantly reduce unnecessary calculations in the large model reasoning process and improve the reasoning speed by introducing the information bottleneck mechanism and the design of the selective activation layer. The experimental results show that the scheme can significantly reduce the reasoning time while maintaining the quality of the generated text, especially when dealing with complex tasks, the reasoning efficiency is significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0028] Figure 1 A schematic diagram of a large model reasoning optimization process based on a lightweight gating mechanism provided by an embodiment of the present invention;

[0029] Figure 2 A structural block diagram of a lightweight gating mechanism provided by an embodiment of the present invention;

[0030] Figure 3 A schematic diagram of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of systems and methods consistent with some aspects of the present invention as detailed in the appended claims.

[0032] The terms used in the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "the" and "the" used in the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0033] It should be understood that although the terms first, second, third, etc. may be used in the present invention to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0034] The present invention is described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.

[0035] Based on the information bottleneck theory and dynamic routing mechanism, the present invention proposes the following specific implementation method:

[0036] Model structure: The LSTM (Long Short-Term Memory) gating mechanism is introduced. The output of each layer is compressed through the linear layer and then passed into the LSTM to obtain a layer importance score. The layer is activated based on the score. If the score is lower than the threshold, the layer is skipped.

[0037] Reasoning acceleration: During the reasoning process, the model will decide whether to skip certain layers based on the amount of information and importance score of each layer. This will reduce invalid calculations while retaining the information flow that has a greater impact on the final result, thereby effectively improving the reasoning speed.

[0038] Evaluation Metrics: The quality of the generated text is evaluated by perplexity and the diversity of the generated text to ensure that the reasoning acceleration does not have a negative impact on the overall effect of text generation. Perplexity is used to measure the quality of the generated text, and repetition rate is used to evaluate the diversity of the text.

[0039] See also Figure 1 The present invention provides a large model reasoning optimization method based on a lightweight gating mechanism, comprising the following steps:

[0040] (1) Pre-training large models: Use large-scale datasets to pre-train large models and generate pre-trained models suitable for reasoning tasks. The input data passes through each layer of the pre-trained model in sequence, and each layer generates intermediate features and passes them to the next layer. Among them, the input data is a dataset for training large language models such as Pile, BookCorpus, PubMed, and CommonCrawl, or text extracted from HTML web pages; Example: When building a dialogue system (such as a chatbot), mixed data containing social media conversations, news reports, etc. can be used to enable the model to understand languages ​​of different styles and contexts. In the pre-trained large model, each layer of the model usually refers to a layer in the Transformer architecture; these layers include the Self-Attention Layer and the Feed-Forward Layer.

[0041] Specifically, the pre-trained large model is implemented through a causal language model of a Transformer structure, including but not limited to the use of model structures such as BERT, GPT, and QWen.

[0042] (2) Dynamic importance evaluation: Based on the pre-trained model, a lightweight gating mechanism is used to quantitatively evaluate the importance score of each layer output to obtain the confidence or importance score of each layer output.

[0043] Specifically, a lightweight LSTM gating unit (gate_lstm) is used to process the input vector. gate_lstm generates a hidden state (hidden), which represents the importance characteristics of the layer in the current context. The generated hidden state passes through the sigmoid activation function and outputs a value in the range of (0,1). This value can be regarded as the importance score of the current layer, that is, the probability of "whether the layer should be skipped". The set gating parameter is used as the "importance threshold". When the importance score of a layer is less than this threshold, the layer will be regarded as unimportant, the output result will be ignored, and the calculation of the current layer will be skipped; otherwise, the layer output will be retained and passed to the next layer.

[0044] (3) Dynamically skipping layers: Based on the confidence or importance score of each layer’s output, dynamically decide whether to skip the calculation of the layer to reduce unnecessary calculations. The pre-selected feature layer will be screened by the dynamic routing module to determine whether the layer is skipped.

[0045] Specifically, define the specific layers processed by the dynamic routing mechanism, and select the layers to be dynamically routed through the selected_layers parameter. This mechanism allows users or developers to select specific processing layers at different levels of the model. Through the dynamic routing mechanism, the model can selectively skip certain layers according to the requirements of the task during the reasoning process, or retain specific layers to ensure the accuracy of the output. For example: for natural language processing tasks, the 6th, 12th, 18th, and 24th layers can be set as dynamic processing layers to reduce the computational burden while retaining the necessary context processing capabilities. Through the selected_layers parameter, developers can customize the model's calculation path according to the complexity of the reasoning task or actual needs, further improving the flexibility and efficiency of the reasoning process.

[0046] (4) Gating mechanism: Through the gated LSTM unit, the output of each layer is gated and evaluated to screen out the key output features; if the output importance score is lower than the set threshold, the layer and subsequent calculations are skipped. Information is propagated layer by layer until the last layer output.

[0047] The output of the layer is linearly transformed (self.linear) to adjust the dimension to adapt to the input requirements of LSTM. The specific operations include:

[0048] A. The output of each layer of the model usually has high-dimensional features. Directly inputting LSTM may lead to low model calculation efficiency due to dimension mismatch. Therefore, linear transformation is needed to adjust the dimension of each layer output to make it consistent with the LSTM input size.

[0049] B. Linear transformation is usually implemented in the form of matrix multiplication, that is, the layer output is transformed through a weight matrix; the parameters of the linear layer can be automatically updated through the training process to ensure its adaptability to the input.

[0050] That is, before entering LSTM, the input undergoes a linear transformation (self.linear) to adjust the dimension to adapt to the input requirements of LSTM.

[0051] Furthermore, the gating mechanism adopts a lightweight neural network structure, using a lightweight gated LSTM (gate_lstm) to decide whether to skip the calculation of the current layer. This LSTM generates a gating value (gate_output) based on the input features, and its value is between 0 and 1, which is processed by the torch.sigmoid() function. If the gate value is less than the set threshold (threshold), the calculation of the layer is skipped and None is returned, otherwise the calculation of the layer continues.

[0052] During the layer-by-layer forward propagation process, the output of each layer can be processed by the dynamic routing function. If the output of a layer is judged to be "low importance" (that is, gate_output is less than the threshold), the calculation of this layer will be skipped and the output of the previous layer will continue to be used.

[0053] The method decides whether to skip the calculation of subsequent layers based on the comparison result between the calculation output and a preset threshold, and the threshold can be dynamically adjusted according to the complexity of the reasoning task.

[0054] (5) Reasoning task execution: The information after layer screening and dynamic routing is passed to the output layer, and the reasoning task is performed based on the screened key output features to generate the reasoning results of the task, thereby improving the reasoning speed and reducing the computational overhead.

[0055] Furthermore, the reasoning task can include a generative text task, where the diversity and fluency of the generated text are evaluated based on the screening results of each layer.

[0056] See also Figure 2 The present invention provides a large model reasoning optimization system based on a lightweight gating mechanism, the system comprising:

[0057] Pre-training module: used to pre-train large models using large-scale data sets;

[0058] Importance evaluation module: used to embed a lightweight gating mechanism to quantitatively evaluate the importance of each layer’s output;

[0059] Dynamic layer skipping module: used to dynamically skip the calculation of certain layers based on the confidence or importance score of the output;

[0060] Gating module: evaluates the output of each layer through a lightweight gating mechanism to determine whether to skip the calculation of the layer;

[0061] Reasoning execution module: Executes subsequent reasoning tasks based on the filtered important output information to improve the reasoning speed and reduce the computational burden.

[0062] The system quantifies the importance of output information at each layer, reduces unnecessary calculations, and maintains the accuracy and diversity of reasoning results.

[0063] Corresponding to the above-mentioned embodiment of the large model reasoning optimization method based on the lightweight gating mechanism, the embodiment of the present application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned large model reasoning optimization method based on the lightweight gating mechanism. Figure 3As shown, a hardware structure diagram of any device with data processing capability in the large model reasoning optimization method based on the lightweight gating mechanism provided in the embodiment of the present application is shown, except Figure 3 In addition to the processor, memory, DMA controller, disk, and non-volatile memory shown, any device with data processing capabilities in which the apparatus in the embodiment is located may also include other hardware, which will not be described in detail, generally based on the actual functions of the device with data processing capabilities.

[0064] Corresponding to the aforementioned embodiment of the large model reasoning optimization method based on the lightweight gating mechanism, an embodiment of the present invention also provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the large model reasoning optimization method based on the lightweight gating mechanism in the above embodiment is implemented.

[0065] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0066] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

[0067] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A large model reasoning optimization method based on a lightweight gating mechanism, characterized in that: The following steps are involved: (1) Use a large-scale dataset to pre-train a large model to generate a pre-trained model suitable for reasoning tasks; input the data into the pre-trained model; (2) Use a lightweight gating mechanism to quantitatively evaluate the importance score of each layer output of the pre-trained model to obtain the confidence or importance score of each layer output; (3) Dynamically decide whether to skip the calculation of each layer based on the confidence or importance score of the output of each layer; (4) Through the gated LSTM unit, the output of each layer is gated and evaluated to screen out the key output features; If the output importance score is lower than the set threshold, the layer and subsequent calculations are skipped; (5) Perform reasoning tasks based on the screened key output features to improve the reasoning speed.

2. The large model reasoning optimization method based on the lightweight gating mechanism according to claim 1 is characterized in that: The pre-training model is implemented through a causal language model of a Transformer structure, including BERT, GPT, and QWen model structures.

3. The large model reasoning optimization method based on the lightweight gating mechanism according to claim 1 is characterized in that: The step (3) includes: defining specific layers processed by the dynamic routing mechanism, and selecting the layers to be dynamically routed through the selected_layers parameter; the dynamic routing mechanism allows the user or developer to select specific processing layers at different levels of the model; through the dynamic routing mechanism, the model selectively skips certain layers according to the requirements of the task during the inference process, or retains specific layers to ensure the accuracy of the output.

4. The large model reasoning optimization method based on the lightweight gating mechanism according to claim 1 is characterized in that: In step (4), the output of the layer is linearly transformed to adjust the dimension to adapt to the input requirements of the LSTM; specifically, the following steps are performed: A. Use linear transformation to adjust the dimension of each layer output to make it consistent with the LSTM input size; B. Linear transformation is implemented in the form of matrix multiplication, that is, the layer output is transformed through a weight matrix; the parameters of the linear layer are automatically updated through the training process to ensure its adaptability to the input.

5. The large model reasoning optimization method based on lightweight gating mechanism as claimed in claim 1, characterized in that: In the step (4), the gating mechanism adopts a lightweight neural network structure, and uses a lightweight gated LSTM to decide whether to skip the calculation of the current layer; this LSTM generates a gating value based on the input features, whose value is between 0 and 1, and is processed by the torch.sigmoid() function; if the gating value is less than the set threshold, the calculation of the layer is skipped and None is returned, otherwise the calculation of the layer continues.

6. The large model reasoning optimization method based on lightweight gating mechanism as claimed in claim 5, characterized in that: During the layer-by-layer forward propagation, the output of each layer is processed by the dynamic routing function dynamic_routing; If the output of a layer is judged to be of low importance, that is, the gating value is less than the set threshold, the calculation of this layer will be skipped and the output of the previous layer will continue to be used; the threshold is dynamically adjusted according to the complexity of the reasoning task.

7. The large model reasoning optimization method based on lightweight gating mechanism as claimed in claim 1, characterized in that: When the reasoning task is a generative text task, the diversity and fluency of the generated text are evaluated based on the screening results of each layer.

8. A large model reasoning optimization system based on a lightweight gating mechanism for implementing the method as claimed in claim 1, characterized in that: The system comprises: Pre-training module: used to pre-train large models using large-scale data sets; Importance evaluation module: used to embed a lightweight gating mechanism to quantitatively evaluate the importance of each layer’s output; Dynamic layer skipping module: used to dynamically skip the calculation of certain layers based on the confidence or importance score of the output; Gating module: evaluates the output of each layer through a lightweight gating mechanism to determine whether to skip the calculation of the layer; Reasoning execution module: performs reasoning tasks based on the filtered important output information.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the large model reasoning optimization method based on the lightweight gating mechanism as described in any one of claims 1-7 above.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by the processor, the large model reasoning optimization method based on the lightweight gating mechanism as described in any one of claims 1 to 7 is implemented.

Images