A large model optimization method and system based on sequence-dependent hierarchical propagation mechanism

Abstract

The present application provides a large model optimization method and system based on a sequence-dependent hierarchical propagation mechanism, which relates to the technical field of large language models. The large model includes a representation learning module and a downstream task module. The specific steps of the representation learning module are: performing natural language processing on the input text to obtain a word vector sequence; based on the sequence-dependent hierarchical propagation mechanism, the word vector sequence is evenly divided into blocks, and the attention mechanism between segments and within segments is used to generate a Key matrix and a Value matrix respectively; using the Key matrix and the Value matrix, the attention mechanism of the Transformer is calculated to obtain the final sequence representation as the input of the downstream task module. The present invention introduces a sequence-dependent hierarchical propagation mechanism, divides the sequence into multiple blocks, and effectively learns the sequence dependency relationship within and between these blocks, thereby significantly reducing the computational complexity of the traditional self-attention mechanism and effectively reducing memory usage.

Description

Technical Field

[0001] The present application relates to the technical field of large language models, and in particular to a large model optimization method and system based on a sequence-dependent hierarchical propagation mechanism. Background Art

[0002] Large Language Models (LLMs), a core breakthrough in artificial intelligence, are profoundly reshaping the knowledge processing paradigm in fields such as healthcare, finance, education, and scientific research. Through intelligent dialogue, knowledge reasoning, and content generation capabilities, LLMs have significantly improved industry efficiency, spawned innovative application scenarios, and driven the evolution of human-machine collaboration. LLMs typically consist of a representation learning module and a downstream task module. The representation learning module generates vector representations for user input text, while the downstream task module uses these vector representations as input to perform specific downstream tasks such as text classification, text generation, time series classification, and time series regression prediction. The core model architecture of the existing representation learning module is the Transformer. Its self-attention mechanism (SA) overcomes the length limitations of traditional sequence models. Through parallel computation and dynamic weight allocation, it enables the model to accurately capture long-range semantic connections. This technology provides the underlying support for groundbreaking models such as GPT and BERT. Its high scalability has directly driven the leap from billions to trillions of model parameters, laying the technical foundation for the intelligent transformation of large language models.

[0003] The Transformer model effectively captures long-distance dependencies through the SA mechanism and supports efficient parallel computing, thus overcoming the gradient vanishing problem of traditional recurrent neural networks when processing long sequences. However, the main drawback of the Transformer model is that the computational complexity of its self-attention mechanism is , where L is the sequence length, and its high memory usage leads to excessively high computational cost of LLM, which limits its further application and promotion.

[0004] While existing improved Transformer-based models, such as Reformer, Flowformer, and Informer, have reduced the computational complexity and improved efficiency of LLM to some extent, they still have significant drawbacks. These improved models typically optimize the self-attention mechanism by introducing complex auxiliary computational processes or additional operations (such as locality-sensitive hashing, network flow computation, and probabilistic sparse attention). However, these improvements often come with additional computational overhead, larger model parameters, or lower training efficiency, limiting their universality and scalability in practical applications. In scenarios involving long-sequence data modeling, existing optimization methods still fail to fully address the high memory usage and computational complexity of large models, and there is still room for further improvement. Summary of the Invention

[0005] In order to solve the above problems, this application proposes a large model optimization method and system based on a sequence-dependent hierarchical propagation mechanism. The sequence-dependent hierarchical propagation mechanism is introduced to divide the sequence into multiple blocks, and the sequence dependency relationship is effectively learned within and between these blocks, thereby significantly reducing the computational complexity of the traditional self-attention mechanism and effectively reducing memory usage.

[0006] According to some embodiments, the present application adopts the following technical solutions:

[0007] A large model optimization method based on a sequence-dependent hierarchical propagation mechanism, wherein the large model includes a representation learning module and a downstream task module. The specific steps of the representation learning module are as follows:

[0008] Perform natural language processing on the input text to obtain a word vector sequence;

[0009] Based on the sequence-dependent hierarchical propagation mechanism, the word vector sequence is evenly divided into blocks, and the attention mechanism between segments and within segments is used to generate Matrix and matrix;

[0010] use Matrix and The matrix is ​​used to calculate the Transformer's attention mechanism to obtain the final sequence representation, which is used as the input of the downstream task module for specific downstream tasks.

[0011] According to some embodiments, the present application adopts the following technical solutions:

[0012] A large model optimization system based on a sequence-dependent hierarchical propagation mechanism, wherein the large model includes a representation learning module and a downstream task module, wherein the representation learning module includes:

[0013] The language processing module is configured to: perform natural language processing on the input text to obtain a word vector sequence;

[0014] The hierarchical dependency module is configured to: divide the word vector sequence into even blocks based on the sequence dependency hierarchical propagation mechanism, and use the attention mechanism between segments and within segments to generate Matrix and matrix;

[0015] The sequence representation module is configured to: Matrix and The matrix is ​​used to calculate the Transformer's attention mechanism to obtain the final sequence representation, which is used as the input of the downstream task module for specific downstream tasks.

[0016] According to some embodiments, the present application adopts the following technical solutions:

[0017] A computer program product includes a computer program, which, when executed by a processor, implements the large model optimization method based on a sequence-dependent hierarchical propagation mechanism.

[0018] According to some embodiments, the present application adopts the following technical solutions:

[0019] A non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, a large model optimization method based on a sequence-dependent hierarchical propagation mechanism is implemented.

[0020] According to some embodiments, the present application adopts the following technical solutions:

[0021] An electronic device comprises: a processor, a memory and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the large model optimization method based on the sequence dependency hierarchical propagation mechanism.

[0022] Compared with the prior art, the present invention has the following advantages:

[0023] This paper introduces a hierarchical propagation mechanism for sequence dependencies, partitioning the sequence into multiple blocks and effectively learning sequence dependencies within and between these blocks. This significantly reduces the computational complexity of traditional self-attention mechanisms and effectively reduces memory usage. Furthermore, by sharing key and value matrices and optimizing the Transformer stacking layer structure, the paper further reduces the model parameter size, significantly improving the model's efficiency and performance when processing long sequence data.

[0024] The present invention uses the sequence dependency hierarchical propagation mechanism to ensure that the representation of each fragment in the generated matrix incorporates the dependency information of the entire sequence, reducing the information loss caused by the generation operation; in addition, the time complexity of the inter-fragment and intra-fragment attention mechanisms are respectively and , the time complexity of the final sequence-dependent hierarchical propagation mechanism is , and the computational complexity of the Transformer based on the sequence-dependent hierarchical propagation mechanism is significantly reduced compared to the traditional Transformer. The computational complexity of the attention mechanism is reduced by Reduce to , which means that the hierarchical propagation mechanism has advantages in terms of efficiency and resource consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings in the specification, which constitute a part of this application, are used to provide further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute improper limitations on this application.

[0026] Figure 1 This is a flow chart of the method of Example 1.

[0027] Figure 2 Detailed diagram of the method of Example 1.

[0028] Figure 3 This is a flow chart of the sequence-dependent hierarchical propagation mechanism of Example 1. DETAILED DESCRIPTION

[0029] The present application will be further described below with reference to the accompanying drawings and embodiments.

[0030] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.

[0031] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0032] Example 1

[0033] One embodiment of the present application, relying on the Shandong Provincial Key Laboratory of Artificial Intelligence Applications for People's Livelihood Services and the Shandong Provincial Higher Education Future Industry Laboratory of General Artificial Intelligence, provides a large model optimization method based on a sequence-dependent hierarchical propagation mechanism. The large model includes a representation learning module and a downstream task module. The specific steps of the representation learning module are as follows:

[0034] Step S1: Perform natural language processing on the input text to obtain a word vector sequence;

[0035] Step S2: Based on the sequence-dependent hierarchical propagation mechanism, the word vector sequence is evenly divided into blocks, and the attention mechanism between segments and within segments is used to generate Matrix and matrix;

[0036] Step S3: Utilization Matrix and The matrix is ​​used to calculate the Transformer's attention mechanism to obtain the final sequence representation, which is used as the input of the downstream task module for specific downstream tasks.

[0037] As an embodiment, the present application provides a large model optimization method based on a sequence dependency hierarchical propagation mechanism. By introducing the sequence dependency hierarchical propagation mechanism, the sequence is divided into multiple blocks and sequence dependencies are effectively learned within and between these blocks, thereby significantly reducing the computational complexity of the traditional self-attention mechanism and effectively reducing memory usage. Figure 1 It is a flowchart of the large model optimization method. Figure 2 This is a detailed diagram of the large model optimization method, such as Figure 1 、 2 As shown in Figure 2, the specific steps of the large model optimization method are:

[0038] Step 1: Obtain word vector sequence through steps such as word segmentation, word list construction, and word embedding.

[0039] Specifically, the original text is first segmented using the Byte Pair Encoding (BPE) algorithm to improve the model's generalization ability for low-frequency and unknown words. Secondly, based on the segmentation results, a vocabulary is constructed. The size of the vocabulary is typically determined based on task requirements and computing resources, striking a balance between representational power and computational overhead. Then, to enable the model to understand the semantic information of words, word embedding is performed on each subword in the vocabulary, mapping them into a high-dimensional continuous space. This provides rich semantic information for subsequent deep learning models. Finally, the text is processed into sequences based on the vocabulary.

[0040] Step 2: Use the sequence-dependent hierarchical propagation mechanism to evenly divide the sequence representation into blocks, and then use the inter-segment and intra-segment attention mechanisms to generate the key matrix and value matrix for the Transformer layer attention score calculation, specifically:

[0041] 1. Perform high-dimensional embedding and position encoding on the word vector sequence to obtain an embedding representation containing position information.

[0042] For word vector sequences, this embodiment uses the matrix Indicates that, Indicates the length of the sequence, that is, the word vector The number of The number of features representing a word vector. To obtain rich sequence dependency information from sequence data, we first embed the original data into a high-dimensional space through a linear layer. Then, we apply positional encoding to the embedded data to preserve the order of the sequence. Finally, we obtain an embedded representation that includes positional information.

[0043] Specifically, given a word vector sequence sample , embed it into the new space, and perform position encoding to obtain the embedded representation , which can be expressed as:

[0044]

[0045] in, and are the parameters of the learnable embedding layer, represents the positional encoding operation, which is formally defined as follows:

[0046]

[0047]

[0048] in, Indicates the first The position of the eigenvector. 、 represent the odd and even positions of the feature respectively.

[0049] 2. Generate the Key matrix and Value matrix based on the embedded representation containing position information.

[0050] Utilize the sequence-dependent hierarchical propagation mechanism to generate compressed Matrix and Matrix, in order to reduce the complexity of the self-attention mechanism, such as Figure 3 As shown, the specific steps are:

[0051] (1) Embedding representation containing position information Padding is performed to make its length adapt to the computational requirements; the padded sequence is then evenly divided into blocks, and each block is mean pooled to obtain a preliminary segment-level representation.

[0052] Specifically, for the entire embedding representation Perform padding-blocking operations, then perform mean pooling on each segment, use the pooled vector as the representation of each sequence segment, and finally obtain a preliminary segment-level representation , which can be expressed as:

[0053]

[0054]

[0055]

[0056]

[0057] in, Chunk, Pad, mean They are sequence uniform slicing operation, filling operation, and average pooling operation. Representation will be embedded in the representation Divided into fragments, represents the i-th sequence fragment, Indicates a floor operation.

[0058] (2) The inter-fragment attention mechanism and the intra-fragment attention mechanism are used on the preliminary fragment-level representation to obtain the final inter-fragment representation and fine-grained intra-fragment representation.

[0059] In order to generate and The matrix can fully reflect the sequence dependency representation of the entire sequence. This embodiment uses a hierarchical propagation mechanism to process the sequence dependency information outside and within a segment.

[0060] Among them, the inter-segment attention mechanism is used to obtain the inter-segment dependency, and the final segment-level representation is obtained by calculating the attention distribution between different segments. , which can be expressed as:

[0061]

[0062]

[0063] in, represents the preliminary fragment-level representation, Represents the softmax function, vector x represents the input of the softmax function, x i …x h Represents the i-th to h-th elements of x.

[0064] The intra-segment attention mechanism is used to obtain intra-segment dependencies, and the fine-grained intra-segment representation is obtained by calculating the attention matrix in each segment. , which can be expressed as:

[0065]

[0066]

[0067] (3) Concatenate the inter-fragment representation and the intra-fragment representation to obtain the Key matrix and the Value matrix, which can be expressed as follows:

[0068]

[0069]

[0070]

[0071] in, They are Value matrix and Key matrix respectively. represents a single linear layer, These are the parameters of this linear layer, Represents a tensor concatenation operation.

[0072] Step 3: In the Transformer, the Key matrix and the Value matrix are used to perform low-complexity attention mechanism calculations, and the Key matrix and the Value matrix are shared between layers to finally obtain the representation of the sequence.

[0073] Using generated and matrix, performs multi-head self-attention mechanism calculation in the Transformer stacking layer, and shares the Key matrix and Value matrix in each stacking layer; in addition, a linear layer is used to replace the original Position-wise Feed-Forward layer of the Transformer, further reducing the parameter scale; this method adopts a multi-head self-attention mechanism, each head independently calculates the Query matrix in the attention mechanism, and combines the Key matrix and Value matrix generated in step 2 to obtain the attention representation; the results of multiple attention heads are spliced ​​and embedded using a linear layer to generate the final context representation, that is, the final sequence representation. In this way, long-range dependencies in the sequence can be efficiently modeled.

[0074] Specifically, in the first Transformer stack layer, the multi-head self-attention mechanism is defined as follows (the rest of the stack layers are similar):

[0075]

[0076]

[0077]

[0078] in, is the number of heads of multi-head self-attention, i represents the i-th head, is the attention score of the i-th head, represents the softmax function, is the number of feature dimensions, is the input of the first Transformer stack layer, , They are Value matrix and Key matrix respectively. represents multi-head self-attention, is the learnable parameter of the multi-head self-attention mechanism, represents the tensor connection operation, is the output of multi-head self-attention.

[0079] The rest of the Transformer stacking layer is as follows:

[0080]

[0081]

[0082]

[0083]

[0084] in, and are model parameters, Represents the layer normalization operation, where and Respectively The mean and variance of and denote the parameter vectors for scaling and translation, respectively, and is the minimum amount to prevent a divide-by-zero error. As the input of the next stacked layer, it continues to participate in the calculation. After stacking layers, the final sequence representation is obtained .

[0085] This method reduces the scale of model parameters through inter-layer sharing mechanism and parameter sharing strategy, shares the parameters of compressed Key matrix and Value matrix between multiple block layers of Transformer to reduce the amount of calculation; uses linear layer to replace the feedforward network layer in traditional Transformer to further reduce the computational cost; these optimization measures improve the scalability of Transformer in large-scale data processing.

[0086] Step 4: Input the sequence representation into the downstream task module to perform downstream tasks.

[0087] In the downstream task module, the input sequence representation , to perform specific downstream tasks, such as text classification, text generation, time series classification, time series regression prediction, etc. The downstream task layer is generally composed of several linear layers.

[0088] Example 2

[0089] In one embodiment of the present application, a large model optimization system based on a sequence-dependent hierarchical propagation mechanism is provided. The large model includes a representation learning module and a downstream task module. The representation learning module includes:

[0090] The language processing module is configured to: perform natural language processing on the input text to obtain a word vector sequence;

[0091] The hierarchical dependency module is configured to: divide the word vector sequence into even blocks based on the sequence dependency hierarchical propagation mechanism, and use the attention mechanism between segments and within segments to generate Matrix and matrix;

[0092] The sequence representation module is configured to: Matrix and The matrix is ​​used to calculate the Transformer's attention mechanism to obtain the final sequence representation, which is used as the input of the downstream task module for specific downstream tasks.

[0093] Example 3

[0094] In one embodiment of the present application, a computer program product is provided, including a computer program, which, when executed by a processor, implements the large model optimization method based on the sequence-dependent hierarchical propagation mechanism.

[0095] Example 4

[0096] In one embodiment of the present application, a non-transitory computer-readable storage medium is provided, which is used to store computer instructions. When the computer instructions are executed by a processor, the large model optimization method based on the sequence-dependent hierarchical propagation mechanism is implemented.

[0097] Example 5

[0098] In one embodiment of the present application, an electronic device is provided, comprising: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the large model optimization method based on the sequence-dependent hierarchical propagation mechanism.

[0099] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0101] Although the above describes the specific implementation methods of the present application in conjunction with the accompanying drawings, it does not limit the scope of protection of the present application. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present application, various modifications or variations that can be made by technical personnel in this field without creative work are still within the scope of protection of the present application.

Claims

1. A large model optimization method based on a sequence-dependent hierarchical propagation mechanism, wherein the large model includes a representation learning module and a downstream task module, characterized in that: The specific steps of the representation learning module are: Perform natural language processing on the input text to obtain a word vector sequence; Based on the sequence-dependent hierarchical propagation mechanism, the word vector sequence is evenly divided into blocks, and the attention mechanism between segments and within segments is used to generate Matrix and Matrix, the specific steps are: Perform high-dimensional embedding and position encoding on the word vector sequence to obtain an embedding representation containing position information; Fill the embedded representation containing position information, divide the filled embedded representation into blocks evenly, and perform mean pooling on each block to obtain a preliminary segment-level representation; The inter-fragment attention mechanism and the intra-fragment attention mechanism are used on the preliminary fragment-level representation to obtain the final inter-fragment representation and fine-grained intra-fragment representation; The inter-segment attention mechanism obtains the dependencies between segments and obtains the final segment-level representation by calculating the attention distribution between different segments. The intra-fragment attention mechanism is to obtain intra-fragment dependencies and obtain fine-grained intra-fragment representations by calculating the attention matrix in each fragment; Concatenate the inter-fragment representation and the intra-fragment representation to obtain the Key matrix and the Value matrix; use Matrix and The matrix is ​​used to calculate the Transformer's attention mechanism to obtain the final sequence representation, which is used as the input of the downstream task module for specific downstream tasks.

2. A large model optimization method based on a sequence-dependent hierarchical propagation mechanism according to claim 1, characterized in that: The natural language processing includes word segmentation, word list construction, and word embedding.

3. A large model optimization method based on a sequence-dependent hierarchical propagation mechanism according to claim 1, characterized in that: The Transformer's attention mechanism calculation is to perform multi-head self-attention mechanism calculation in the Transformer stacking layer, the Key matrix and Value matrix between multiple stacking layers, and use linear layers to replace the feedforward network layer in the traditional Transformer.

4. A large model optimization method based on a sequence-dependent hierarchical propagation mechanism according to claim 1, characterized in that: The downstream tasks include but are not limited to text classification, text generation, time series classification, and time series regression prediction.

5. A large model optimization system based on a sequence-dependent hierarchical propagation mechanism, wherein the large model includes a representation learning module and a downstream task module, characterized in that: The representation learning module includes: The language processing module is configured to: perform natural language processing on the input text to obtain a word vector sequence; The hierarchical dependency module is configured to: divide the word vector sequence into even blocks based on the sequence dependency hierarchical propagation mechanism, and use the attention mechanism between segments and within segments to generate Matrix and Matrix, the specific steps are: Perform high-dimensional embedding and position encoding on the word vector sequence to obtain an embedding representation containing position information; Fill the embedded representation containing position information, divide the filled embedded representation into blocks evenly, and perform mean pooling on each block to obtain a preliminary segment-level representation; The inter-fragment attention mechanism and the intra-fragment attention mechanism are used on the preliminary fragment-level representation to obtain the final inter-fragment representation and fine-grained intra-fragment representation; The inter-segment attention mechanism obtains the dependencies between segments and obtains the final segment-level representation by calculating the attention distribution between different segments. The intra-fragment attention mechanism is to obtain intra-fragment dependencies and obtain fine-grained intra-fragment representations by calculating the attention matrix in each fragment; Concatenate the inter-fragment representation and the intra-fragment representation to obtain the Key matrix and the Value matrix; The sequence representation module is configured to: Matrix and The matrix is ​​used to calculate the Transformer's attention mechanism to obtain the final sequence representation, which is used as the input of the downstream task module for specific downstream tasks.

6. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the large model optimization method based on the sequence-dependent hierarchical propagation mechanism described in any one of claims 1 to 4 is implemented.

7. A non-transitory computer-readable storage medium, characterized in that The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, a large model optimization method based on a sequence-dependent hierarchical propagation mechanism as described in any one of claims 1 to 4 is implemented.

8. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement a large model optimization method based on a sequence-dependent hierarchical propagation mechanism as described in any one of claims 1 to 4.

Images