Large model output layer optimization method, system and device based on word co-occurrence and storage medium
By optimizing the parameters and vocabulary of the output layer of a large model based on word co-occurrence data, the problems of high computational overhead and high memory pressure of large models are solved, achieving optimization effects with low computational redundancy and low memory usage.
Patent Information
- Application Number
- CN202511607854.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-05
- Publication Date
- 2025-12-26
AI Technical Summary
Large models suffer from high computational overhead, high memory and bandwidth pressure, and significant computational redundancy in their output layer. Existing optimization methods are ill-suited for dynamic vocabulary and multi-domain tasks.
By acquiring word co-occurrence data of the current input sequence, the parameter matrix and vocabulary of the output layer of the large model are optimized, reducing redundant computation and lowering the requirements for GPU memory and bandwidth.
It effectively reduces the computational overhead and memory requirements of the output layer of large models, reduces computational redundancy, and provides an optimization method with low computing power and low memory requirements.
Smart Images
Figure CN121212221A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of artificial intelligence and natural language processing (NLP) technology, and more particularly to a method, system, device, and storage medium for optimizing the output layer of a large model based on word co-occurrence. Background Technology
[0002] With the development of deep learning and pre-training techniques, large models based on the Transformer architecture have been widely used in natural language processing tasks such as machine translation, text generation, dialogue systems, and code generation. These models typically contain billions to hundreds of billions of parameters, among which the vocabulary mapping of the output layer is a crucial computational step in the inference stage.
[0003] In typical large models, the output layer usually uses a fully connected structure to map the hidden layer vectors to a vocabulary space, the dimension of which is equal to the vocabulary size. The vocabulary can contain tens of thousands to hundreds of thousands of tokens. This process requires large-scale matrix multiplication and softmax normalization operations, and the computational complexity is linearly related to the vocabulary size, leading to the following problems: The computational overhead is high. For a vocabulary containing 50,000 to 200,000 words, each generated token requires calculating the dot product between the hidden layer vector and the entire vocabulary matrix, and also requires exponential operations and normalization. As the model parameter size and inference length increase, the output layer computation accounts for a significant proportion of the computational power and latency in the inference stage.
[0004] High memory and bandwidth pressure. The output layer weight matrix has a large number of parameters (for example, when the hidden layer dimension is 4,096 and the vocabulary is 50,000, the number of parameters exceeds 200 million). The matrix needs to be loaded frequently during inference, resulting in high memory usage and bandwidth bottleneck.
[0005] The computational redundancy is significant. In many task scenarios, the final output probability of the model is concentrated on a small subset of words, with most words having extremely low probabilities. However, traditional Softmax requires calculating complete scores and normalized values for all words, resulting in a large amount of ineffective computation.
[0006] To address the aforementioned problems, several optimization schemes have emerged in existing technologies, such as hierarchical softmax, sampled softmax, adaptive softmax, and vocabulary pruning. However, these methods have the following shortcomings: The hierarchical and adaptive method needs a complex vocabulary grouping or clustering strategy, and is difficult to adapt to dynamic vocabulary and multi-field tasks.
[0007] The sampling Softmax cannot completely avoid calculating the normalization probability of all tokens in the inference stage, and still has redundant calculation.
[0008] The vocabulary pruning method sacrifices certain generation flexibility and accuracy, and needs to frequently adjust the vocabulary for different tasks.
[0009] Therefore, the existing large model output layer still has a large optimization space in terms of computing power occupation, memory consumption and inference delay, and an output layer vocabulary computing power optimization method is needed, which can reduce redundant calculation and maintain generation accuracy and adaptability. SUMMARY
[0010] In view of the above defects of the prior art, the present application provides a large model output layer optimization method, system and device based on word co-occurrence and a storage medium, which aims to solve the problems of large calculation overhead, high memory and bandwidth pressure and significant calculation redundancy caused by the size of the vocabulary of the large model.
[0011] In order to solve the above technical problems, the technical scheme adopted by the present application is as follows: The first aspect of the present application provides a large model output layer optimization method based on word co-occurrence, which comprises: obtaining first word co-occurrence data of the last token of the current input sequence, and calculating the current first word co-occurrence data according to the first word co-occurrence data; optimizing the large model output layer parameter matrix according to the current first word co-occurrence data to obtain a first parameter matrix; optimizing the large model output layer vocabulary according to the current first word co-occurrence data to obtain first vocabulary data; using the first parameter matrix as the current large model output layer parameter matrix, and using the first vocabulary data as the current large model output layer vocabulary to perform the large model output layer calculation.
[0012] The large model output layer optimization method based on word co-occurrence, wherein the first word co-occurrence data construction process comprises: obtaining corpus and vocabulary data; constructing the first word co-occurrence data of each vocabulary item according to the corpus and the vocabulary data; storing the first word co-occurrence data of each vocabulary item.
[0013] The large model output layer optimization method based on word co-occurrence, wherein the first word co-occurrence data of each vocabulary item is constructed according to the corpus and the vocabulary data, comprising: The corpus fragments are read sequentially, and the corpus fragments are converted into a sequence of terms in the word list according to the corpus order. The next consecutive word in the word list sequence is added to the word co-occurrence list of the current word list sequence item in turn; The vocabulary is traversed, and the first word co-occurrence data of each vocabulary item is calculated based on the word co-occurrence list of each vocabulary item.
[0014] The large model output layer optimization method based on word co-occurrence, wherein the first word co-occurrence data includes each item in the first word co-occurrence data being unique in the first word co-occurrence data.
[0015] The aforementioned optimization method for the output layer of a large model based on word co-occurrence, wherein the storage method for storing the first word co-occurrence data for each vocabulary entry includes: Local file system, using structured text or binary format for storage; Key-value databases include in-memory key-value caches and persistent key-value databases; A relational database stores word list entries in association with the first word co-occurrence data through a mapping table, or stores each word list entry and its corresponding first word co-occurrence data entry together as a single record.
[0016] The large model output layer optimization method based on word co-occurrence, wherein calculating the current first word co-occurrence data based on the first word co-occurrence data includes directly using the first word co-occurrence data as the current first word co-occurrence data.
[0017] The large model output layer optimization method based on word co-occurrence, wherein calculating the current first word co-occurrence data based on the first word co-occurrence data includes calculating the current first word co-occurrence data based on the word list corresponding to the current context token and the first word co-occurrence data of the last token of the current input sequence.
[0018] In a second aspect, the present invention provides a large model output layer optimization system based on word co-occurrence, the system comprising: Retrieval module: retrieves and outputs the first word co-occurrence data of the last token in the current input sequence; Word co-occurrence calculation module: Calculates the current co-occurrence data of the first word based on the first word co-occurrence data output by the retrieval module; Optimization module: Optimizes the parameter matrix and vocabulary of the output layer of the large model based on the current co-occurrence data of the first word to obtain the first parameter matrix and first vocabulary data; replaces the parameter matrix and vocabulary of the current output layer with the first parameter matrix and the first vocabulary data, and participates in the output layer calculation.
[0019] A third aspect of the present invention provides a large model output layer optimization device based on word co-occurrence, the device comprising: Retrieval module: retrieves and outputs the first word co-occurrence data of the last token in the current input sequence; Word co-occurrence calculation module: Calculates the current co-occurrence data of the first word based on the first word co-occurrence data output by the retrieval module; Optimization module: Optimizes the parameter matrix and vocabulary of the output layer of the large model based on the current co-occurrence data of the first word to obtain the first parameter matrix and first vocabulary data; replaces the parameter matrix and vocabulary of the current output layer with the first parameter matrix and the first vocabulary data, and participates in the output layer calculation.
[0020] In a fourth aspect, the present invention provides a large model output layer optimization storage medium based on word co-occurrence, wherein the large model output layer optimization storage medium based on word co-occurrence stores computer instructions and / or the first word co-occurrence data, and the computer instructions are used to cause a processor to implement the large model output layer optimization method based on word co-occurrence as described in any one of claims 1-7 when executed.
[0021] Therefore, it can be seen that: The method, system, apparatus, and storage medium of this invention effectively solve the technical challenges of high computational overhead, high GPU memory and bandwidth pressure, and high computational redundancy in the output layer of large models. It also provides technical support for larger-scale vocabularies and offers a method, system, apparatus, and storage medium with low computational power, low GPU memory overhead, and low computational redundancy for large models. By combining word co-occurrence statistics based on the corpus with the co-occurrence data of the first word of the last token of the current input, this invention reduces the size of the parameter matrix and the vocabulary of the output layer of the large model, thereby reducing the computational power, GPU memory, and bandwidth requirements of the output layer and minimizing computational redundancy. Attached Figure Description
[0022] Figure 1 A flowchart of a method for constructing the first word co-occurrence data provided in an embodiment of the present invention;
[0023] Figure 2 The flowchart illustrates a method, system, apparatus, and storage medium for optimizing the output layer of a large model based on word co-occurrence, as provided in this embodiment of the invention. Detailed Implementation
[0024] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0025] Please see Figure 1 This invention proposes a method for constructing co-occurrence data of the first word, the method comprising:
[0026] S101. Obtain the corpus and the vocabulary data.
[0027] Optionally, the corpus includes a collection of text fragments.
[0028] Optionally, the vocabulary data includes the vocabulary of the large model.
[0029] Optionally, the vocabulary data may further include the parameter matrix of the output layer of the large model.
[0030] In one exemplary example, all ancient Chinese texts are obtained as the corpus, and the vocabulary of deepseek R1 is used as the vocabulary data.
[0031] In another exemplary example, the texts of all ancient and modern books, both Chinese and foreign, are obtained as the corpus, and the vocabulary of qwen3 is used as the vocabulary data.
[0032] S102. Construct the first word co-occurrence data for each word entry based on the corpus and word list data.
[0033] Optionally, this step includes:
[0034] Based on the corpus, compile a list of the next optional word entries for each word entry in the corpus;
[0035] Based on the list of next optional vocabulary entries for each vocabulary entry in the corpus, construct the first word co-occurrence data for each vocabulary entry.
[0036] Optionally, the first word co-occurrence data includes the next optional word entry list.
[0037] Optionally, the first word co-occurrence data further includes the parameter matrix of the large model output layer corresponding to the next optional word list. The parameter matrix of the large model output layer corresponding to the next optional word list includes the parameter matrix in the large model output layer that is related to the word list output of the next optional word list.
[0038] In an exemplary example, for word term a, the corpus is traversed to obtain a set A consisting of all word terms that immediately follow word term a in the corpus. Set A is the list of the next possible word terms for word term a. Set A is directly used as the first word co-occurrence data for word term a.
[0039] In another exemplary example, the first word co-occurrence data of the vocabulary item a also includes the parameter matrix composed of the parameters of the large model output layer that participate in the calculation of the vocabulary item output corresponding to the set A.
[0040] S103. Store the co-occurrence data of the first word for each word entry.
[0041] This step stores the first word co-occurrence data for each of the aforementioned word entries.
[0042] Optionally, the present invention stores the first word co-occurrence data of each of the vocabulary entries in a file system. In an exemplary example, the first word co-occurrence data is stored in a JSON file in JSON format.
[0043] Optionally, the present invention stores the first word co-occurrence data of each vocabulary entry in a key-value database. In an exemplary example, the vocabulary entry is used as the key, and the first word co-occurrence data corresponding to the vocabulary entry is used as the value to store the data in a Redis database.
[0044] Optionally, the present invention stores the first word co-occurrence data of each word entry in a relational database. In an exemplary example, the word entry and the corresponding first word co-occurrence data are used as table fields, and the first word co-occurrence data of each word entry is stored in a MySQL table.
[0045] Figure 2 This is a flowchart of a large model output layer optimization method based on word co-occurrence proposed in an embodiment of the present invention, as follows: Figure 2 As shown, the process includes the following steps:
[0046] S201. Obtain the first word co-occurrence data of the last token in the current input sequence, and calculate the current first word co-occurrence data based on the first word co-occurrence data.
[0047] Optionally, this step includes:
[0048] Based on the storage method of the first word co-occurrence data and the last token of the current input sequence as the word list entry, the first word co-occurrence data of the last token of the current input sequence is obtained;
[0049] The current co-occurrence data of the first word is calculated based on the co-occurrence data of the first word.
[0050] Optionally, the current co-occurrence data of the first word includes the co-occurrence data of the first word of the last token in the current input sequence.
[0051] Optionally, the current first word co-occurrence data further includes using the word list corresponding to the current context as the next optional word list in step S102, and constructing the first word co-occurrence data according to step S102. Optionally, the first word co-occurrence data can be calculated based on a subset of the word list corresponding to the current context. In an exemplary example, the first word co-occurrence data is constructed after removing preposition word list items from the context and / or removing high-probability word list items based on their occurrence probability.
[0052] In an exemplary example, after obtaining the first word co-occurrence data of the last token of the current input sequence from Redis using the last token of the current input sequence as the key, the word list item list corresponding to the current context is used as the next optional word list item list in step S102, and the first word co-occurrence data corresponding to the current context is calculated. Then, it is merged with the first word co-occurrence data of the last token of the current input sequence to obtain the current first word co-occurrence data.
[0053] S202. Optimize the output layer parameter matrix of the large model based on the current co-occurrence data of the first word to obtain the first parameter matrix.
[0054] Optionally, if the current first word co-occurrence data includes the parameter matrix of the large model output layer corresponding to the next optional word list, then the first parameter matrix is the parameter matrix of the large model output layer corresponding to the next optional word list.
[0055] Optionally, if the current first word co-occurrence data does not include the parameter matrix of the large model output layer corresponding to the next optional word list, then the first parameter matrix is a matrix composed of parameters related to the word list calculation in the large model output layer calculation that are related to the word list calculation of the next optional word list corresponding to the current first word co-occurrence data.
[0056] In an exemplary example, the first parameter matrix is obtained by acquiring only the matrix composed of the large model output layer parameters related to the word list corresponding to the next optional word list in the current first word co-occurrence data.
[0057] S203. Optimize the output layer vocabulary of the large model based on the current co-occurrence data of the first word to obtain the first vocabulary data.
[0058] Optionally, the word list is filtered out, and only the word list entries corresponding to the next optional word list entries in the current first word co-occurrence data are retained to obtain the first word list data.
[0059] In one exemplary example, the list of next optional word entries in the current first word co-occurrence data is directly used as the first word list data.
[0060] S204. Using the first parameter matrix as the parameter matrix of the current large model output layer, and using the first vocabulary data as the vocabulary of the current large model output layer, the large model output layer is calculated.
[0061] Optionally, this step includes:
[0062] The first parameter matrix is used as the output layer parameter matrix of the current output layer of the large model in the calculation.
[0063] The first vocabulary data is used as the vocabulary of the current output layer of the large model for calculation.
[0064] In an exemplary example, let the last token of the current input sequence be the word term i, and the list of the next selectable word terms in its first word co-occurrence data be: The parameter matrix corresponding to the vocabulary item i is and Then the first parameter moment is and The first vocabulary data is .
[0065] The method embodiments described above are merely illustrative. Those skilled in the art can select some or all of the steps to achieve the purpose of this embodiment without creative effort, according to actual needs.
[0066] Based on the above method embodiments, this invention also proposes a large model output layer optimization system based on word co-occurrence, the system comprising:
[0067] Retrieval module: retrieves and outputs the first word co-occurrence data of the last token in the current input sequence;
[0068] Word co-occurrence calculation module: Calculates the current co-occurrence data of the first word based on the first word co-occurrence data output by the retrieval module;
[0069] Optimization module: Optimizes the parameter matrix and word list of the large model output layer based on the current first word co-occurrence data to obtain the first parameter matrix and first word list data; replaces the parameter matrix and word list of the large model output layer with the first parameter matrix and the first word list data, and participates in the calculation of the large model output layer.
[0070] The above system embodiments and method embodiments are one-to-one correspondences. For a brief description of the system embodiments, please refer to the method embodiments.
[0071] The system embodiments described above are merely illustrative. Those skilled in the art can select some or all of the modules to achieve the purpose of this embodiment without creative effort, based on actual needs.
[0072] Based on the above method embodiments, the present invention also proposes a large model output layer optimization device based on word co-occurrence, the device comprising:
[0073] Retrieval module: retrieves and outputs the first word co-occurrence data of the last token in the current input sequence;
[0074] Word co-occurrence calculation module: Calculates the current co-occurrence data of the first word based on the first word co-occurrence data output by the retrieval module;
[0075] Optimization module: Optimizes the parameter matrix and word list of the large model output layer based on the current first word co-occurrence data to obtain the first parameter matrix and first word list data; replaces the parameter matrix and word list of the large model output layer with the first parameter matrix and the first word list data, and participates in the calculation of the large model output layer.
[0076] The above device embodiments and method embodiments are one-to-one correspondences. For a brief description of the device embodiments, please refer to the method embodiments.
[0077] The device embodiments described above are merely illustrative. Those skilled in the art can select some or all of the modules to achieve the purpose of this embodiment without any creative effort, based on actual needs.
[0078] Based on the above method embodiments, the present invention also proposes a large model output layer optimization storage medium based on word co-occurrence. The large model output layer optimization storage medium based on word co-occurrence stores computer instructions and / or the first word co-occurrence data. The computer instructions are used to cause the processor to implement the large model output layer optimization method based on word co-occurrence as described in any one of claims 1-7 when executed.
[0079] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for optimizing the output layer of a large model based on word co-occurrence, characterized in that, The method includes: Obtain the co-occurrence data of the first word of the last token in the current input sequence, and calculate the current co-occurrence data of the first word based on the co-occurrence data of the first word; The first parameter matrix is obtained by optimizing the output layer parameter matrix of the large model based on the current co-occurrence data of the first word. The first word list data is obtained by optimizing the output layer vocabulary of the large model based on the current co-occurrence data of the first word. The first parameter matrix is used as the parameter matrix of the current large model output layer, and the first vocabulary data is used as the vocabulary of the current large model output layer for the calculation of the large model output layer.
2. The method for optimizing the output layer of a large model based on word co-occurrence as described in claim 1, characterized in that, The process of constructing the first word co-occurrence data includes: Obtain corpus and vocabulary data; Construct the first word co-occurrence data for each word entry based on the corpus and the vocabulary data; Store the first word co-occurrence data for each of the vocabulary entries.
3. The method for optimizing the output layer of a large model based on word co-occurrence according to claim 2, characterized in that, The construction of the first word co-occurrence data for each word entry based on the corpus and the vocabulary includes: The corpus fragments are read sequentially, and the corpus fragments are converted into a sequence of terms in the word list according to the corpus order. The next consecutive word in the word list sequence is added to the word co-occurrence list of the current word list sequence item in turn; The vocabulary is traversed, and the first word co-occurrence data of each vocabulary item is calculated based on the word co-occurrence list of each vocabulary item.
4. The method for optimizing the output layer of a large model based on word co-occurrence according to claim 2, characterized in that, The first word co-occurrence data includes each item in the first word co-occurrence data being unique within the first word co-occurrence data.
5. The method for optimizing the output layer of a large model based on word co-occurrence according to claim 2, characterized in that, The storage method for storing the first word co-occurrence data of each vocabulary entry includes: Local file systems use structured text or binary formats for storage; Key-value databases include in-memory key-value caches and persistent key-value databases; A relational database stores word list entries in association with the first word co-occurrence data through a mapping table, or stores each word list entry and its corresponding first word co-occurrence data entry together as a single record.
6. The method for optimizing the output layer of a large model based on word co-occurrence according to claim 1, characterized in that, The step of calculating the current co-occurrence data of the first word based on the first word co-occurrence data includes directly using the first word co-occurrence data as the current co-occurrence data of the first word.
7. The method for optimizing the output layer of a large model based on word co-occurrence according to claim 1, characterized in that, The step of calculating the current co-occurrence data of the first word based on the first word co-occurrence data includes calculating the current co-occurrence data of the first word based on the word list corresponding to the current context token and the first word co-occurrence data of the last token of the current input sequence.
8. A large model output layer optimization system based on word co-occurrence, characterized in that, The system includes: Retrieval module: retrieves and outputs the first word co-occurrence data of the last token in the current input sequence; Word co-occurrence calculation module: Calculates the current co-occurrence data of the first word based on the first word co-occurrence data output by the retrieval module; Optimization module: Optimizes the parameter matrix and vocabulary of the large model output layer based on the current co-occurrence data of the first word to obtain the first parameter matrix and the first vocabulary data; replaces the parameter matrix and vocabulary of the current output layer with the first parameter matrix and the first vocabulary data, and participates in the output layer calculation.
9. A large model output layer optimization device based on word co-occurrence, characterized in that, The device includes: Retrieval module: retrieves and outputs the first word co-occurrence data of the last token in the current input sequence; Word co-occurrence calculation module: Calculates the current co-occurrence data of the first word based on the first word co-occurrence data output by the retrieval module; Optimization module: Optimizes the parameter matrix and vocabulary of the large model output layer based on the current co-occurrence data of the first word to obtain the first parameter matrix and the first vocabulary data; replaces the parameter matrix and vocabulary of the current output layer with the first parameter matrix and the first vocabulary data, and participates in the output layer calculation.
10. An optimized storage medium for the output layer of a large model based on word co-occurrence, characterized in that, The large model output layer optimization storage medium based on word co-occurrence stores computer instructions and / or the first word co-occurrence data. The computer instructions are used to cause the processor to implement the large model output layer optimization method based on word co-occurrence as described in any one of claims 1-7.