Code generation method and device based on large language model, equipment and storage medium

Abstract

The invention discloses a code generation method and device based on a large language model, equipment and a storage medium, and relates to the technical field of large models, and the method comprises the steps: generating a current mark based on an initial reasoning path used for code generation of the model, caching a query vector of the current mark, and constructing a selection window based on the query vectors of a plurality of to-be-processed marks; a target model layer obtained based on a selection window and a sliding window attention processing historical mark of a target attention head are used to obtain an importance scoring result and a paragraph generation result; dividing the to-be-compressed mark sequence into semantic blocks by using a semantic block division strategy determined based on an importance scoring result and a semantic sparsity result of each paragraph generation result, and generating a target compression strategy based on an importance score of each semantic block and by using a compression control parameter determined based on model characteristics and reasoning requirements, and compressing the initial reasoning path by using a target compression strategy to obtain a target reasoning path to generate a code. Therefore, the code generation efficiency can be improved.

Description

Technical Field

[0001] The present invention relates to the technical field of large models, and particularly to a code generation method, device, equipment and storage medium based on a large language model. Background Art

[0002] Currently, with the wide application of large language models (LLMs) in high-complexity tasks such as logical reasoning, mathematical calculation, and code generation, explicit reasoning strategies such as chain of thought are widely adopted. These strategies improve the interpretability and accuracy of the model by generating detailed intermediate reasoning paths, but also bring significant computational overhead, especially causing the rapid expansion of the KV (Key-Value) cache during the reasoning stage, seriously consuming video memory resources and affecting the reasoning speed.

[0003] Current KV compression technologies mostly focus on the input prompt stage, that is, when the model receives long text or context information, the KV entries are reduced by means of cache truncation, summary extraction, etc. However, during the model reasoning and generation stage, the tokens are generated step by step, and their importance is unpredictable during generation, making it difficult to directly apply existing compression methods. Once the KV information of key tokens is deleted by mistake, it may seriously damage the model's context understanding and affect the accuracy and consistency of the final output.

[0004] Further research finds that there is a common semantic sparsity phenomenon in the generation process of inference-based LLMs: that is, the inference paths often contain a large number of segments with repeated content, semantic redundancy, or logical idling. Although these tokens occupy a large amount of KV cache, they contribute limitedly to the final inference result. Previous compression methods have not fully utilized this feature and cannot effectively identify and eliminate invalid tokens during the reasoning stage, resulting in storage waste and performance bottlenecks.

[0005] As can be seen from the above, how to improve the efficiency of code generation in the process of code generation based on a large language model is an urgent problem to be solved at present. Summary of the Invention

[0006] In view of this, the purpose of the present invention is to provide a code generation method, device, equipment and storage medium based on a large language model, which can improve the efficiency of code generation in the process of code generation based on a large language model, and further reduce the memory for storing the inference path. The specific solutions are as follows:

[0007] In a first aspect, the present application provides a code generation method based on a large language model, including:

[0008] Generate each current token using a preset autoregressive method and based on the initial inference path for code generation corresponding to the large language model, and cache the query vectors corresponding to each of the current tokens. Then, select a preset number of the most recently generated tokens to be processed from each of the current tokens to construct a selection window based on the query vectors corresponding to each of the tokens to be processed;

[0009] Obtain the sliding window attention corresponding to the target model layer and the target attention head based on the selection window, and process the corresponding historical tokens based on the sliding window attention to obtain an importance scoring result and a passage generation result. Then, evaluate each of the passage generation results to obtain a corresponding semantic sparsity result; the target model layer is the model layer corresponding to the historical tokens among the model layers of the large language model; the target attention head is the attention head corresponding to the historical tokens among the attention heads of the large language model;

[0010] Determine a semantic block partitioning strategy based on the importance scoring result and the semantic sparsity result, and partition the token sequence to be compressed including each of the current tokens into semantic blocks based on the semantic block partitioning strategy. Then, sort the semantic blocks based on the importance scores corresponding to each of the semantic blocks to obtain a sorting result;

[0011] Determine compression control parameters based on the model characteristics and inference requirements of the large language model to generate a target compression strategy based on the compression control parameters, and use the target compression strategy to compress the initial inference path corresponding to the large language model to obtain a target inference path; the compression control parameters include a compression interval, a selection window size, and a compression ratio;

[0012] Generate code using the large language model and based on the target inference path.

[0013] Optionally, the step of generating each current token using a preset autoregressive method and based on the initial inference path for code generation corresponding to the large language model, and caching the query vectors corresponding to each of the current tokens, and then selecting a preset number of the most recently generated tokens to be processed from each of the current tokens to construct a selection window based on the query vectors corresponding to each of the tokens to be processed includes:

[0014] Generate each current token using a preset autoregressive method and based on the initial inference path corresponding to the large language model, and determine the query vector, key vector, and value vector corresponding to each of the current tokens;

[0015] Store each of the query vectors, the key vectors, and the value vectors in a query buffer and a full key-value cache area; wherein, the query buffer only retains the query vectors corresponding to a preset number of the most recently generated current tokens;

[0016] Extract the preset number of query vectors from the query buffer, and determine a window construction criterion based on each of the query vectors, so as to construct a selection window with each of the current tokens based on the window construction criterion.

[0017] Optionally, obtaining the sliding window attention corresponding to the target model layer and the target attention head respectively based on the selection window, and processing the corresponding historical tokens based on the sliding window attention to obtain an importance scoring result and a paragraph generation result, including:

[0018] Determine each model layer and each attention head in the large language model, and determine the number of model layers and the number of attention head sequences corresponding to the historical tokens, and then determine the sliding window attention corresponding to the historical tokens based on the selection window, the query vector, the key vector, the value vector, the number of model layers, and the number of attention head sequences; wherein, the numerical value of the sliding window attention is proportional to the contribution value of the historical token in the selection window;

[0019] Count the total number of model layers corresponding to each model layer and the total number of attention heads corresponding to each attention head, so as to determine the importance score to be smoothed based on the total number of model layers, the total number of attention heads, and the sliding window attention;

[0020] Establish a paragraph generation result based on each of the current tokens, and smooth the importance score to be smoothed based on the window width corresponding to the selection window to obtain an importance scoring result.

[0021] Optionally, evaluating each of the paragraph generation results to obtain a corresponding semantic sparsity result, including:

[0022] Using a preset entropy value determination formula and generating an initial entropy value result corresponding to each of the preset tuple sets based on the paragraph generation result, a plurality of preset tuple sets, and the probability distribution result corresponding to the paragraph generation result, and determining a target entropy value result based on each of the initial entropy value results;

[0023] Judge whether the target entropy value result is greater than a preset paragraph length threshold. If the target entropy value result is not greater than the preset paragraph length threshold, then judge whether the importance scoring result corresponding to the paragraph generation result is greater than a preset importance scoring threshold. If the importance scoring result corresponding to the paragraph generation result is not greater than the preset importance scoring threshold, then set the area corresponding to the paragraph generation result as a semantic sparse area, and set the token corresponding to the paragraph generation result as a token to be compressed;

[0024] If the importance score result corresponding to the paragraph generation result is greater than the preset importance score threshold, then set the area corresponding to the paragraph generation result as a semantically dense area, and prohibit setting the label corresponding to the paragraph generation result as a label to be compressed;

[0025] If the target entropy value result is greater than the preset paragraph length threshold, then set the area corresponding to the paragraph generation result as a semantically dense area, and prohibit setting the label corresponding to the paragraph generation result as a label to be compressed;

[0026] Alternatively, determine whether the number of selection windows corresponding to generating a preset number of current labels is greater than the preset window number threshold. If the number of selection windows corresponding to generating a preset number of current labels is greater than the preset window number threshold, then set the area corresponding to the paragraph generation result as a semantically sparse area, and set the label corresponding to the paragraph generation result as a label to be compressed.

[0027] Optionally, determining a semantic block division strategy based on the importance score result and the semantic sparsity result, and dividing the sequence of labels to be compressed including each of the current labels into semantic blocks based on the semantic block division strategy, and then sorting based on the importance scores corresponding to each semantic block to obtain a sorting result, including:

[0028] Determine the first label to be compressed from each of the current labels based on the importance score result and the semantic sparsity result, and construct a first sequence of labels to be compressed based on each of the first labels to be compressed, so as to extract the semantic representation vectors corresponding to each of the first labels to be compressed from the first sequence of labels to be compressed; the semantic representation vector is a context vector corresponding to the model layer and a weighted combination vector;

[0029] Use a preset clustering algorithm to divide each of the semantic representation vectors into several clusters that are semantically related, and perform label assignment on each of the first labels to be compressed based on each cluster, so as to determine semantic blocks based on the label assignment result; the arrangement order of each of the first labels to be compressed corresponding to each semantic block is the same as the arrangement order of each of the first labels to be compressed in the first sequence of labels to be compressed;

[0030] Perform an overall importance score on each of the semantic blocks based on the importance score result to obtain the corresponding importance scores, and sort each of the semantic blocks in ascending order of each of the importance scores to obtain a sorting result.

[0031] Optionally, determining compression control parameters based on the model characteristics and inference requirements of the large language model, so as to generate a target compression strategy based on the compression control parameters, and using the target compression strategy to compress the initial inference path corresponding to the large language model to obtain a target inference path, including:

[0032] Determine compression control parameters based on the model characteristics and inference requirements of the large language model, and generate a target compression strategy based on the compression control parameters; wherein, the model characteristics include each model layer and each attention head;

[0033] Determine the number of semantic block retention blocks based on the compression interval and the length of the semantic blocks corresponding to the semantic blocks, so as to retain the full-scale key-value buffer areas corresponding to the first-mentioned number of semantic block retention blocks of the semantic blocks in the sorting result, and set the current tokens corresponding to the remaining semantic blocks as the second tokens to be compressed, and construct a second sequence to be compressed based on each of the second tokens to be compressed;

[0034] Record the position information, importance scores, and corresponding semantic block numbers of each of the second tokens to be compressed, and use the target compression strategy and the second sequence to be compressed to compress the initial inference path corresponding to the large language model to obtain a target inference path.

[0035] Optionally, after generating the code using the large language model and based on the target inference path, it further includes:

[0036] Detect and collect metrics for the process of generating the target inference path, and use the collected metrics to adjust the compression interval, the selection window size, and the compression ratio in the compression control parameters to obtain new compression control parameters; the metrics include n-gram semantic entropy, average token importance, full-scale key-value buffer growth rate, model output confidence fluctuation, and compression rollback rate;

[0037] Record the current token step number, and use a preset compression suppression window mechanism to determine a compression cooling period based on the current token step number and the target compression suppression window length, so as to suppress the compression operation during the compression cooling period; wherein, the target compression suppression window length is a compression suppression window length determined based on a preset adjustment coefficient, a preset basic cooling length, the model output confidence decrease amount, and the compression rollback rate; the preset adjustment coefficient is used to amplify the influence of the collected unstable feedback information on the compression cooling period;

[0038] Construct an index list based on the target inference path, and use a preset weak context recovery mechanism to process the target inference path based on the meta-information corresponding to the target inference path and the cache pointer corresponding to the model layer to obtain a new target inference path; the preset weak context recovery mechanism is used to restore a preset amount of full-scale key-value buffer information in the full-scale key-value buffer area when it is monitored that the inference quality corresponding to the target inference path decreases;

[0039] Structurally record the meta-information corresponding to the target inference path in the compressed log; the meta-information includes the position of the target inference path in the to-be-compressed token sequence, the importance score, the semantic block number, the token step number, and the soft deletion flag indicating whether recovery is allowed.

[0040] Determine whether the target inference path meets the preset generation end condition. If the target inference path meets the preset generation end condition, end the current process of generating the target inference path; the target inference path includes the number of compression execution rounds, the compression ratio, the semantic entropy change amount, the total number of compressed tokens, the number of rollback trigger times, and the full-key value cache size; the preset generation end condition includes the termination flag token generation, the length of the target inference path reaching the preset length threshold, and the confidence of the large language model meeting the preset confidence convergence condition.

[0041] In a second aspect, the present application provides a code generation device based on a large language model, including:

[0042] A selection window determination module, configured to generate each current token using a preset autoregressive method and based on the initial inference path for code generation corresponding to the large language model, cache the query vector corresponding to each current token, and then select a preset number of the most recently generated to-be-processed tokens from each current token to construct a selection window based on the query vectors corresponding to the to-be-processed tokens respectively.

[0043] A semantic sparsity result determination module, configured to obtain the sliding window attention corresponding to the target model layer and the target attention head based on the selection window, process the corresponding historical tokens based on the sliding window attention to obtain the importance scoring result and the paragraph generation result, and then evaluate each paragraph generation result to obtain the corresponding semantic sparsity result; the target model layer is the model layer corresponding to the historical tokens among the model layers of the large language model; the target attention head is the attention head corresponding to the historical tokens among the attention heads of the large language model.

[0044] A sorting result determination module, configured to determine the semantic block division strategy based on the importance scoring result and the semantic sparsity result, divide the to-be-compressed token sequence including each current token into semantic blocks based on the semantic block division strategy, and then sort based on the importance scores corresponding to the semantic blocks to obtain the sorting result.

[0045] A compression result determination module, configured to determine compression control parameters based on the model characteristics and inference requirements of the large language model, generate a target compression strategy based on the compression control parameters, and use the target compression strategy to compress the initial inference path corresponding to the large language model to obtain a target inference path; the compression control parameters include a compression interval, a selection window size, and a compression ratio;

[0046] A code generation module, configured to generate code by using the large language model and based on the target inference path.

[0047] In a third aspect, the present application provides an electronic device, including:

[0048] A memory, configured to store a computer program;

[0049] A processor, configured to execute the computer program to implement the foregoing code generation method based on a large language model.

[0050] In a fourth aspect, the present application provides a computer-readable storage medium, configured to store a computer program, wherein the computer program, when executed by a processor, implements the foregoing code generation method based on a large language model.

[0051] As can be seen from the above, before performing code generation based on a large language model, the present application needs to generate each current token by using a preset autoregressive manner and based on the initial inference path corresponding to the large language model, cache the query vectors corresponding to each current token, and then select a preset number of recently generated query vectors from each current token to construct a selection window based on each query vector; obtain the sliding window attention corresponding to the historical tokens and each model layer and each attention head in the large language model based on the selection window, process the corresponding historical tokens based on the sliding window attention to obtain an importance scoring result and a paragraph generation result, then evaluate each paragraph generation result to obtain a corresponding semantic sparsity result; determine a semantic block division strategy based on the importance scoring result and the semantic sparsity result, divide the to-be-compressed token sequence including each current token into semantic blocks based on the semantic block division strategy, and then sort the importance scores corresponding to each semantic block to obtain a sorting result; determine compression control parameters based on the model characteristics and inference requirements of the large language model, generate a target compression strategy based on the compression control parameters, and use the target compression strategy to compress the initial inference path corresponding to the large language model to obtain a target inference path; the compression control parameters include a compression interval, a selection window size, and a compression ratio.

[0052] It can be seen that, first, the present application needs to generate each current token using a preset autoregressive method and based on the initial inference path corresponding to the large language model, and cache the query vectors corresponding to each current token. Then, a preset number of the most recently generated query vectors are selected from each current token to construct a selection window based on each query vector. Subsequently, based on the selection window, the sliding window attention corresponding to each historical token and each model layer and each attention head in the large language model is obtained, and the corresponding historical tokens are processed based on the sliding window attention to obtain an importance scoring result and a paragraph generation result. Then, each paragraph generation result is evaluated to obtain a corresponding semantic sparsity result. Then, based on the importance scoring result and the semantic sparsity result, a semantic block division strategy is determined, and the token sequence to be compressed including each current token is divided into semantic blocks based on the semantic block division strategy. Then, the semantic blocks are sorted based on the importance scores corresponding to each semantic block to obtain a sorting result. Finally, compression control parameters are determined based on the model characteristics and inference requirements of the large language model, so as to generate a target compression strategy based on the compression control parameters, and the initial inference path corresponding to the large language model is compressed using the target compression strategy to obtain a target inference path. In this way, the code generation efficiency is improved during the code generation based on the large language model, and thus the memory for storing the inference path is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only the embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings according to the provided drawings without creative efforts.

[0054] Figure 1 It is a flowchart of a code generation method based on a large language model disclosed in the present application;

[0055] Figure 2 It is a schematic structural diagram of a code generation device based on a large language model disclosed in the present application;

[0056] Figure 3 It is a structural diagram of an electronic device disclosed in the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, rather than all of them. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0058] Currently, with the widespread adoption of explicit reasoning strategies such as chain of thought in large language models. Such strategies improve the interpretability and accuracy of the model by generating detailed intermediate reasoning paths, but also bring significant computational overhead. Current KV compression techniques mostly focus on the input prompt stage, that is, when the model receives long text or context information, reducing KV entries through methods such as cache truncation and summary extraction. However, in the model inference generation stage, tokens are generated step by step, and their importance is unpredictable during generation, making it difficult to directly apply existing compression methods. Once the KV information of a key token is deleted by mistake, it may seriously damage the model's context understanding and affect the accuracy and consistency of the final output. Further research finds that there is a common semantic sparsity phenomenon in the generation process of inference-based large language models, and thus this feature is not fully utilized, and invalid tokens cannot be effectively identified and removed during the inference stage, resulting in storage waste and performance bottlenecks. For this reason, the present application provides a code generation method based on a large language model, which can improve the efficiency of code generation during the process of code generation based on a large language model, and further reduce the memory for storing inference paths.

[0059] See Figure 1 As shown, an embodiment of the present invention discloses a code generation method based on a large language model, including:

[0060] Step S11, generating each current token using a preset autoregressive manner and based on an initial inference path corresponding to the large language model for code generation, and caching query vectors corresponding to each of the current tokens, and then selecting a preset number of each recently generated to-be-processed tokens from each of the current tokens to construct a selection window based on the query vectors corresponding to each of the to-be-processed tokens.

[0061] In this embodiment, during the process of code generation based on a large language model, the embodiment of the present application selects to compress the initial inference path corresponding to the large language model for code generation, so as to effectively utilize the semantic sparsity in the inference path, reduce memory occupancy and increase token generation throughput on the premise of ensuring high inference accuracy, thereby realizing efficient inference of the inference-based LLM, and the process is: parameter configuration initialization, inference token generation and multi-layer attention perception, semantic block construction and compression candidate identification, adaptive compression strategy scheduling, KV cache compression and weak context recovery mechanism, and inference termination determination and output.

[0062] It is worth mentioning that parameter configuration initialization, as the preparation stage of the entire inference path compression system, is used to provide necessary control parameters, state cache structures, and module functional units for subsequent compression processes. Among them, the initialization operation is directly related to the efficiency, stability, and inference performance of compression. Therefore, the embodiment of the present application needs to set key parameters according to actual requirements and model characteristics before performing inference path compression, and the parameters are in turn: compression interval : Compression interval corresponding to each trigger of the compression operation; Select window size : Number of recent tokens used to evaluate importance; Compression ratio : Proportion of KV cache retained; Average pooling window size : Used to control the smoothing of local semantic continuity; Number of Transformer layers And the number of attention heads Are all inherent parameters of the model.

[0063] Subsequently, before starting the compression mechanism, the embodiments of this application need to collect the degree to which each generated token is attended to by subsequent tokens in the Transformer layer, and combine semantic sparsity analysis to provide a basis for context contribution measurement for subsequent compression operations, that is, importance scores. The process of determining importance scores is as follows: In the decoding stage of the large language model, the embodiments of this application need to generate new tokens autoregressively, and the expression for generating the current token is as follows:

[0064] ;

[0065] Among them, Is the new token generated by the large language model at the th step, , And Are all output tokens from the 1st step to the th step.

[0066] In this embodiment, after generating the token, the embodiments of this application need to cache the Query, Key, and Value vectors corresponding to the selected window size of tokens into the Query buffer and the full KV cache respectively, and the expression is as follows:

[0067] ;

[0068] Furthermore, after selecting the Query vectors of the most recently generated tokens as the attention source, it is selected to use the attention source to evaluate the contribution of previous tokens. Assuming that the current step corresponding to the generated token is the th step, the embodiments of this application need to construct a selection window based on the most recently generated tokens, and the expression is as follows:

[0069] ;

[0070] Among them, Is the set of vectors selected at the current step (i.e., the selection window), containing the vectors of the most recent tokens, Are respectively from the Step to the generated Query vector, which is the size of the sliding window and is used to control the coverage of the attention source.

[0071] Subsequently, obtain the window attention corresponding to the token for the historical tokens at different Transformer layers and different heads. In a specific implementation, for each historical token , the window corresponding to each token , and the layer and the th head corresponding attention determination formula is as follows:

[0072] ;

[0073] where, is the Query vector at the layer and the th head, is the historical token at the layer and the th head Key vector, is the dimension corresponding to the Key vector and the Query vector.

[0074] It is worth mentioning that the embodiments of the present application need to traverse each layer in the large language model and each attention head , and for each token calculate its corresponding attention , in order to utilize to reflect the contribution of the historical to the current decision window.

[0075] Specifically, using a preset autoregressive method and based on the initial inference path corresponding to the large language model for code generation to generate each current token, and caching the query vectors corresponding to each current token, then selecting a preset number of the most recently generated to-be-processed tokens from each current token, in order to construct a selection window based on the query vectors corresponding to each to-be-processed token, may include: using a preset autoregressive method and based on the initial inference path corresponding to the large language model to generate each current token, and determining the query vector, key vector, and value vector corresponding to each current token; storing each query vector, key vector, and value vector in the query buffer and the full key-value cache area; where the query buffer only retains the query vectors corresponding to a preset number of the most recently generated current tokens; extracting a preset number of query vectors from the query buffer, and determining a window construction criterion based on each query vector, in order to construct a selection window based on the window construction criterion and each current token.

[0076] Step S12: Obtain the sliding window attention corresponding to the target model layer and the target attention head based on the selection window, and process the corresponding historical tokens based on the sliding window attention to obtain the importance scoring result and the paragraph generation result. Then, evaluate each paragraph generation result to obtain the corresponding semantic sparsity result. The target model layer is the model layer corresponding to the historical tokens among the model layers of the large language model. The target attention head is the attention head corresponding to the historical tokens among the attention heads of the large language model.

[0077] In this embodiment, after obtaining the sliding window attention, the embodiments of the present application need to aggregate the sliding window attention of each token under different layers or different heads to form a unified importance score. And the expression is as follows:

[0078] ;

[0079] Where, is the total number of network layers, is the total number of attention heads.

[0080] Further, to avoid important tokens being overwhelmed by neighboring low-score tokens, the embodiments of the present application choose to use local window smoothing:

[0081] ;

[0082] Where, is the smoothed attention intensity, which is used to reduce the masking effect of local fluctuations on important tokens. is the window width, which is used to improve the consistency of the scores of tokens within a semantic segment to reduce misjudgment caused by single-point high attention and ensure a more reasonable compression granularity. In a specific implementation, if tokens all belong to the same logical block, but a certain token in the middle is strongly concerned, then the entire logical block will obtain a higher smoothed score, thereby preventing part of the above logical block from being compressed and discarded.

[0083] In this embodiment, it is necessary to determine whether the currently generated paragraph is redundant, that is, construct a token sliding window and update the counting hash table in real time, and update the value once for each generated token, and set a threshold to trigger the determination of redundant segments. Among them, based on the token sequence corresponding to the current window is the triple entropy, and the expression is as follows:

[0084] ;

[0085] Where, is the set of all possible triples within the current window, is the triple appearance probability in the window.

[0086] Subsequently, if , it is determined as a semantically sparse region, that is, the higher the repetition rate of the token sequence, the smaller, the corresponding region of the currently generated paragraph is a semantically sparse region, where the expression of the token sequence is as follows:

[0087] ;

[0088] Furthermore, if and are both lower than , it is determined as a compressible region; if is higher than or is higher than , the compression action is delayed or abandoned. Among them, in the actual large model inference process, repeated patterns of different lengths may appear at different granularities: short repetitions (such as ) may reflect local grammar templates; medium repetitions (such as ) often correspond to sentence patterns; long repetitions (such as and above) may indicate overall logical redundancy. Also, relying only on a fixed value, such as being 3 may miss some meaningful semantically sparse fragments or cause misjudgments.

[0089] Specifically, based on the selected window, obtain the sliding window attention corresponding to the target model layer and the target attention head respectively, and process the corresponding historical tokens based on the sliding window attention to obtain the importance score result and the paragraph generation result, which may include: determining each model layer and each attention head in the large language model, and determining the number of model layers and the number of attention head sequences corresponding to the historical tokens, and then determining the sliding window attention corresponding to the historical tokens based on the selected window, query vector, key vector, value vector, number of model layers and number of attention head sequences; where the numerical value of the sliding window attention is proportional to the contribution value of the historical token in the selected window; counting the total number of model layers corresponding to each model layer and the total number of attention heads corresponding to each attention head, so as to determine the importance score to be smoothed based on the total number of model layers, total number of attention heads and sliding window attention; establishing the paragraph generation result based on each current token, and smoothing the importance score to be smoothed based on the window width corresponding to the selected window to obtain the importance score result.

[0090] Furthermore, to improve the robustness and sensitivity of detection, the embodiments of the present application introduce multi-scale The entropy analysis mechanism, that is, calculating the corresponding entropy at different values , and , and constructing a more comprehensive sparsity score based on this, and fusing the entropy after obtaining the above entropy, and the expression is as follows:

[0091] ;

[0092] Among them, the value range of is 0.2, 0.5, 0.3.

[0093] Subsequently, after obtaining the importance score and the semantic sparsity detection result, the embodiment of the present application needs to make a decision on whether to compress the tokens, which tokens to compress, and which tokens to retain, and introduce a semantic block aggregation mechanism to perform grouped granularity compression on the tokens instead of single-token granularity compression, so as to maintain the integrity of the semantic structure.

[0094] Subsequently, the embodiment of the present application needs to determine whether the compression condition is met to avoid affecting the inference accuracy due to premature or frequent compression. Among them, the two basic conditions for triggering compression are the token quantity trigger condition: compress a window once every generated tokens and the context state trigger condition; it is worth mentioning that when the following two formulas are satisfied at the same time, the embodiment of the present application can determine that the currently generated content has high repeatability and low semantic density, and trigger the compression operation, and the expressions of the above two formulas are as follows:

[0095] ;

[0096] ;

[0097] Among them, is the average importance, is the multi-scale entropy value.

[0098] Furthermore, the embodiment of the present application can also dynamically adjust , , according to the inference task type or the historical compression effect.

[0099] Specifically, evaluate the generation results of each paragraph to obtain corresponding semantic sparsity results, which may include: determining initial entropy value results corresponding to each preset tuple set by using a preset entropy value determination formula and based on the paragraph generation results, a probability distribution result corresponding to a plurality of preset tuple sets and the paragraph generation results, and determining a target entropy value result based on each initial entropy value result; determining whether the target entropy value result is greater than a preset paragraph length threshold. If the target entropy value result is not greater than the preset paragraph length threshold, then determine whether the importance score result corresponding to the paragraph generation result is greater than a preset importance score threshold. If the importance score result corresponding to the paragraph generation result is not greater than the preset importance score threshold, then set the area corresponding to the paragraph generation result as a semantic sparse area and set the mark corresponding to the paragraph generation result as a mark to be compressed; if the importance score result corresponding to the paragraph generation result is greater than the preset importance score threshold, then set the area corresponding to the paragraph generation result as a semantic dense area and prohibit setting the mark corresponding to the paragraph generation result as a mark to be compressed; if the target entropy value result is greater than the preset paragraph length threshold, then set the area corresponding to the paragraph generation result as a semantic dense area and prohibit setting the mark corresponding to the paragraph generation result as a mark to be compressed; or, determine whether the number of selection windows corresponding to generating a preset number of current marks is greater than a preset window number threshold. If the number of selection windows corresponding to generating a preset number of current marks is greater than the preset window number threshold, then set the area corresponding to the paragraph generation result as a semantic sparse area and set the mark corresponding to the paragraph generation result as a mark to be compressed.

[0100] Step S13: Determine a semantic block division strategy based on the importance score result and the semantic sparsity result, divide the sequence of marks to be compressed including each current mark into semantic blocks based on the semantic block division strategy, and then sort the importance scores corresponding to each semantic block to obtain a sorting result.

[0101] In this embodiment, a semantic block division method based on vector clustering is required to divide the tokens into semantically coherent semantic blocks to avoid interrupting important logical paragraphs during the compression process. The division process is as follows: First, extract the semantic representation vectors of each token from the token sequence to be compressed, and the semantic representation vectors include the context vectors of the Transformer intermediate layer or the multi-layer weighted combination vectors. Subsequently, use a clustering algorithm to divide the semantic representation vectors into several clusters with close semantics, and each cluster constitutes a semantic block. In a specific embodiment, if the current window contains 4096 tokens and the expected length of each block is 32, the number of clusters is set to 128. After clustering, each token is assigned a cluster label, thus constructing a structured set of semantic blocks. To maintain the original order of the tokens, the tokens within each semantic block are sorted in ascending order according to their positions in the original sequence. Subsequently, the embodiments of the present application need to calculate the average importance score of each semantic block as the basis for subsequent compression decision sorting. In addition, for each semantic block The formula for determining the average importance score corresponding to it is as follows:

[0102] ;

[0103] In this way, through the above semantic clustering method, the embodiments of the present application realize that the compression operation is processed in units of semantically consistent blocks while maintaining semantic integrity, greatly reducing the risk of inference chain breakage, and at the same time enhancing the structure perception ability and adaptability of the compression strategy.

[0104] After obtaining the average importance scores corresponding to all semantic blocks, the embodiments of the present application need to sort all semantic blocks according to their importance, select and retain the tokens corresponding to the first several blocks, and then set the remaining tokens as compressible tokens. In a specific embodiment, each cycle retains tokens. If the average length of each semantic block is , the formula for determining the number of blocks to be retained is as follows:

[0105] ;

[0106] Among them, is the total number of tokens to be retained per cycle, is the compression factor, is the number of blocks to be retained.

[0107] That is, all semantic blocks are sorted in descending order according to , and the KV corresponding to the tokens of the first blocks are retained, and then the tokens of the remaining blocks are all marked as compressible candidate tokens.

[0108] Specifically, a semantic block division strategy is determined based on the importance scoring result and the semantic sparsity result, and the to-be-compressed token sequence including each current token is divided into semantic blocks based on the semantic block division strategy. Then, the semantic blocks are sorted based on the importance scores corresponding to each semantic block to obtain a sorting result, which may include: determining a first to-be-compressed token from each current token based on the importance scoring result and the semantic sparsity result, and constructing a first to-be-compressed sequence based on each first to-be-compressed token to extract a semantic representation vector corresponding to each first to-be-compressed token from the first to-be-compressed sequence; the semantic representation vector is a context vector corresponding to the model layer and a weighted combination vector; using a preset clustering algorithm to divide each semantic representation vector into several clusters with semantic relevance, and performing label assignment on each first to-be-compressed token based on each cluster to determine semantic blocks based on the label assignment result; the arrangement order of each first to-be-compressed token in the semantic block is consistent with the arrangement order of each first to-be-compressed token in the first to-be-compressed sequence; performing an overall importance scoring on each semantic block based on the importance scoring result to obtain a corresponding importance score, and sorting each semantic block in ascending order of each importance score to obtain a sorting result.

[0109] It is worth mentioning that regardless of how the compression strategy changes, the last tokens always retain their KV caches and do not participate in compression, and the expression is as follows:

[0110] ;

[0111] Among them, represents the set of caches (KV caches) finally retained. The above KV caches will not be compressed or discarded and are retained in the model's memory for subsequent inference. represents the set of KV caches in the selected important semantic blocks. These blocks are considered semantically important and are therefore retained. represents the set of KV caches of the last tokens.

[0112] Step S14: Determine compression control parameters based on the model characteristics and inference requirements of the large language model, generate a target compression strategy based on the compression control parameters, and use the target compression strategy to compress the initial inference path corresponding to the large language model to obtain a target inference path; the compression control parameters include a compression interval, a selection window size, and a compression ratio.

[0113] In this embodiment, the formula for determining the retained KV cache list is as follows:

[0114] ;

[0115] Among them, Represents the set of tags selected for retention, and the determination formula for the discarded KV cache list is as follows:

[0116] ;

[0117] Subsequently, the embodiments of the present application need to record the position information, score, semantic block number, etc. of each compressed tag for subsequent recovery or analysis.

[0118] In a specific embodiment, if the current cycle length is 4096, the semantic block length is 32, and the compression ratio is 4, that is, only 1024 tags are retained per cycle, which is the number of tags corresponding to 32 semantic blocks. Subsequently, 4096 tags are divided into 128 semantic blocks, then the average importance score of each block is calculated and sorted, and the top 32 blocks are selected from the sorted results for retention, and the rest are discarded. Then, the compression plan is executed, and the remaining tag KV caches are deleted.

[0119] In this embodiment, after KV compression is completed, the embodiments of the present application need to structurally record the meta-information of all compressed tags, including their positions in the original sequence, importance scores, semantic block numbers to which they belong, the tag steps at the time of compression, and soft deletion tags indicating whether recovery is allowed. Among them, the above information will be written into the compression log for subsequent context rollback, compression effect analysis, or policy optimization, so as to ensure the traceability and interpretability of the compression process.

[0120] Specifically, compression control parameters are determined based on the model characteristics and inference requirements of the large language model, a target compression policy is generated based on the compression control parameters, and the initial inference path corresponding to the large language model is compressed using the target compression policy to obtain a target inference path, which may include: determining compression control parameters based on the model characteristics and inference requirements of the large language model, and generating a target compression policy based on the compression control parameters; wherein, the model characteristics include each model layer and each attention head; determining the number of semantic block retention blocks based on the compression interval and the semantic block length corresponding to the semantic block, retaining the full key-value buffer areas corresponding to the top semantic block retention block number of semantic blocks in the sorted result, setting the current tags corresponding to the remaining semantic blocks as the second tags to be compressed, and constructing a second sequence to be compressed based on each second tag to be compressed; recording the position information, importance score, and corresponding semantic block number corresponding to each second tag to be compressed, and compressing the initial inference path corresponding to the large language model using the target compression policy and the second sequence to be compressed to obtain a target inference path.

[0121] Step S15, generating code using the large language model and based on the target inference path.

[0122] In this embodiment, to adapt to the dynamic changes in states such as semantic redundancy, importance distribution, and model stability during the inference process, the embodiment of the present application adopts an adaptive compression strategy regulation mechanism to adjust the compression strategy. That is, in each round of inference, multiple indicators are monitored in real time, such as semantic entropy, average token importance, KV cache growth rate, model output confidence fluctuation, and compression rollback rate, to collect real-time operation state indicators related to compression, providing a quantitative basis for subsequent strategy adjustment.

[0123] In a specific implementation, when the semantic redundancy increases and the token importance is generally low, the embodiment of the present application needs to shorten the compression period and increase the compression ratio; if the instability of the model inference result is detected or the compressed tokens are frequently rolled back, the embodiment of the present application needs to reduce the compression intensity and expand the evaluation window.

[0124] Subsequently, the embodiment of the present application needs to control the triggering frequency and execution delay of the compression action to align it with the model state in real time, avoiding frequent perturbation of the model inference process. For this purpose, a compression suppression window mechanism is introduced to dynamically regulate the execution timing and rhythm of compression. Among them, the compression suppression window mechanism is used to set a compression cooling period after a compression operation is completed, during which new compression operations are prohibited or delayed from being triggered. After each compression, within no next-round compression operation will be triggered; assume that a compression operation is completed currently, and record the current token step as , then subsequent compressions will be suppressed until one of the following conditions is met: the current token step or a compression state emergency is triggered (such as the memory approaching the upper limit, or the output stability significantly decreasing). Among them, is the compression suppression window length.

[0125] It is worth mentioning that can be dynamically adjusted according to the current state of the model, and the adjustment formula is as follows:[[]]

[0126] ;

[0127] Among them: is the basic cooling length (such as 2048); is the decrease in the model output confidence after compression; is the rollback ratio of the compressed tokens; , are both adjustment coefficients, used to amplify the impact of unstable feedback on the cooling time.

[0128] Subsequently, when the model experiences unstable inference or context loss after compression operations, the embodiments of this application need to automatically extend the cooling period to avoid information loss caused by continuous compression. For example, after the normal compression operation in the embodiments of this application, if the running state of the model is stable, then set to 2048, that is, compression will not be triggered again within the next 2048 tokens; if it is found that the output quality of the model decreases, the entropy value increases, and the rollback ratio increases after compression, then will be extended to 4096 or higher; if the GPU video memory pressure is too high or the cache surges, the suppression window can be forcibly interrupted and compression operations will be immediately triggered.

[0129] It is worth mentioning that, without affecting the inference accuracy of the language model, in order to safely and efficiently compress redundant KV caches, the embodiments of this application introduce a weak context recovery mechanism to dynamically recover some of the compressed KV information when the inference quality decreases and ensure the coherence and stability of the inference chain. Among them, during the operation of semantic block division and importance scoring, the embodiments of this application will generate the following two types of token lists: RetainList: including tokens in important semantic blocks, the last tokens in the Selector window, and PruneList: including low-importance tokens in redundant regions. Subsequently, the embodiments of this application need to construct a new KV cache index through RetainList to form a sparsely compressed KV structure, and the expression is as follows:

[0130] ;

[0131] In this example, the steps for performing compression operations on the current KV cache include: first, constructing a retain index mask, then reconstructing a sparse KV tensor based on the mask, and then updating the Transformer cache pointer to ensure that the new KV structure is used in the decoding stage. Furthermore, record the original positions, attention scores, and the semantic block numbers of the compressed tokens.

[0132] Among them, a boolean mask list or index list is used to indicate which token KV entries should be retained and which should be cleared currently. Subsequently, the embodiments of this application can obtain the retained token set RetainList after semantic block screening, and the retained token set is the index of the tokens in the generated sequence , in order to construct a boolean vector with a length of the current token step using the retained token set, and the construction formula is as follows:

[0133] ;

[0134] Subsequently, after completing the construction operation of the token index, the embodiments of the present application need to extract corresponding entries from the original Key and Value tensors according to the index to reconstruct the compressed sparse KV tensor. In a specific implementation, the embodiments of the present application can compress the original , into new , . It is worth mentioning that the obtained and are significantly reduced in size and only contain the KV pairs of the retained tokens, so as to ensure that the sparse reconstruction of the KV cache does not disrupt the token order and provide a structurally compact and computationally efficient attention context for the subsequent decoding process. Among them, represents the sequence length.

[0135] Furthermore, after reconstructing the sparse KV tensor, the embodiments of the present application need to write the compressed , into the KV cache structure inside the Transformer model to replace the original complete cache. Among them, for a model with a multi-layer structure, this operation will update the KV cache pointer of each layer layer by layer to ensure that only the retained context information is referenced in the subsequent decoding stage. At the same time, if the model uses rotary position encoding or absolute position embedding, the corresponding position information needs to be updated synchronously to prevent attention misalignment and ensure the logical continuity of the compressed context.

[0136] In addition, considering that some tokens that may potentially contribute to subsequent inference may be accidentally deleted during the compression process, the above-mentioned weak context recovery mechanism can dynamically recover some of the compressed tokens. Among them, during the continuous generation process after compression, if any of the following conditions are met: the confidence of the model output tokens drops significantly and consecutive generated tokens are repeated or invalid, the embodiments of the present application can then initiate the recovery logic. Among them, the specific recovery logic is to selectively recover the KV pairs of some compressed tokens, and the priority is determined by the sorting of the original attention scores. In addition, the number of recoveries is limited by a threshold, such as recovering up to the first 64 tokens at most, to prevent over-recovery, and insert the recovered tokens into the current KV structure and reconstruct the corresponding position encoding mapping table. It is worth mentioning that the above process is equivalent to a lightweight rollback, thus recovering the performance degradation caused by the loss of context information without disturbing the main inference path.

[0137] In this embodiment, during the inference termination and output recording stage, the embodiment of the present application first needs to judge multiple conditions to determine whether the generation process should end, including cases such as generating a termination flag, reaching the maximum length limit, or the confidence of the model output has converged stably. Once any termination condition is met, the token generation operation is immediately aborted, and the complete inference result is output, including the generated text sequence, structured information (such as answers or code segments), and optional token confidence trajectories. At the same time, the embodiment of the present application needs to synchronously record the compression path information during this round of inference, such as key metrics like the number of compression execution rounds, the retention ratio per round, semantic entropy changes, the number of compressed tokens, the number of rollback triggers, and the final KV cache size. Among them, the above data can be output in a structured form, which can be used for interface return, debugging visualization, or feedback and update of the compression strategy learning module, so as to realize the traceability, interpretability, and optimizability of the inference process, thus ensuring that the model inference still has output integrity and behavior transparency under compression intervention, and further constructing a closed-loop and highly stable inference compression system.

[0138] Specifically, after generating code using a large language model and based on the target inference path, it may further include: detecting and collecting metrics during the process of generating the target inference path, and adjusting the compression interval, selection window size, and compression ratio in the compression control parameters using the collected metrics to obtain new compression control parameters; the metrics include Semantic entropy, average token importance, full key-value cache growth rate, model output confidence fluctuation, and compression rollback rate; record the current token step, and use a preset compression suppression window mechanism to determine a compression cooling period based on the current token step and the target compression suppression window length, so as to suppress the compression operation during the compression cooling period; wherein, the target compression suppression window length is a compression suppression window length determined based on a preset adjustment coefficient, a preset base cooling length, the decrease in model output confidence, and the compression rollback rate; the preset adjustment coefficient is used to amplify the influence of the collected unstable feedback information on the compression cooling period; construct an index list based on the target inference path, so as to process the target inference path using a preset weak context recovery mechanism based on the meta-information corresponding to the target inference path and the cache pointer corresponding to the model layer to obtain a new target inference path; the preset weak context recovery mechanism is used to restore a full key-value cache information of a preset quantity size in the full key-value cache area when it is monitored that the inference quality corresponding to the target inference path decreases; structurally record the meta-information corresponding to the target inference path into the compression log; the meta-information includes the position of the target inference path in the token sequence to be compressed, the importance score, the semantic block number, the token step, and a soft deletion flag indicating whether recovery is allowed; determine whether the target inference path meets a preset generation end condition, and if the target inference path meets the preset generation end condition, end the current process of generating the target inference path; the target inference path includes the number of compression execution rounds, the compression ratio, the change in semantic entropy, the total number of compressed tokens, the number of rollback triggers, and the full key-value cache size; the preset generation end condition includes terminating the generation of the flag token, the length of the target inference path reaching a preset length threshold, and the confidence of the large language model meeting a preset confidence convergence condition.

[0139] As can be seen from the above, in the embodiment of the present application, first, it is necessary to use a preset autoregressive method and generate each current token based on the initial inference path corresponding to the large language model, and cache the query vectors corresponding to each current token. Then, a preset number of the most recently generated query vectors are selected from each current token to construct a selection window based on the query vectors corresponding to each of the to-be-processed tokens. Subsequently, based on the selection window, the sliding window attention corresponding to the historical tokens and each model layer and each attention head in the large language model is obtained, and the corresponding historical tokens are processed based on the sliding window attention to obtain an importance scoring result and a paragraph generation result. Then, each paragraph generation result is evaluated to obtain a corresponding semantic sparsity result. Then, based on the importance scoring result and the semantic sparsity result, a semantic block division strategy is determined, and the to-be-compressed token sequence including each current token is divided into semantic blocks based on the semantic block division strategy. Then, the importance scores corresponding to each semantic block are sorted to obtain a sorting result. Finally, compression control parameters are determined based on the model characteristics and inference requirements of the large language model to generate a target compression strategy based on the compression control parameters, and the initial inference path corresponding to the large language model is compressed using the target compression strategy to obtain a target inference path. In this way, the efficiency of code generation is improved during the process of code generation based on the large language model, thereby reducing the memory for storing the inference path.

[0140] Correspondingly, referring to Figure 2 as shown, the present application also provides a code generation device based on a large language model, including:

[0141] A selection window determination module 11, configured to use a preset autoregressive method and generate each current token based on the initial inference path for code generation corresponding to the large language model, cache the query vectors corresponding to each current token, and then select a preset number of the most recently generated to-be-processed tokens from each current token to construct a selection window based on the query vectors corresponding to each of the to-be-processed tokens;

[0142] A semantic sparsity result determination module 12, configured to obtain the sliding window attention corresponding to the target model layer and the target attention head based on the selection window, and process the corresponding historical tokens based on the sliding window attention to obtain an importance scoring result and a paragraph generation result, and then evaluate each paragraph generation result to obtain a corresponding semantic sparsity result; the target model layer is the model layer corresponding to the historical tokens among the model layers of the large language model; the target attention head is the attention head corresponding to the historical tokens among the attention heads of the large language model;

[0143] A sorting result determination module 13 is configured to determine a semantic block division strategy based on the importance scoring result and the semantic sparsity result, divide a to-be-compressed token sequence including each of the current tokens into semantic blocks based on the semantic block division strategy, and then sort based on the importance scores corresponding to each of the semantic blocks to obtain a sorting result;

[0144] A compression result determination module 14 is configured to determine compression control parameters based on the model characteristics and inference requirements of the large language model, generate a target compression strategy based on the compression control parameters, and use the target compression strategy to compress the initial inference path corresponding to the large language model to obtain a target inference path; the compression control parameters include a compression interval, a selection window size, and a compression ratio;

[0145] A code generation module 15 is configured to generate code by using the large language model and based on the target inference path.

[0146] As can be seen from the above, before code generation based on the large language model in the embodiments of the present application, it is first necessary to generate each current token by using a preset autoregressive method and based on the initial inference path corresponding to the large language model, and cache the query vectors corresponding to each current token, and then select a preset number of recently generated query vectors from each current token to construct a selection window based on each query vector; subsequently, obtain the sliding window attention corresponding to the historical tokens and each model layer and each attention head in the large language model based on the selection window, and process the corresponding historical tokens based on the sliding window attention to obtain an importance scoring result and a paragraph generation result, and then evaluate each paragraph generation result to obtain a corresponding semantic sparsity result; then, determine a semantic block division strategy based on the importance scoring result and the semantic sparsity result, divide the to-be-compressed token sequence including each current token into semantic blocks based on the semantic block division strategy, and then sort based on the importance scores corresponding to each semantic block to obtain a sorting result; finally, determine compression control parameters based on the model characteristics and inference requirements of the large language model, generate a target compression strategy based on the compression control parameters, and use the target compression strategy to compress the initial inference path corresponding to the large language model to obtain a target inference path. In this way, the efficiency of code generation is improved during the process of code generation based on the large language model, and thus the memory for storing the inference path is reduced.

[0147] In some specific embodiments, the selection window determination module 11 may specifically include:

[0148] A token generation unit is configured to generate each current token by using a preset autoregressive method and based on the initial inference path corresponding to the large language model, and determine the query vector, key vector, and value vector corresponding to each of the current tokens;

[0149] A vector storage unit for storing each of the query vectors, the key vectors, and the value vectors into a query buffer and a full-scale key-value buffer; wherein, only the query vectors corresponding to a preset number of currently generated tokens are retained in the query buffer;

[0150] A selection window construction unit for extracting the preset number of query vectors from the query buffer and determining a window construction criterion based on each of the query vectors, so as to construct a selection window based on the window construction criterion and each of the current tokens.

[0151] In some specific embodiments, the semantic sparsity result determination module 12 may specifically include:

[0152] A sliding window attention determination unit for determining each model layer and each attention head in the large language model, and determining the number of model layers and the number of attention head sequences corresponding to historical tokens, and then determining the sliding window attention corresponding to the historical tokens based on the selection window, the query vectors, the key vectors, the value vectors, the number of model layers, and the attention head sequences; wherein, the numerical size of the sliding window attention is proportional to the contribution value size of the historical token in the selection window;

[0153] A to-be-smoothed importance score determination unit for counting the total number of model layers corresponding to each model layer and the total number of attention heads corresponding to each attention head, so as to determine a to-be-smoothed importance score based on the total number of model layers, the total number of attention heads, and the sliding window attention;

[0154] An importance score result determination unit for establishing a paragraph generation result based on each of the current tokens and smoothing the to-be-smoothed importance score based on the window width corresponding to the selection window to obtain an importance score result.

[0155] In some specific embodiments, the semantic sparsity result determination module 12 may specifically include:

[0156] An entropy value result determination unit for using a preset entropy value determination formula and generating an initial entropy value result corresponding to each of the preset tuple sets based on the paragraph generation result, a plurality of preset tuple sets, and the probability distribution result corresponding to the paragraph generation result, and determining a target entropy value result based on each of the initial entropy value results;

[0157] The first to-be-compressed mark determination unit is configured to determine whether the target entropy value result is greater than a preset paragraph length threshold. If the target entropy value result is not greater than the preset paragraph length threshold, then it is determined whether the importance score result corresponding to the paragraph generation result is greater than a preset importance score threshold. If the importance score result corresponding to the paragraph generation result is not greater than the preset importance score threshold, then the area corresponding to the paragraph generation result is set as a semantically sparse area, and the mark corresponding to the paragraph generation result is set as a to-be-compressed mark;

[0158] The second to-be-compressed mark determination unit is configured to, if the importance score result corresponding to the paragraph generation result is greater than the preset importance score threshold, set the area corresponding to the paragraph generation result as a semantically dense area, and prohibit setting the mark corresponding to the paragraph generation result as a to-be-compressed mark;

[0159] The third to-be-compressed mark determination unit is configured to, if the target entropy value result is greater than the preset paragraph length threshold, set the area corresponding to the paragraph generation result as a semantically dense area, and prohibit setting the mark corresponding to the paragraph generation result as a to-be-compressed mark;

[0160] The fourth to-be-compressed mark determination unit is configured to determine whether the number of selection windows corresponding to generating a preset number of current marks is greater than a preset window number threshold. If the number of selection windows corresponding to generating a preset number of current marks is greater than the preset window number threshold, then the area corresponding to the paragraph generation result is set as a semantically sparse area, and the mark corresponding to the paragraph generation result is set as a to-be-compressed mark.

[0161] In some specific embodiments, the sorting result determination module 13 may specifically include:

[0162] The first to-be-compressed sequence determination unit is configured to determine first to-be-compressed marks from each of the current marks based on the importance score result and the semantic sparsity result, and construct a first to-be-compressed sequence based on each of the first to-be-compressed marks, so as to extract semantic representation vectors corresponding to each of the first to-be-compressed marks from the first to-be-compressed sequence; the semantic representation vector is a context vector corresponding to the model layer and a weighted combination vector;

[0163] The semantic block determination unit is configured to divide each of the semantic representation vectors into several clusters that are semantically related by using a preset clustering algorithm, and perform label assignment on each of the first to-be-compressed marks based on each cluster, so as to determine semantic blocks based on the label assignment result; the arrangement order of the first to-be-compressed marks corresponding to each of the first to-be-compressed marks in the semantic block is the same as the arrangement order of each of the first to-be-compressed marks in the first to-be-compressed sequence;

[0164] A sorting result determination unit is configured to perform an overall importance scoring on each of the semantic blocks based on the importance scoring result, obtain corresponding importance scores, and sort each of the semantic blocks in ascending order of the importance scores to obtain a sorting result.

[0165] In some specific embodiments, the compression result determination module 14 may specifically include:

[0166] A compression strategy determination unit is configured to determine compression control parameters based on the model characteristics and inference requirements of the large language model, so as to generate a target compression strategy based on the compression control parameters; wherein, the model characteristics include each model layer and each attention head;

[0167] A second sequence to be compressed determination unit is configured to determine the number of semantic block retention blocks based on the compression interval and the semantic block length corresponding to the semantic block, so as to retain the full amount key-value buffer areas corresponding to the first several semantic block retention blocks of the semantic blocks in the sorting result, and set the current tokens corresponding to the remaining semantic blocks as second tokens to be compressed, and construct a second sequence to be compressed based on each of the second tokens to be compressed;

[0168] A target inference path determination unit is configured to record the position information, importance score and corresponding semantic block number corresponding to each of the second tokens to be compressed, and use the target compression strategy and the second sequence to be compressed to compress the initial inference path corresponding to the large language model to obtain a target inference path.

[0169] In some specific embodiments, the code generation device based on the large language model may further include:

[0170] A compression control parameter adjustment unit is configured to detect and collect metrics during the process of generating the target inference path, and use the collected metrics to adjust the compression interval, the selection window size and the compression ratio in the compression control parameters to obtain new compression control parameters; the metrics include n-gram semantic entropy, average token importance, full amount key-value buffer growth rate, model output confidence fluctuation and compression rollback rate;

[0171] A compression cooling period determination unit is configured to record the current token step number, and use a preset compression suppression window mechanism to determine a compression cooling period based on the current token step number and the target compression suppression window length, so as to suppress the compression operation during the compression cooling period; wherein, the target compression suppression window length is a compression suppression window length determined based on a preset adjustment coefficient, a preset base cooling length, the model output confidence decrease amount and the compression rollback rate; the preset adjustment coefficient is used to amplify the influence of the collected unstable feedback information on the compression cooling period;

[0172] An inference path processing unit, configured to build an index list based on the target inference path, so as to utilize a preset weak context recovery mechanism and process the target inference path based on the meta-information corresponding to the target inference path and the cache pointer corresponding to the model layer, and obtain a new target inference path; the preset weak context recovery mechanism is used to recover a preset amount of full key-value cache information in the full key-value cache area when it is detected that the inference quality corresponding to the target inference path decreases;

[0173] An information recording unit, configured to structurally record the meta-information corresponding to the target inference path into a compressed log; the meta-information includes the position of the target inference path in the to-be-compressed token sequence, the importance score, the semantic block number, the token step number, and a soft deletion flag indicating whether recovery is allowed;

[0174] An inference path judgment unit, configured to judge whether the target inference path meets a preset generation end condition. If the target inference path meets the preset generation end condition, the process of generating the current target inference path is ended; the target inference path includes the number of compression execution rounds, the compression ratio, the change in semantic entropy, the total number of compressed tokens, the number of rollback triggers, and the full key-value cache size; the preset generation end conditions include the generation of a termination flag, the length of the target inference path reaching a preset length threshold, and the confidence of the large language model meeting a preset confidence convergence condition.

[0175] Furthermore, an embodiment of the present application also discloses an electronic device, Figure 3 which is a structural diagram of an electronic device 20 shown according to an exemplary embodiment. The content in the figure cannot be regarded as any limitation on the scope of use of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. Among them, the memory 22 is used to store a computer program, and the computer program is loaded and executed by the processor 21 to implement the relevant steps in the code generation method based on a large language model disclosed in any of the foregoing embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0176] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows is any communication protocol applicable to the technical solution of the present application, and no specific limitation is imposed thereon here; the input / output interface 25 is used to obtain external input data or output data to the outside, and its specific interface type can be selected according to specific application requirements, and no specific limitation is made here.

[0177] In addition, as a carrier for storing resources, the memory 22 can be a read-only memory, a random access memory, a magnetic disk, an optical disc, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc. The storage method can be temporary storage or permanent storage.

[0178] Among them, the operating system 221 is used to manage and control each hardware device and the computer program 222 on the electronic device 20, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the code generation method based on the large language model executed by the electronic device 20 disclosed in any of the foregoing embodiments, the computer program 222 can further include computer programs that can be used to complete other specific tasks.

[0179] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the code generation method based on the large language model disclosed above is implemented. For the specific steps of this method, reference can be made to the corresponding content disclosed in the foregoing embodiments, and details will not be repeated here.

[0180] In this specification, the various embodiments are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts among the various embodiments can be referred to each other. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple. For the relevant parts, reference can be made to the description of the method part.

[0181] Those skilled in the art can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed in this article can be implemented by electronic hardware, computer software, or a combination of the two. To clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been generally described according to their functions in the above description. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of this application.

[0182] The steps of the methods or algorithms described in combination with the embodiments disclosed in this article can be directly implemented by hardware, software modules executed by a processor, or a combination of the two. The software modules can be placed in a random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium well-known in the technical field.

[0183] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0184] The technical solutions provided in this application have been introduced in detail above. Specific examples are used in this text to elaborate on the principles and implementation manners of this application. The descriptions of the above embodiments are only used to help understand the method and its core idea of this application. At the same time, for those of ordinary skill in the art, according to the idea of this application, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to this application.

Claims

1. A code generation method based on a large language model, characterized in that Including: Generating each current token using a preset autoregressive manner and based on an initial inference path for code generation corresponding to the large language model, and caching query vectors corresponding to each of the current tokens, then selecting a preset number of the most recently generated to-be-processed tokens from each of the current tokens to construct a selection window based on the query vectors respectively corresponding to each of the to-be-processed tokens; Obtaining sliding window attention respectively corresponding to a target model layer and a target attention head based on the selection window, and processing corresponding historical tokens based on the sliding window attention to obtain an importance scoring result and a paragraph generation result, then evaluating each of the paragraph generation results to obtain a corresponding semantic sparsity result; The target model layer is the model layer corresponding to the historical tokens among the model layers of the large language model; the target attention head is the attention head corresponding to the historical tokens among the attention heads of the large language model; Determining a semantic block division strategy based on the importance scoring result and the semantic sparsity result, dividing a to-be-compressed token sequence including each of the current tokens into semantic blocks based on the semantic block division strategy, and then sorting based on the importance scores corresponding to each of the semantic blocks to obtain a sorting result; Determining compression control parameters based on the model characteristics and inference requirements of the large language model to generate a target compression strategy based on the compression control parameters, and compressing the initial inference path corresponding to the large language model using the target compression strategy to obtain a target inference path; the compression control parameters include a compression interval, a selection window size, and a compression ratio; Generating code using the large language model and based on the target inference path.

2. The code generation method based on a large language model according to claim 1, wherein The step of generating each current token using a preset autoregressive manner and based on an initial inference path for code generation corresponding to the large language model, and caching query vectors corresponding to each of the current tokens, then selecting a preset number of the most recently generated to-be-processed tokens from each of the current tokens to construct a selection window based on the query vectors respectively corresponding to each of the to-be-processed tokens includes: Generating each current token using a preset autoregressive manner and based on the initial inference path corresponding to the large language model, and determining query vectors, key vectors, and value vectors corresponding to each of the current tokens; Storing each of the query vectors, the key vectors, and the value vectors into a query buffer and a full-scale key-value buffer; wherein, the query buffer only retains query vectors corresponding to a preset number of the most recently generated current tokens; Extracting the preset number of query vectors from the query buffer, and determining a window construction criterion based on each of the query vectors to construct a selection window based on the window construction criterion and each of the current tokens.

3. The code generation method based on a large language model according to claim 2, wherein The step of obtaining sliding window attention respectively corresponding to a target model layer and a target attention head based on the selection window, and processing corresponding historical tokens based on the sliding window attention to obtain an importance scoring result and a paragraph generation result includes: Identify each model layer and each attention head in the large language model, and determine the number of model layers and the number of attention head sequences corresponding to the historical tokens. Then, based on the selection window, the query vector, the key vector, the value vector, the number of model layers, and the number of attention head sequences, determine the sliding window attention corresponding to the historical tokens; wherein, the numerical value of the sliding window attention is proportional to the contribution value of the historical tokens in the selection window. Count the total number of model layers corresponding to each model layer and the total number of attention heads corresponding to each attention head, and determine the importance score to be smoothed based on the total number of model layers, the total number of attention heads, and the sliding window attention. Establish a paragraph generation result based on each current token, and smooth the importance score to be smoothed based on the window width corresponding to the selection window to obtain an importance score result.

4. The code generation method based on a large language model according to claim 1, characterized in that Evaluate each of the paragraph generation results to obtain corresponding semantic sparsity results, including: Use a preset entropy value determination formula and generate an initial entropy value result corresponding to each preset tuple set based on the paragraph generation result, several preset tuple sets, and the probability distribution result corresponding to the paragraph generation result, and determine a target entropy value result based on each initial entropy value result. Judge whether the target entropy value result is greater than a preset paragraph length threshold. If the target entropy value result is not greater than the preset paragraph length threshold, then judge whether the importance score result corresponding to the paragraph generation result is greater than a preset importance score threshold. If the importance score result corresponding to the paragraph generation result is not greater than the preset importance score threshold, then set the area corresponding to the paragraph generation result as a semantic sparse area, and set the token corresponding to the paragraph generation result as a token to be compressed. If the importance score result corresponding to the paragraph generation result is greater than the preset importance score threshold, then set the area corresponding to the paragraph generation result as a semantic dense area, and prohibit setting the token corresponding to the paragraph generation result as a token to be compressed. If the target entropy value result is greater than the preset paragraph length threshold, then set the area corresponding to the paragraph generation result as a semantic dense area, and prohibit setting the token corresponding to the paragraph generation result as a token to be compressed. Alternatively, judge whether the number of selection windows corresponding to generating a preset number of current tokens is greater than a preset window number threshold. If the number of selection windows corresponding to generating a preset number of current tokens is greater than the preset window number threshold, then set the area corresponding to the paragraph generation result as a semantic sparse area, and set the token corresponding to the paragraph generation result as a token to be compressed.

5. The code generation method based on a large language model according to claim 1, wherein Determine a semantic block division strategy based on the importance score result and the semantic sparsity result, divide the sequence of tokens to be compressed including each current token into semantic blocks based on the semantic block division strategy, and then sort the importance scores corresponding to each semantic block to obtain a sorting result, including: Determine the first tokens to be compressed from each of the current tokens based on the importance scoring result and the semantic sparsity result, and construct a first sequence to be compressed based on each of the first tokens to be compressed, so as to extract the semantic representation vectors corresponding to each of the first tokens to be compressed from the first sequence to be compressed; the semantic representation vectors are context vectors and weighted combination vectors corresponding to the model layer; Use a preset clustering algorithm to divide each of the semantic representation vectors into several clusters with semantic relevance, and perform label assignment on each of the first tokens to be compressed based on each cluster, so as to determine semantic blocks based on the label assignment result; the arrangement order of each of the first tokens to be compressed in the semantic block is consistent with the arrangement order of each of the first tokens to be compressed in the first sequence to be compressed; Perform an overall importance scoring on each of the semantic blocks based on the importance scoring result to obtain corresponding importance scores, and sort each of the semantic blocks in ascending order of each of the importance scores to obtain a sorting result.

6. The code generation method based on a large language model according to claim 1, wherein Determine compression control parameters based on the model characteristics and inference requirements of the large language model, generate a target compression strategy based on the compression control parameters, and use the target compression strategy to compress the initial inference path corresponding to the large language model to obtain a target inference path, including: Determine compression control parameters based on the model characteristics and inference requirements of the large language model, and generate a target compression strategy based on the compression control parameters; wherein, the model characteristics include each model layer and each attention head; Determine the number of semantic block retention blocks based on the compression interval and the semantic block length corresponding to the semantic block, so as to retain the full amount key-value buffer areas corresponding to the first semantic block retention block number of semantic blocks in the sorting result, and set the current tokens corresponding to the remaining semantic blocks to be the second tokens to be compressed, and construct a second sequence to be compressed based on each of the second tokens to be compressed; Record the position information, importance score and corresponding semantic block number corresponding to each of the second tokens to be compressed, and use the target compression strategy and the second sequence to be compressed to compress the initial inference path corresponding to the large language model to obtain a target inference path.

7. The code generation method based on a large language model according to any one of claims 1 to 6, characterized in that, After generating the code using the large language model and based on the target inference path, further include: Detect and collect metrics for the process of generating the target inference path, and use the collected metrics to adjust the compression interval, the selection window size and the compression ratio in the compression control parameters to obtain new compression control parameters; the metrics include n-gram semantic entropy, average token importance, full amount key-value buffer growth rate, model output confidence fluctuation and compression rollback rate; Record the current marked step count, and use a preset compression suppression window mechanism to determine a compression cooling period based on the current marked step count and the target compression suppression window length, so as to suppress the compression operation during the compression cooling period; wherein, the target compression suppression window length is a compression suppression window length determined based on a preset adjustment coefficient, a preset basic cooling length, the decrease in the confidence of the model output, and the compression rollback rate; the preset adjustment coefficient is used to amplify the impact of the collected unstable feedback information on the compression cooling period; Construct an index list based on the target inference path, so as to use a preset weak context recovery mechanism and process the target inference path based on the meta information corresponding to the target inference path and the cache pointer corresponding to the model layer to obtain a new target inference path; the preset weak context recovery mechanism is used to restore a preset amount of full key-value cache information in the full key-value cache area when it is monitored that the inference quality corresponding to the target inference path decreases; Structurally record the meta information corresponding to the target inference path in the compression log; the meta information includes the position of the target inference path in the to-be-compressed token sequence, the importance score, the semantic block number, the marked step count, and a soft deletion flag indicating whether recovery is allowed; Judge whether the target inference path meets the preset generation end condition. If the target inference path meets the preset generation end condition, end the current process of generating the target inference path; the target inference path includes the number of compression execution rounds, the compression ratio, the change in semantic entropy, the total number of compressed tokens, the number of rollback trigger times, and the full key-value cache size; the preset generation end condition includes the termination of the flag token generation, the length of the target inference path reaching a preset length threshold, and the confidence of the large language model meeting the preset confidence convergence condition.

8. A code generation device based on a large language model, characterized in that, Include: A selection window determination module, configured to generate each current token using a preset autoregressive method based on the initial inference path for code generation corresponding to the large language model, cache the query vector corresponding to each current token, and then select a preset number of the most recently generated to-be-processed tokens from each current token, so as to construct a selection window based on the query vectors respectively corresponding to the to-be-processed tokens; A semantic sparsity result determination module, configured to obtain the sliding window attention corresponding to the target model layer and the target attention head based on the selection window, process the corresponding historical tokens based on the sliding window attention to obtain an importance scoring result and a paragraph generation result, and then evaluate each paragraph generation result to obtain the corresponding semantic sparsity result; The target model layer is the model layer corresponding to the historical tokens among the model layers of the large language model; the target attention head is the attention head corresponding to the historical tokens among the attention heads of the large language model; A sorting result determination module, configured to determine a semantic block partitioning strategy based on the importance scoring result and the semantic sparsity result, partition a to-be-compressed token sequence including each of the current tokens into semantic blocks based on the semantic block partitioning strategy, and then sort based on the importance scores corresponding to each of the semantic blocks to obtain a sorting result; A compression result determination module, configured to determine compression control parameters based on the model characteristics and inference requirements of the large language model, generate a target compression strategy based on the compression control parameters, and use the target compression strategy to compress the initial inference path corresponding to the large language model to obtain a target inference path; the compression control parameters include a compression interval, a selection window size, and a compression ratio; A code generation module, configured to generate code using the large language model and based on the target inference path.

9. An electronic device, characterized in that, Comprising: A memory, configured to store a computer program; A processor, configured to execute the computer program to implement the large language model-based code generation method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, For storing a computer program, wherein the computer program, when executed by a processor, implements the large language model-based code generation method according to any one of claims 1 to 7.

Images