A code generation method, device, equipment and storage medium based on large language model

Abstract

The present application discloses a code generation method, apparatus, device and storage medium based on a large language model, which relates to the field of large model technology, including: generating a current tag based on an initial reasoning path for code generation of the model, caching a query vector of the current tag, and constructing a selection window based on the query vectors of several tags to be processed; using a sliding window attention process of a target model layer and a target attention head obtained based on the selection window to obtain an importance score result and a paragraph generation result; using a semantic block partitioning strategy determined based on the importance score result and the semantic sparsity result of each paragraph generation result to divide the tag sequence to be compressed into semantic blocks; generating a target compression strategy based on the importance score of each semantic block and using compression control parameters determined based on model characteristics and reasoning requirements; and compressing the initial reasoning path using the target compression strategy to obtain a target reasoning path to generate code. This can improve the efficiency of code generation.

Description

Technical Field

[0001] The present invention relates to the field of large model technology, and in particular to a code generation method, apparatus, device and storage medium based on a large language model. Background Art

[0002] With the widespread application of large language models (LLMs) in complex tasks such as logical reasoning, mathematical computation, and code generation, explicit reasoning strategies such as chaining are becoming increasingly popular. These strategies improve model interpretability and accuracy by generating detailed intermediate reasoning paths, but they also incur significant computational overhead. In particular, they can lead to rapid expansion of the key-value (KV) cache during the inference phase, severely consuming graphics memory resources and slowing down inference speed.

[0003] Current KV compression techniques primarily focus on the input prompt stage. When a model receives long text or contextual information, they reduce KV entries through methods like cache truncation and summary extraction. However, during the model inference generation phase, tags are generated incrementally, and their importance is unpredictable at generation time, making existing compression methods difficult to directly apply. Accidentally deleting KV information for key tags can severely undermine the model's understanding of context, impacting the accuracy and consistency of the final output.

[0004] Further research revealed that semantic sparsity is common during the generation of reasoning LLMs: the reasoning path often contains numerous fragments with repeated content, semantic redundancy, or logical inaction. Although these tags occupy a large portion of the KV cache, their contribution to the final reasoning results is limited. Previous compression methods have failed to fully utilize this characteristic, failing to effectively identify and remove invalid tags during the reasoning phase, resulting in wasted storage and performance bottlenecks.

[0005] As can be seen from the above, how to improve the efficiency of code generation in the code generation process based on large language models is an urgent problem that needs to be solved. Summary of the Invention

[0006] In view of this, the present invention aims to provide a code generation method, apparatus, device, and storage medium based on a large language model, which can improve the efficiency of code generation during the code generation process based on a large language model and thereby reduce the memory required to store the inference path. The specific solution is as follows:

[0007] In a first aspect, the present application provides a code generation method based on a large language model, comprising:

[0008] Generate each current token using a preset autoregressive method based on an initial inference path for code generation corresponding to the large language model, cache a query vector corresponding to each current token, and then select a preset number of most recently generated tokens to be processed from each current token to construct a selection window based on the query vectors corresponding to each token to be processed;

[0009] Based on the selection window, the sliding window attention corresponding to the target model layer and the target attention head is obtained, and the corresponding historical tags are processed based on the sliding window attention to obtain an importance score result and a paragraph generation result, and then each of the paragraph generation results is evaluated to obtain a corresponding semantic sparsity result; the target model layer is the model layer corresponding to the historical tags in each model layer of the large language model; the target attention head is the attention head corresponding to the historical tags in each attention head of the large language model;

[0010] Determining a semantic block division strategy based on the importance score result and the semantic sparsity result, dividing the to-be-compressed tag sequence including each current tag into semantic blocks based on the semantic block division strategy, and then sorting the semantic blocks based on the importance scores corresponding to the semantic blocks to obtain a sorting result;

[0011] Determining compression control parameters based on the model characteristics and inference requirements of the large language model, generating a target compression strategy based on the compression control parameters, and compressing an initial inference path corresponding to the large language model using the target compression strategy to obtain a target inference path; the compression control parameters include a compression interval, a selection window size, and a compression ratio;

[0012] Code is generated based on the target reasoning path using the large language model.

[0013] Optionally, the method of generating each current token using a preset autoregressive method based on an initial inference path for code generation corresponding to the large language model, caching a query vector corresponding to each current token, and then selecting a preset number of most recently generated tokens to be processed from each current token to construct a selection window based on the query vectors corresponding to each token to be processed, includes:

[0014] Generate each current token using a preset autoregressive method and based on an initial inference path corresponding to the large language model, and determine a query vector, a key vector, and a value vector corresponding to each current token;

[0015] Storing each of the query vectors, the key vector, and the value vector in a query buffer and a full key-value cache; wherein the query buffer only retains the query vectors corresponding to the most recently generated preset number of current tags;

[0016] The preset number of query vectors are extracted from the query buffer, and a window construction criterion is determined based on each of the query vectors, so as to construct a selection window based on the window construction criterion and each of the current marks.

[0017] Optionally, acquiring sliding window attention corresponding to the target model layer and the target attention head respectively based on the selection window, and processing corresponding historical tags based on the sliding window attention to obtain importance scoring results and paragraph generation results, including:

[0018] Determine each model layer and each attention head in the large language model, and determine the number of model layers and the number of attention head sequences corresponding to the historical marker, and then determine the sliding window attention corresponding to the historical marker based on the selection window, the query vector, the key vector, the value vector, the number of model layers, and the number of attention head sequences; wherein the numerical value corresponding to the sliding window attention is proportional to the contribution value of the historical marker in the selection window;

[0019] Counting the total number of model layers corresponding to each of the model layers and the total number of attention heads corresponding to each of the attention heads, so as to determine an importance score to be smoothed based on the total number of model layers, the total number of attention heads, and the sliding window attention;

[0020] A paragraph generation result is established based on each of the current marks, and the importance score to be smoothed is smoothed based on a window width corresponding to the selection window to obtain an importance score result.

[0021] Optionally, evaluating each paragraph generation result to obtain a corresponding semantic sparsity result includes:

[0022] generating an initial entropy value result corresponding to each of the preset tuple sets using a preset entropy value determination formula and based on the paragraph generation result and probability distribution results corresponding to a plurality of preset tuple sets and the paragraph generation result, and determining a target entropy value result based on each of the initial entropy value results;

[0023] Determine whether the target entropy value result is greater than a preset paragraph length threshold; if the target entropy value result is not greater than the preset paragraph length threshold, determine whether the importance score result corresponding to the paragraph generation result is greater than a preset importance score threshold; if the importance score result corresponding to the paragraph generation result is not greater than the preset importance score threshold, set the area corresponding to the paragraph generation result as a semantically sparse area, and set the mark corresponding to the paragraph generation result as a mark to be compressed;

[0024] If the importance score result corresponding to the paragraph generation result is greater than the preset importance score threshold, the area corresponding to the paragraph generation result is set as a semantically dense area, and the mark corresponding to the paragraph generation result is prohibited from being set as a mark to be compressed;

[0025] If the target entropy value result is greater than the preset paragraph length threshold, the area corresponding to the paragraph generation result is set as a semantically dense area, and the mark corresponding to the paragraph generation result is prohibited from being set as a mark to be compressed;

[0026] Or, determine whether the number of selection windows corresponding to the number of current tags generated by generating a preset number of tags is greater than the preset window number threshold; if the number of selection windows corresponding to the number of current tags generated by generating a preset number of tags is greater than the preset window number threshold, then set the area corresponding to the paragraph generation result as a semantically sparse area, and set the tag corresponding to the paragraph generation result as a tag to be compressed.

[0027] Optionally, determining a semantic block division strategy based on the importance score result and the semantic sparsity result, dividing the to-be-compressed tag sequence including each current tag into semantic blocks based on the semantic block division strategy, and then sorting the semantic blocks based on the importance scores corresponding to the semantic blocks to obtain a sorting result, including:

[0028] Determining a first to-be-compressed token from each of the current tokens based on the importance score result and the semantic sparsity result, and constructing a first to-be-compressed sequence based on each of the first to-be-compressed tokens, so as to extract a semantic representation vector corresponding to each of the first to-be-compressed tokens from the first to-be-compressed sequence; the semantic representation vector being a context vector corresponding to the model layer and a weighted combination vector;

[0029] Using a preset clustering algorithm, each of the semantic representation vectors is divided into a plurality of semantically related clusters, and labels are assigned to each of the first to-be-compressed tokens based on each cluster, so as to determine a semantic block based on the label assignment results; the arrangement order of each of the first to-be-compressed tokens in the semantic block is consistent with the arrangement order of each of the first to-be-compressed tokens in the first to-be-compressed sequence;

[0030] Based on the importance scoring result, the overall importance of each semantic block is scored to obtain a corresponding importance score, and the semantic blocks are sorted in order from low to high according to the importance score to obtain a sorting result.

[0031] Optionally, determining compression control parameters based on the model characteristics and inference requirements of the large language model, generating a target compression strategy based on the compression control parameters, and compressing the initial inference path corresponding to the large language model using the target compression strategy to obtain a target inference path includes:

[0032] Determining compression control parameters based on model characteristics and inference requirements of the large language model, and generating a target compression strategy based on the compression control parameters; wherein the model characteristics include each model layer and each attention head;

[0033] Determining the number of reserved semantic blocks based on the compression interval and the length of the semantic block corresponding to the semantic block, retaining the full key value cache corresponding to the semantic blocks of the first number of reserved semantic blocks in the sorting result, setting the current marks corresponding to the remaining semantic blocks as second to-be-compressed marks, and constructing a second to-be-compressed sequence based on each of the second to-be-compressed marks;

[0034] Record the position information, importance score, and corresponding semantic block number corresponding to each second mark to be compressed, and use the target compression strategy and the second sequence to be compressed to compress the initial reasoning path corresponding to the large language model to obtain a target reasoning path.

[0035] Optionally, after generating code using the large language model and based on the target reasoning path, the method further includes:

[0036] The process of generating the target inference path is detected and indicators are collected, and the compression interval, the selection window size, and the compression ratio in the compression control parameters are adjusted using the collected indicators to obtain new compression control parameters; the indicators include n-gram semantic entropy, average tag importance, full key-value cache growth rate, model output confidence fluctuation, and compression rollback rate;

[0037] Recording the current number of marked steps, and determining a compression cooldown period based on the current number of marked steps and a target compression cooldown period using a preset compression suppression window mechanism, so as to suppress compression operations during the compression cooldown period; wherein the target compression cooldown period is a compression cooldown period determined based on a preset adjustment coefficient, a preset basic cooling period, a decrease in model output confidence, and the compression rollback rate; the preset adjustment coefficient is used to amplify the effect of the collected unstable feedback information on the compression cooldown period;

[0038] An index list is constructed based on the target reasoning path to obtain a new target reasoning path by processing the target reasoning path using a preset weak context recovery mechanism based on the metadata corresponding to the target reasoning path and the cache pointer corresponding to the model layer; the preset weak context recovery mechanism is used to restore a preset amount of full key-value cache information in the full key-value cache area when it is detected that the reasoning quality corresponding to the target reasoning path has deteriorated;

[0039] The meta-information corresponding to the target reasoning path is recorded in a structured manner in the compression log; the meta-information includes the position of the target reasoning path in the to-be-compressed mark sequence, the importance score, the semantic block number, the number of marking steps, and the soft deletion flag indicating whether recovery is allowed;

[0040] Determine whether the target reasoning path meets the preset generation end conditions. If the target reasoning path meets the preset generation end conditions, end the current process of generating the target reasoning path; the target reasoning path includes the number of compression execution rounds, compression ratio, semantic entropy change, total number of compressed tags, number of rollback triggers and full key-value cache size; the preset generation end conditions include termination flag generation, the length corresponding to the target reasoning path reaches a preset length threshold, and the confidence corresponding to the large language model meets the preset confidence convergence conditions.

[0041] In a second aspect, the present application provides a code generation device based on a large language model, comprising:

[0042] a selection window determination module, configured to generate each current token using a preset autoregressive method based on an initial inference path for code generation corresponding to the large language model, cache query vectors corresponding to each current token, and then select a preset number of most recently generated tokens to be processed from each current token to construct a selection window based on the query vectors corresponding to each token to be processed;

[0043] A semantic sparsity result determination module is configured to obtain, based on the selection window, the sliding window attention corresponding to the target model layer and the target attention head, respectively, and process the corresponding historical tags based on the sliding window attention to obtain an importance score result and a paragraph generation result, and then evaluate each of the paragraph generation results to obtain a corresponding semantic sparsity result; the target model layer is the model layer corresponding to the historical tags among the model layers of the large language model; the target attention head is the attention head corresponding to the historical tags among the attention heads of the large language model;

[0044] a sorting result determination module, configured to determine a semantic block division strategy based on the importance score result and the semantic sparsity result, and to divide the sequence of to-be-compressed tags including each current tag into semantic blocks based on the semantic block division strategy, and then sort the semantic blocks based on the importance scores corresponding to the semantic blocks to obtain a sorting result;

[0045] a compression result determination module, configured to determine compression control parameters based on the model characteristics and inference requirements of the large language model, generate a target compression strategy based on the compression control parameters, and compress the initial inference path corresponding to the large language model using the target compression strategy to obtain a target inference path; the compression control parameters include a compression interval, a selection window size, and a compression ratio;

[0046] A code generation module is configured to generate code based on the target reasoning path using the large language model.

[0047] In a third aspect, the present application provides an electronic device, comprising:

[0048] Memory, used to store computer programs;

[0049] A processor is used to execute the computer program to implement the aforementioned code generation method based on the large language model.

[0050] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned code generation method based on a large language model.

[0051] As can be seen from the above, before performing code generation based on the large language model, the present application needs to use a preset autoregressive method and generate each current tag based on the initial reasoning path corresponding to the large language model, and cache the query vector corresponding to each current tag, and then select a preset number of recently generated query vectors from each current tag to build a selection window based on each query vector; based on the selection window, obtain the sliding window attention corresponding to the historical tags and each model layer and each attention head in the large language model, and process the corresponding historical tags based on the sliding window attention to obtain the importance score result and paragraph generation result, and then generate the result for each paragraph. An evaluation is performed to obtain a corresponding semantic sparsity result; a semantic block division strategy is determined based on the importance score result and the semantic sparsity result, and a sequence of tags to be compressed including each current tag is divided into semantic blocks based on the semantic block division strategy, and then sorted based on the importance score corresponding to each semantic block to obtain a sorting result; compression control parameters are determined based on the model characteristics and inference requirements of the large language model, so as to generate a target compression strategy based on the compression control parameters, and the initial inference path corresponding to the large language model is compressed using the target compression strategy to obtain a target inference path; the compression control parameters include compression interval, selection window size and compression ratio.

[0052] It can be seen that the present application first needs to use a preset autoregressive method and generate each current tag based on the initial reasoning path corresponding to the large language model, and cache the query vector corresponding to each current tag, and then select a preset number of recently generated query vectors from each current tag to construct a selection window based on each query vector; then, based on the selection window, obtain the sliding window attention corresponding to the historical tags and each model layer and each attention head in the large language model, and process the corresponding historical tags based on the sliding window attention to obtain the importance score result and the paragraph generation result, and then evaluate each paragraph generation result to obtain the corresponding semantic sparsity result; then, determine the semantic block division strategy based on the importance score result and the semantic sparsity result, and divide the sequence of tags to be compressed including each current tag into semantic blocks based on the semantic block division strategy, and then sort them based on the importance score corresponding to each semantic block to obtain the sorting result; finally, determine the compression control parameters based on the model characteristics and reasoning requirements of the large language model to generate a target compression strategy based on the compression control parameters, and use the target compression strategy to compress the initial reasoning path corresponding to the large language model to obtain the target reasoning path. In this way, the efficiency of code generation is improved during code generation based on large language models, thereby reducing the memory required to store the inference path. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0054] Figure 1 This is a flow chart of a code generation method based on a large language model disclosed in this application;

[0055] Figure 2 This is a schematic diagram of the structure of a code generation device based on a large language model disclosed in this application;

[0056] Figure 3 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0057] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0058] At present, with the widespread adoption of explicit reasoning strategies such as chain thinking for large language models. This type of strategy improves the interpretability and accuracy of the model by generating detailed intermediate reasoning paths, but it also brings significant computational overhead. Current KV compression technologies are mostly concentrated in the input prompt stage, that is, when the model receives long text or contextual information, KV entries are reduced by cache truncation, summary extraction, etc. However, in the model reasoning generation stage, tags are generated step by step, and their importance is not predictable at the time of generation, which makes it difficult to directly apply existing compression methods. Once the KV information of key tags is mistakenly deleted, the contextual understanding of the model may be seriously damaged, affecting the accuracy and consistency of the final output. Further research has found that semantic sparsity is common in the generation process of inference-type large language models, which fails to fully utilize this feature. In the reasoning stage, invalid tags cannot be effectively identified and eliminated, resulting in storage waste and performance bottlenecks. To this end, the present application provides a code generation method based on a large language model, which can improve the efficiency of code generation in the process of code generation based on a large language model, thereby reducing the memory used to store reasoning paths.

[0059] See also Figure 1 As shown, an embodiment of the present invention discloses a code generation method based on a large language model, comprising:

[0060] Step S11: Generate each current tag using a preset autoregressive method and based on the initial inference path for code generation corresponding to the large language model, cache the query vector corresponding to each current tag, and then select a preset number of most recently generated to-be-processed tags from each current tag to construct a selection window based on the query vector corresponding to each to-be-processed tag.

[0061] In this embodiment, during the process of code generation based on a large language model, the embodiment of the present application chooses to compress the initial reasoning path for code generation corresponding to the large language model, so as to effectively utilize the semantic sparsity in the reasoning path, reduce memory usage and improve tag generation throughput while ensuring high reasoning accuracy, thereby realizing efficient reasoning of the inference-type LLM, and the process is: parameter configuration initialization, reasoning tag generation and multi-layer attention perception, semantic block construction and compression candidate identification, adaptive compression strategy scheduling, KV cache compression and weak context recovery mechanism and reasoning termination judgment and output.

[0062] It is worth mentioning that parameter configuration initialization is the preparation stage of the entire inference path compression system, which is used to provide the necessary control parameters, state cache structure and module functional units for the subsequent compression process. Among them, the initialization operation is directly related to the efficiency, stability and inference performance of compression. Therefore, before performing inference path compression, the embodiment of this application needs to set key parameters according to actual needs and model characteristics, and the parameters are as follows: compression interval : The compression interval corresponding to each compression operation triggered; select the window size : The number of recent tags used to evaluate importance; compression ratio : KV cache ratio to be retained; average pooling window size : Used to control local semantic continuity smoothing; Transformer layer number and the number of attention heads are all intrinsic parameters of the model.

[0063] Subsequently, before the compression mechanism is started, the embodiment of the present application needs to collect the degree to which each generated token is followed by subsequent tokens in the Transformer layer, and combine it with semantic sparsity analysis to provide a context contribution measurement basis for subsequent compression operations, namely, an importance score. The process of determining the importance score is as follows: In the decoding stage of the large language model, the embodiment of the present application needs to generate a new token through autoregression, and the expression for generating the current token is as follows:

[0064] ;

[0065] in, For large language models The new tag generated by step, 、 and All steps 1 to 2 Output tag of the step.

[0066] In this embodiment, after generating the mark, the embodiment of the present application needs to cache the Query, Key, and Value vectors corresponding to the selected window size mark into the Query buffer and the full KV cache respectively, and the expression is as follows:

[0067] ;

[0068] Further, when selecting the most recently generated After the query vector of the tag is used as the attention source, the attention source is selected to evaluate the contribution of the previous tag. Let the step corresponding to the current generated tag be Step, the embodiment of this application needs to be based on the most recently generated A marker is used to construct a selection window, and the expression is as follows:

[0069] ;

[0070] in, The vector set selected for the current step (i.e., the selection window), which contains the vectors of the most recent markers. From the Step to The generated Query vector, is the size of the sliding window, which is used to control the coverage of the source of interest.

[0071] Then, the window attention of the historical mark corresponding to the mark in different Transformer layers and different heads is obtained. In a specific embodiment, for each historical mark , each mark corresponds to the window , and the Tier The formula for determining the attention corresponding to each head is as follows:

[0072] ;

[0073] in, for In the Tier The query vector of the head, Historical Marker In the Tier The key vector of each head, The dimension corresponding to the Key vector and the Query vector.

[0074] It is worth mentioning that the embodiment of this application needs to traverse each layer in the large language model With each attention head , and for each marker Calculate its corresponding attention , to utilize Reflects history Contribution to the current decision window.

[0075] Specifically, each current tag is generated using a preset autoregressive method and based on an initial reasoning path for code generation corresponding to a large language model, and a query vector corresponding to each current tag is cached. Then, a preset number of recently generated tags to be processed are selected from each current tag to construct a selection window based on the query vectors corresponding to each tag to be processed. This may include: generating each current tag using a preset autoregressive method and based on an initial reasoning path corresponding to a large language model, and determining a query vector, a key vector, and a value vector corresponding to each current tag; storing each query vector, key vector, and value vector in a query buffer and a full key-value cache; wherein the query buffer only retains query vectors corresponding to a preset number of recently generated current tags; extracting a preset number of query vectors from the query buffer, and determining a window construction standard based on each query vector, so as to construct a selection window based on the window construction standard and each current tag.

[0076] Step S12: Based on the selection window, the sliding window attention corresponding to the target model layer and the target attention head is obtained respectively, and the corresponding historical tags are processed based on the sliding window attention to obtain the importance score result and the paragraph generation result, and then each of the paragraph generation results is evaluated to obtain the corresponding semantic sparsity result; the target model layer is the model layer corresponding to the historical tag in each model layer of the large language model; the target attention head is the attention head corresponding to the historical tag in each attention head of the large language model.

[0077] In this embodiment, after obtaining the sliding window attention, the embodiment of the present application needs to aggregate the sliding window attention of each tag in different layers or different heads to form a unified importance score , and the expression is as follows:

[0078] ;

[0079] in, is the total number of network layers, is the total number of attention heads.

[0080] Furthermore, to prevent important markers from being overwhelmed by neighboring low-scoring markers, the embodiment of the present application chooses to use local window smoothing:

[0081] ;

[0082] in, is the smoothed attention strength, which is used to reduce the masking effect of local fluctuations on important marks. is the window width, which is used to improve the consistency of the tag scores within the semantic segment to reduce the misjudgment caused by high attention to a single point and ensure a more reasonable compression granularity. If they all belong to the same logical block, but a certain mark in the middle is strongly concerned, the entire logical block will get a higher smoothing score, thereby preventing the above logical block from being compressed and discarded.

[0083] In this embodiment, it is necessary to determine whether the currently generated paragraph is redundant, that is, to construct a marking sliding window and update it in real time. A counting hash table that is updated each time a marker is generated value and set the threshold , to trigger the redundant segment determination. Among them, based on the tag sequence corresponding to the current window is the triple entropy and is expressed as follows:

[0084] ;

[0085] in, is the set of all possible triples in the current window, For triples The probability of appearing in the window.

[0086] Then, if , it is determined to be a semantically sparse region, that is, the higher the repetition rate of the tag sequence, The smaller it is, the area corresponding to the current generated paragraph is a semantically sparse area, where the expression of the tag sequence is as follows:

[0087] ;

[0088] Furthermore, if and Are lower than , it is determined to be a compressible region; if Higher than or Higher than , then the compression action is delayed or abandoned. In the actual large model reasoning process, repeated patterns of different lengths may appear at different granularities: short repetitions (such as ) may reflect a local grammar template; medium repetitions (e.g. ) often corresponds to sentence structure; long repetition (such as and above) may indicate that the entire logic is redundant. Also, only relying on a fixed value, such as A value of 3 may miss some meaningful semantically sparse segments or cause misjudgment.

[0089] Specifically, based on the selection window, the sliding window attention corresponding to the target model layer and the target attention head is obtained respectively, and the corresponding historical mark is processed based on the sliding window attention to obtain the importance score result and the paragraph generation result, which may include: determining each model layer and each attention head in the large language model, and determining the number of model layers and the number of attention head sequences corresponding to the historical mark, and then determining the sliding window attention corresponding to the historical mark based on the selection window, query vector, key vector, value vector, the number of model layers and the number of attention head sequences; wherein the numerical value corresponding to the sliding window attention is proportional to the contribution value of the historical mark in the selection window; counting the total number of model layers corresponding to each model layer and the total number of attention heads corresponding to each attention head, so as to determine the importance score to be smoothed based on the total number of model layers, the total number of attention heads and the sliding window attention; establishing a paragraph generation result based on each current mark, and smoothing the importance score to be smoothed based on the window width corresponding to the selection window to obtain the importance score result.

[0090] Furthermore, in order to improve the robustness and sensitivity of detection, the present embodiment introduces a multi-scale Entropy analysis mechanism, that is, calculating different The entropy corresponding to the value 、 and , and use this to construct a more comprehensive sparsity score, and after obtaining the above entropy, fuse the entropy, and the expression is as follows:

[0091] ;

[0092] in, The value range is , and in a specific embodiment, is 0.2, is 0.5, is 0.3.

[0093] Subsequently, after obtaining the importance score and semantic sparsity detection results, the embodiment of the present application needs to determine whether to compress the tags, which tags to compress, and which tags to retain, and introduce a semantic block aggregation mechanism to compress the tags at a group granularity rather than a single tag granularity, so as to maintain the integrity of the semantic structure.

[0094] Subsequently, the embodiment of the present application needs to determine whether the compression conditions have been met to avoid premature or over-frequent compression affecting the accuracy of inference. Among them, the two basic conditions for triggering compression are the number of markers triggering conditions: each generated A tag triggers a compression window and context state trigger condition; it is worth mentioning that when the following two formulas are satisfied at the same time, the embodiment of the present application can determine that the currently generated content has high repetitiveness and low semantic density, and trigger the compression operation, and the expressions of the above two formulas are as follows:

[0095] ;

[0096] ;

[0097] in, is the average importance, is the multi-scale entropy value.

[0098] Furthermore, the embodiment of the present application can also dynamically adjust the 、 、 .

[0099] Specifically, evaluating each paragraph generation result to obtain a corresponding semantic sparsity result may include: using a preset entropy value determination formula and generating an initial entropy value result corresponding to each preset tuple set based on the paragraph generation result, a number of preset tuple sets and the probability distribution results corresponding to the paragraph generation result, and determining a target entropy value result based on each initial entropy value result; judging whether the target entropy value result is greater than a preset paragraph length threshold; if the target entropy value result is not greater than the preset paragraph length threshold, judging whether the importance score result corresponding to the paragraph generation result is greater than a preset importance score threshold; if the importance score result corresponding to the paragraph generation result is not greater than the preset importance score threshold, setting the area corresponding to the paragraph generation result as a semantically sparse area, and setting the mark corresponding to the paragraph generation result as a mark to be compressed. Note: if the importance score result corresponding to the paragraph generation result is greater than the preset importance score threshold, the area corresponding to the paragraph generation result is set as a semantically dense area, and it is prohibited to set the mark corresponding to the paragraph generation result as a mark to be compressed; if the target entropy value result is greater than the preset paragraph length threshold, the area corresponding to the paragraph generation result is set as a semantically dense area, and it is prohibited to set the mark corresponding to the paragraph generation result as a mark to be compressed; or, determine whether the number of selection windows corresponding to the preset number of mark generation times the current mark is greater than the preset window number threshold. If the number of selection windows corresponding to the preset number of mark generation times the current mark is greater than the preset window number threshold, the area corresponding to the paragraph generation result is set as a semantically sparse area, and the mark corresponding to the paragraph generation result is set as a mark to be compressed.

[0100] Step S13: determine a semantic block division strategy based on the importance score result and the semantic sparsity result, and divide the sequence of to-be-compressed tags including each of the current tags into semantic blocks based on the semantic block division strategy, and then sort them based on the importance score corresponding to each of the semantic blocks to obtain a sorting result.

[0101] In this embodiment, a semantic block partitioning method based on vector clustering is required to divide the tags into semantically coherent semantic blocks to avoid interrupting important logical paragraphs during the compression process, and the partitioning process is as follows: First, the semantic representation vector of each tag is extracted from the tag sequence to be compressed, and the semantic representation vector includes the context vector of the Transformer middle layer or the multi-layer weighted combination vector. Subsequently, a clustering algorithm is used to divide the semantic representation vector into several semantically similar clusters, and each cluster constitutes a semantic block. In a specific embodiment, if the current window contains 4096 tags and the expected length of each block is 32, the number of clusters is set to 128. After clustering is completed, each tag is assigned a cluster label to construct a structured set of semantic blocks. In order to maintain the original order of the tags, each semantic block is sorted in ascending order according to the position of the tag in the original sequence. Subsequently, the embodiment of the present application needs to calculate the average importance score of each semantic block as the basis for subsequent compression decision sorting. In addition, for each semantic block, The corresponding average importance score is determined as follows:

[0102] ;

[0103] In this way, the embodiment of the present application uses the above-mentioned semantic clustering method to achieve compression operations while maintaining semantic integrity, processing them in units of semantically consistent blocks, greatly reducing the risk of reasoning chain breaks, while enhancing the structural perception and adaptability of the compression strategy.

[0104] After obtaining the average importance scores of all semantic blocks, the embodiment of the present application needs to sort all semantic blocks according to their importance, select and retain the marks corresponding to the first several blocks, and then set the remaining marks as compressible marks. tags, if the average length of each semantic block is , the formula for determining the number of blocks to be retained is as follows:

[0105] ;

[0106] in, is the total number of tags to be retained per cycle, is the compression factor, The number of blocks to be retained.

[0107] That is, all semantic blocks are Sort in descending order and keep the first The KV corresponding to the mark of the block is obtained, and then the marks of the remaining blocks are all marked as compressible candidate marks.

[0108] Specifically, a semantic block division strategy is determined based on the importance score result and the semantic sparsity result, and the sequence of to-be-compressed tags including each current tag is divided into semantic blocks based on the semantic block division strategy, and then sorted based on the importance score corresponding to each semantic block to obtain a sorting result, which may include: determining the first to-be-compressed tag from each current tag based on the importance score result and the semantic sparsity result, and constructing a first to-be-compressed sequence based on each first to-be-compressed tag to extract a semantic representation vector corresponding to each first to-be-compressed tag from the first to-be-compressed sequence; the semantic representation vector is a context vector and a weighted combination vector corresponding to the model layer; using a preset clustering algorithm to divide each semantic representation vector into several semantically related clusters, and assigning labels to each first to-be-compressed tag based on each cluster to determine the semantic block based on the label assignment result; the arrangement order corresponding to each first to-be-compressed tag in the semantic block is consistent with the arrangement order of each first to-be-compressed tag in the first to-be-compressed sequence; performing an overall importance score on each semantic block based on the importance score result to obtain a corresponding importance score, and sorting each semantic block in order from low to high according to the importance score to obtain a sorting result.

[0109] It is worth mentioning that no matter how the compression strategy changes, the recent A tag always retains its KV cache and does not participate in compression, and the expression is as follows:

[0110] ;

[0111] in, Indicates the final retention Cache (KV cache) collection. The above KV cache will not be compressed or discarded and will be retained in the model's memory for subsequent reasoning. Represents the KV cache set of selected important semantic blocks. These blocks are considered semantically important and are therefore retained. Indicates recent A set of marked KV caches.

[0112] Step S14: Determine compression control parameters based on the model characteristics and inference requirements of the large language model, generate a target compression strategy based on the compression control parameters, and use the target compression strategy to compress the initial inference path corresponding to the large language model to obtain the target inference path; the compression control parameters include compression interval, selection window size and compression ratio.

[0113] In this embodiment, the formula for determining the retained KV cache list is as follows:

[0114] ;

[0115] in, The formula for determining the KV cache list that represents the set of tags selected for retention and discarded is as follows:

[0116] ;

[0117] Subsequently, the embodiment of the present application needs to record the position information, score, semantic block number, etc. of each compressed mark for subsequent recovery or analysis.

[0118] In one embodiment, if the current cycle length 4096, semantic block length is 32, compression ratio If the value is 4, only 1024 tags are retained per cycle, which is the number of tags corresponding to 32 semantic blocks. Subsequently, the 4096 tags are divided into 128 semantic blocks. The average importance score of each block is calculated and sorted. The top 32 blocks are selected from the sorted results and retained, while the rest are discarded. The compaction plan is then executed, and the remaining tag KV cache is deleted.

[0119] In this embodiment, after completing KV compression, the metadata of all compressed items is structured and recorded, including their position in the original sequence, importance score, semantic block number, number of steps marked when the compression occurred, and a soft-deletion flag indicating whether recovery is allowed. This information is written to the compression log for subsequent context rollback, compression effect analysis, or policy optimization, ensuring the traceability and explainability of the compression process.

[0120] Specifically, compression control parameters are determined based on the model characteristics and reasoning requirements of the large language model, a target compression strategy is generated based on the compression control parameters, and the initial reasoning path corresponding to the large language model is compressed using the target compression strategy to obtain the target reasoning path, which may include: determining compression control parameters based on the model characteristics and reasoning requirements of the large language model, and generating a target compression strategy based on the compression control parameters; wherein the model characteristics include each model layer and each attention head; determining the number of semantic block retention blocks based on the compression interval and the semantic block length corresponding to the semantic block, retaining the full key-value cache corresponding to the number of semantic blocks of the first semantic block retention blocks in the sorting result, and setting the current tags corresponding to the remaining semantic blocks as second tags to be compressed, and constructing a second sequence to be compressed based on each second tag to be compressed; recording the position information, importance score and corresponding semantic block number corresponding to each second tag to be compressed, and compressing the initial reasoning path corresponding to the large language model using the target compression strategy and the second sequence to be compressed to obtain the target reasoning path.

[0121] Step S15: Generate code based on the target reasoning path using the large language model.

[0122] In this embodiment, in order to adapt to the dynamic changes of semantic redundancy, importance distribution and model stability during the reasoning process, the embodiment of the present application adopts an adaptive compression strategy control mechanism to adjust the compression strategy. That is, in each round of reasoning, multiple indicators such as Semantic entropy, average tag importance, KV cache growth rate, model output confidence fluctuation, and compression rollback rate are collected in real time to collect compression-related operating status indicators and provide a quantitative basis for subsequent policy adjustments.

[0123] In a specific embodiment, when semantic redundancy increases and the importance of tags is generally low, the embodiment of the present application needs to shorten the compression cycle and increase the compression ratio; if instability of the model reasoning results is detected or the compressed tags are frequently rolled back, the embodiment of the present application needs to reduce the compression intensity and expand the evaluation window.

[0124] Subsequently, the embodiment of the present application needs to control the trigger frequency and execution delay of the compression action to align it with the model state in real time to avoid frequent disturbances in the model reasoning process. To this end, a compression suppression window mechanism is introduced to dynamically regulate the execution timing and rhythm of compression. Among them, the compression suppression window mechanism is used to set a compression cooling period after a compression operation is completed, and prohibit or delay the triggering of new compression operations within this interval. After each compression, The next round of compression operation will not be triggered within ; suppose that a compression operation is completed at present, and the current marking step number is recorded as , subsequent compression will be suppressed until one of the following conditions is met: the current marking step Or the compression state is triggered urgently (such as memory is close to the upper limit, or output stability is significantly reduced). This is the compression suppression window length.

[0125] It is worth mentioning that It can be dynamically adjusted according to the current state of the model, and the adjustment formula is as follows:

[0126] ;

[0127] in: is the basic cooling length (such as 2048); Output confidence reduction for the compressed model; Rollback ratio for compression mark; , Both are adjustment coefficients used to amplify the effect of unstable feedback on cooling time.

[0128] Subsequently, when the model is subjected to compression and the reasoning becomes unstable or context is lost, the embodiment of the present application needs to automatically extend the cooling period to avoid information loss caused by continuous compression. For example, after the normal compression operation of the embodiment of the present application, if the running state of the model is stable, then set 2048, that is, compression will not be triggered again within the next 2048 marks; if the model output quality decreases, the entropy value increases, and the rollback ratio increases after compression, the compression period will be extended. To 4096 or higher; if the GPU memory pressure is too high or the cache surges, the suppression window can be forcibly interrupted and the compression operation will be triggered immediately.

[0129] It is worth mentioning that, in order to safely and efficiently compress the redundant KV cache without affecting the accuracy of language model reasoning, the embodiment of the present application introduces a weak context recovery mechanism, which can dynamically restore some compressed KV information when the reasoning quality is found to be degraded, and ensure the coherence and stability of the reasoning chain. In the process of semantic block division and importance scoring, the embodiment of the present application will generate the following two types of tag lists: Retain List: including tags in important semantic blocks and the most recent tags in the Selector window. Tags and compression candidate list (PruneList): including low-importance tags in the redundant area. Subsequently, the embodiment of the present application needs to construct a new KV cache index through RetainList to form a sparsely compressed KV structure, and the expression is as follows:

[0130] ;

[0131] In this example, the steps for performing compression on the current KV cache include: first, constructing a reserved index mask, then reconstructing the sparse KV tensor based on the mask, and then updating the Transformer cache pointer to ensure that the new KV structure is used in the decoding stage. Finally, the original position, attention score and semantic block number of the compressed token are recorded.

[0132] Among them, the Boolean mask list or index list is used to indicate which marked KV entries should be retained and which should be cleared. Subsequently, the embodiment of the present application can obtain the retained mark set RetainList obtained after the semantic block screening, and the retained mark set is the index of the mark in the generated sequence , using the reserved marker set to construct the length of the current marker step Boolean vector of , and the construction formula is as follows:

[0133] ;

[0134] Then, after completing the tag index construction operation, the embodiment of the present application needs to extract the corresponding entries from the original Key and Value tensors according to the index to reconstruct the compressed sparse KV tensor. In a specific embodiment, the embodiment of the present application can select the original 、 Compress to new 、 It is worth mentioning that the obtained and The size of is significantly reduced, and only the KV pairs with retained tags are included to ensure that the sparse reconstruction of the KV cache does not destroy the tag order and provides a compact and computationally efficient attention context for the subsequent decoding process, where Indicates the sequence length.

[0135] Furthermore, after reconstructing the sparse KV tensor, the embodiment of the present application needs to compress the 、 Writes to the KV cache structure within the Transformer model, replacing the original complete cache. For models with a multi-layer structure, this operation updates the KV cache pointer for each layer, ensuring that subsequent decoding stages reference only the retained context information. Furthermore, if the model uses rotated position encoding or absolute position embedding, the corresponding position information must be updated synchronously to prevent attention dislocation and ensure the logical continuity of the compressed context.

[0136] In addition, considering that tags that have potential contributions to subsequent reasoning may be mistakenly deleted during the compression process, the above-mentioned weak context recovery mechanism can realize the dynamic recovery of some compressed tags. Among them, in the continuous generation process after compression, if any of the following conditions is met: the confidence of the model output tag drops significantly and duplicate or invalid tags are continuously generated, the embodiment of the present application can start the recovery logic. Among them, the specific recovery logic is to selectively restore some of the KV pairs that have been compressed, and the priority is determined by the original attention score ranking. In addition, the number of restorations is limited to a threshold, such as restoring the first 64 tags at most to prevent excessive recovery, and inserting the restored tags into the current KV structure, and rebuilding the corresponding position encoding mapping table. It is worth mentioning that the above process is equivalent to a lightweight rollback, thereby recovering the performance degradation caused by the loss of context information without interfering with the main reasoning path.

[0137] In this embodiment, during the inference termination and output recording phase, the embodiment of the present application first needs to judge multiple conditions to determine whether the generation process should be terminated, including the generation of a termination marker, the reaching of the maximum length limit, or the stable convergence of the model output confidence. Once any termination condition is met, the tag generation operation is immediately terminated, and the complete inference result is output, including the generated text sequence, structured information (such as the answer or code snippet), and an optional tag confidence track. At the same time, the embodiment of the present application needs to synchronously record the compression path information during this round of inference, such as the number of compression execution rounds, the retention ratio of each round, the change in semantic entropy, the number of compressed tags, the number of rollback triggers, the final KV cache size, and other key indicators. Among them, the above data can be output in a structured form and can be used for interface return, debugging visualization, or compression strategy learning module feedback update, thereby achieving traceability, explainability, and optimizability of the inference process, thereby ensuring that the model inference still has output integrity and behavior transparency under compression intervention, thereby building a closed-loop, highly stable inference compression system.

[0138] Specifically, after using the large language model and generating code based on the target reasoning path, the method may further include: detecting and collecting indicators of the process of generating the target reasoning path, and using the collected indicators to adjust the compression interval, selection window size and compression ratio in the compression control parameters to obtain new compression control parameters; the indicators include Semantic entropy, average importance of tags, growth rate of full key-value cache, fluctuation of model output confidence and compression rollback rate; record the current number of tag steps, and use the preset compression suppression window mechanism to determine the compression cooling period based on the current number of tag steps and the target compression suppression window length to suppress the compression operation during the compression cooling period; wherein, the target compression suppression window length is the compression suppression window length determined based on the preset adjustment coefficient, the preset basic cooling length, the decrease in model output confidence and the compression rollback rate; the preset adjustment coefficient is used to amplify the impact of the collected unstable feedback information on the compression cooling period; construct an index list based on the target reasoning path, and use the preset weak context recovery mechanism to process the target reasoning path based on the metadata corresponding to the target reasoning path and the cache pointer corresponding to the model layer to obtain a new target reasoning path; the preset weak context recovery mechanism is used when monitoring When the reasoning quality corresponding to the target reasoning path decreases, the full key-value cache information of a preset size is restored in the full key-value cache area; the metadata corresponding to the target reasoning path is structured and recorded in the compression log; the metadata includes the position of the target reasoning path in the sequence of tags to be compressed, the importance score, the semantic block number, the number of tag steps and the soft deletion mark indicating whether recovery is allowed; it is judged whether the target reasoning path meets the preset generation end conditions. If the target reasoning path meets the preset generation end conditions, the current process of generating the target reasoning path is ended; the target reasoning path includes the number of compression execution rounds, the compression ratio, the change in semantic entropy, the total number of compressed tags, the number of rollback triggers and the full key-value cache size; the preset generation end conditions include the generation of the termination mark, the length corresponding to the target reasoning path reaches the preset length threshold, and the confidence corresponding to the large language model meets the preset confidence convergence conditions.

[0139] As can be seen from the above, the embodiment of the present application first needs to use a preset autoregressive method and generate each current tag based on the initial reasoning path corresponding to the large language model, and cache the query vector corresponding to each current tag, and then select a preset number of recently generated query vectors from each current tag to construct a selection window based on each query vector; then, based on the selection window, obtain the sliding window attention corresponding to the historical tags and each model layer and each attention head in the large language model, and process the corresponding historical tags based on the sliding window attention to obtain the importance score result and the paragraph generation result, and then evaluate each paragraph generation result to obtain the corresponding semantic sparsity result; then, determine the semantic block partitioning strategy based on the importance score result and the semantic sparsity result, and divide the sequence of tags to be compressed including each current tag into semantic blocks based on the semantic block partitioning strategy, and then sort them based on the importance score corresponding to each semantic block to obtain a sorting result; finally, determine the compression control parameters based on the model characteristics and reasoning requirements of the large language model to generate a target compression strategy based on the compression control parameters, and use the target compression strategy to compress the initial reasoning path corresponding to the large language model to obtain the target reasoning path. In this way, the efficiency of code generation is improved during code generation based on large language models, thereby reducing the memory required to store the inference path.

[0140] Accordingly, see Figure 2 As shown, the present application also provides a code generation device based on a large language model, comprising:

[0141] A selection window determination module 11 is configured to generate each current token using a preset autoregressive method based on an initial inference path for code generation corresponding to the large language model, cache query vectors corresponding to each current token, and then select a preset number of most recently generated tokens to be processed from each current token to construct a selection window based on the query vectors corresponding to each token to be processed;

[0142] A semantic sparsity result determination module 12 is configured to obtain the sliding window attention corresponding to the target model layer and the target attention head respectively based on the selection window, and process the corresponding historical tags based on the sliding window attention to obtain an importance score result and a paragraph generation result, and then evaluate each of the paragraph generation results to obtain a corresponding semantic sparsity result; the target model layer is the model layer corresponding to the historical tags in each model layer of the large language model; the target attention head is the attention head corresponding to the historical tags in each attention head of the large language model;

[0143] a sorting result determination module 13, configured to determine a semantic block division strategy based on the importance score result and the semantic sparsity result, and to divide the sequence of to-be-compressed tags including each current tag into semantic blocks based on the semantic block division strategy, and then sort the semantic blocks based on the importance score corresponding to each semantic block to obtain a sorting result;

[0144] A compression result determination module 14 is configured to determine compression control parameters based on the model characteristics and inference requirements of the large language model, generate a target compression strategy based on the compression control parameters, and compress the initial inference path corresponding to the large language model using the target compression strategy to obtain a target inference path; the compression control parameters include a compression interval, a selection window size, and a compression ratio;

[0145] The code generation module 15 is configured to generate code based on the target reasoning path using the large language model.

[0146] As can be seen from the above, before performing code generation based on a large language model, the embodiment of the present application first needs to use a preset autoregressive method and generate each current tag based on the initial reasoning path corresponding to the large language model, and cache the query vector corresponding to each current tag. Then, a preset number of recently generated query vectors are selected from each current tag to construct a selection window based on each query vector; then, based on the selection window, the sliding window attention corresponding to the historical tags and each model layer and each attention head in the large language model is obtained, and the corresponding historical tags are processed based on the sliding window attention to obtain an importance score result and a paragraph generation result. Then, each paragraph generation result is evaluated to obtain a corresponding semantic sparsity result; then, based on the importance score result and the semantic sparsity result, a semantic block partitioning strategy is determined, and based on the semantic block partitioning strategy, the sequence of tags to be compressed including each current tag is divided into semantic blocks, and then sorted based on the importance score corresponding to each semantic block to obtain a sorting result; finally, based on the model characteristics and reasoning requirements of the large language model, a compression control parameter is determined to generate a target compression strategy based on the compression control parameter, and the initial reasoning path corresponding to the large language model is compressed using the target compression strategy to obtain a target reasoning path. In this way, the efficiency of code generation is improved during code generation based on large language models, thereby reducing the memory required to store the inference path.

[0147] In some specific implementations, the selection window determination module 11 may specifically include:

[0148] a token generation unit, configured to generate each current token using a preset autoregressive method and based on an initial inference path corresponding to the large language model, and determine a query vector, a key vector, and a value vector corresponding to each current token;

[0149] a vector storage unit, configured to store each of the query vectors, the key vector, and the value vector in a query buffer and a full key-value cache; wherein the query buffer only retains the query vectors corresponding to a preset number of current tags that were most recently generated;

[0150] The selection window construction unit is configured to extract the preset number of query vectors from the query buffer, determine a window construction standard based on each query vector, and construct a selection window based on the window construction standard and each current mark.

[0151] In some specific implementations, the semantic sparse result determination module 12 may specifically include:

[0152] a sliding window attention determination unit, configured to determine each model layer and each attention head in the large language model, and determine the number of model layers and the number of attention head sequences corresponding to the historical marker, and then determine the sliding window attention corresponding to the historical marker based on the selection window, the query vector, the key vector, the value vector, the number of model layers, and the number of attention head sequences; wherein the numerical value corresponding to the sliding window attention is proportional to the contribution value of the historical marker in the selection window;

[0153] a unit for determining an importance score to be smoothed, configured to count the total number of model layers corresponding to each of the model layers and the total number of attention heads corresponding to each of the attention heads, so as to determine the importance score to be smoothed based on the total number of model layers, the total number of attention heads, and the sliding window attention;

[0154] The importance score result determination unit is configured to establish a paragraph generation result based on each of the current tags, and to smooth the importance score to be smoothed based on a window width corresponding to the selection window to obtain an importance score result.

[0155] In some specific implementations, the semantic sparse result determination module 12 may specifically include:

[0156] an entropy value result determining unit, configured to generate an initial entropy value result corresponding to each of the preset tuple sets using a preset entropy value determination formula and based on the paragraph generation result and probability distribution results corresponding to a plurality of preset tuple sets and the paragraph generation result, and determine a target entropy value result based on each of the initial entropy value results;

[0157] a first to-be-compressed mark determination unit, configured to determine whether the target entropy value result is greater than a preset paragraph length threshold; if the target entropy value result is not greater than the preset paragraph length threshold, determine whether the importance score result corresponding to the paragraph generation result is greater than a preset importance score threshold; if the importance score result corresponding to the paragraph generation result is not greater than the preset importance score threshold, set the area corresponding to the paragraph generation result as a semantically sparse area, and set the mark corresponding to the paragraph generation result as a to-be-compressed mark;

[0158] a second to-be-compressed mark determination unit, configured to set the region corresponding to the paragraph generation result as a semantically dense region and prohibit setting the mark corresponding to the paragraph generation result as a to-be-compressed mark if the importance score result corresponding to the paragraph generation result is greater than the preset importance score threshold;

[0159] a third to-be-compressed mark determination unit, configured to, if the target entropy value result is greater than the preset paragraph length threshold, set the region corresponding to the paragraph generation result as a semantically dense region, and prohibit setting the mark corresponding to the paragraph generation result as a to-be-compressed mark;

[0160] The fourth unit for determining the mark to be compressed is used to determine whether the number of selection windows corresponding to the number of current marks generated by generating the preset marks is greater than the preset window number threshold. If the number of selection windows corresponding to the number of current marks generated by generating the preset marks is greater than the preset window number threshold, the area corresponding to the paragraph generation result is set as a semantically sparse area, and the mark corresponding to the paragraph generation result is set as a mark to be compressed.

[0161] In some specific implementations, the ranking result determination module 13 may specifically include:

[0162] a first to-be-compressed sequence determination unit, configured to determine a first to-be-compressed token from each of the current tokens based on the importance score result and the semantic sparsity result, and construct a first to-be-compressed sequence based on each of the first to-be-compressed tokens, so as to extract a semantic representation vector corresponding to each of the first to-be-compressed tokens from the first to-be-compressed sequence; the semantic representation vector being a context vector corresponding to the model layer and a weighted combination vector;

[0163] a semantic block determining unit configured to divide each of the semantic representation vectors into a plurality of semantically related clusters using a preset clustering algorithm, and assign labels to each of the first to-be-compressed tokens based on each cluster, so as to determine a semantic block based on the label assignment results; wherein the arrangement order of the first to-be-compressed tokens in the semantic block is consistent with the arrangement order of the first to-be-compressed tokens in the first to-be-compressed sequence;

[0164] The ranking result determining unit is used to perform an overall importance score on each of the semantic blocks based on the importance score result to obtain a corresponding importance score, and to sort each of the semantic blocks in order from low to high importance scores to obtain a ranking result.

[0165] In some specific implementations, the compression result determination module 14 may specifically include:

[0166] a compression strategy determination unit, configured to determine compression control parameters based on model characteristics and inference requirements of the large language model, and generate a target compression strategy based on the compression control parameters; wherein the model characteristics include each model layer and each attention head;

[0167] a second sequence to be compressed determining unit, configured to determine the number of reserved semantic blocks based on the compression interval and the length of the semantic block corresponding to the semantic block, so as to retain the full key value cache corresponding to the semantic blocks of the first number of reserved semantic blocks in the sorting result, set the current marks corresponding to the remaining semantic blocks as second marks to be compressed, and construct a second sequence to be compressed based on the second marks to be compressed;

[0168] A target inference path determination unit is configured to record the position information, importance score, and corresponding semantic block number corresponding to each second mark to be compressed, and to compress the initial inference path corresponding to the large language model using the target compression strategy and the second sequence to be compressed to obtain a target inference path.

[0169] In some specific implementations, the code generation apparatus based on the large language model may further include:

[0170] A compression control parameter adjustment unit is configured to detect and collect indicators of the process of generating the target inference path, and to adjust the compression interval, the selection window size, and the compression ratio in the compression control parameters using the collected indicators to obtain new compression control parameters; the indicators include n-gram semantic entropy, average tag importance, full key-value cache growth rate, model output confidence fluctuation, and compression rollback rate;

[0171] a compression cooling period determination unit, configured to record a current number of marked steps and determine a compression cooling period based on the current number of marked steps and a target compression cooling period length using a preset compression suppression window mechanism, so as to suppress compression operations during the compression cooling period; wherein the target compression cooling period length is a compression cooling period length determined based on a preset adjustment coefficient, a preset basic cooling period length, a decrease in model output confidence, and the compression rollback rate; and the preset adjustment coefficient is used to amplify the effect of collected unstable feedback information on the compression cooling period.

[0172] An inference path processing unit is configured to construct an index list based on the target inference path, and to process the target inference path based on the metadata corresponding to the target inference path and the cache pointer corresponding to the model layer using a preset weak context recovery mechanism to obtain a new target inference path; the preset weak context recovery mechanism is configured to restore a preset amount of full key-value cache information in the full key-value cache area when it is detected that the inference quality corresponding to the target inference path has deteriorated;

[0173] An information recording unit, configured to record the metadata corresponding to the target reasoning path in a structured manner into a compression log; the metadata includes the position of the target reasoning path in the to-be-compressed tag sequence, its importance score, its semantic block number, its tag step count, and a soft-deletion flag indicating whether recovery is allowed;

[0174] An inference path judgment unit is used to judge whether the target inference path meets the preset generation end conditions. If the target inference path meets the preset generation end conditions, the process of currently generating the target inference path is terminated; the target inference path includes the number of compression execution rounds, the compression ratio, the change in semantic entropy, the total number of compressed tags, the number of rollback triggers and the full key-value cache size; the preset generation end conditions include the generation of a termination flag, the length corresponding to the target inference path reaches a preset length threshold, and the confidence corresponding to the large language model meets the preset confidence convergence conditions.

[0175] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be considered as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the code generation method based on the large language model disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0176] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0177] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0178] The operating system 221 is used to manage and control the hardware devices on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of implementing the code generation method based on the large language model and executed by the electronic device 20 as disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program capable of implementing other specific tasks.

[0179] Furthermore, this application discloses a computer-readable storage medium for storing a computer program; wherein, when executed by a processor, the computer program implements the aforementioned large language model-based code generation method. The specific steps of this method can be found in the corresponding content disclosed in the aforementioned embodiments and will not be repeated here.

[0180] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0181] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0182] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0183] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0184] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A code generation method based on a large language model, characterized in that: include: Generate each current token using a preset autoregressive method based on an initial inference path for code generation corresponding to the large language model, cache a query vector corresponding to each current token, and then select a preset number of most recently generated tokens to be processed from each current token to construct a selection window based on the query vectors corresponding to each token to be processed; Obtaining the sliding window attention corresponding to the target model layer and the target attention head respectively based on the selected window, and processing the corresponding historical tags based on the sliding window attention to obtain an importance score result and a paragraph generation result, and then evaluating each of the paragraph generation results to obtain a corresponding semantic sparsity result; The target model layer is the model layer corresponding to the historical marker among the model layers of the large language model; the target attention head is the attention head corresponding to the historical marker among the attention heads of the large language model; Determining a semantic block division strategy based on the importance score result and the semantic sparsity result, dividing the to-be-compressed tag sequence including each current tag into semantic blocks based on the semantic block division strategy, and then sorting the semantic blocks based on the importance scores corresponding to the semantic blocks to obtain a sorting result; Determining compression control parameters based on the model characteristics and inference requirements of the large language model, generating a target compression strategy based on the compression control parameters, and compressing an initial inference path corresponding to the large language model using the target compression strategy to obtain a target inference path; the compression control parameters include a compression interval, a selection window size, and a compression ratio; Code is generated based on the target reasoning path using the large language model.

2. The code generation method based on a large language model according to claim 1, characterized in that The method includes generating each current token using a preset autoregressive method and based on an initial inference path for code generation corresponding to the large language model, caching a query vector corresponding to each current token, and then selecting a preset number of most recently generated tokens to be processed from each current token to construct a selection window based on the query vectors corresponding to each token to be processed, including: Generate each current token using a preset autoregressive method and based on an initial inference path corresponding to the large language model, and determine a query vector, a key vector, and a value vector corresponding to each current token; Storing each of the query vectors, the key vector, and the value vector in a query buffer and a full key-value cache; wherein the query buffer only retains the query vectors corresponding to the most recently generated preset number of current tags; The preset number of query vectors are extracted from the query buffer, and a window construction criterion is determined based on each of the query vectors, so as to construct a selection window based on the window construction criterion and each of the current marks.

3. The code generation method based on a large language model according to claim 2, characterized in that: The method of acquiring the sliding window attention corresponding to the target model layer and the target attention head based on the selection window, and processing the corresponding historical tags based on the sliding window attention to obtain the importance scoring result and the paragraph generation result includes: Determine each model layer and each attention head in the large language model, and determine the number of model layers and the number of attention head sequences corresponding to the historical marker, and then determine the sliding window attention corresponding to the historical marker based on the selection window, the query vector, the key vector, the value vector, the number of model layers, and the number of attention head sequences; wherein the numerical value corresponding to the sliding window attention is proportional to the contribution value of the historical marker in the selection window; Counting the total number of model layers corresponding to each of the model layers and the total number of attention heads corresponding to each of the attention heads, so as to determine an importance score to be smoothed based on the total number of model layers, the total number of attention heads, and the sliding window attention; A paragraph generation result is established based on each of the current marks, and the importance score to be smoothed is smoothed based on a window width corresponding to the selection window to obtain an importance score result.

4. The code generation method based on a large language model according to claim 1, characterized in that The evaluation of each paragraph generation result to obtain a corresponding semantic sparsity result includes: generating an initial entropy value result corresponding to each of the preset tuple sets using a preset entropy value determination formula and based on the paragraph generation result and probability distribution results corresponding to a plurality of preset tuple sets and the paragraph generation result, and determining a target entropy value result based on each of the initial entropy value results; Determine whether the target entropy value result is greater than a preset paragraph length threshold; if the target entropy value result is not greater than the preset paragraph length threshold, determine whether the importance score result corresponding to the paragraph generation result is greater than a preset importance score threshold; if the importance score result corresponding to the paragraph generation result is not greater than the preset importance score threshold, set the area corresponding to the paragraph generation result as a semantically sparse area, and set the mark corresponding to the paragraph generation result as a mark to be compressed; If the importance score result corresponding to the paragraph generation result is greater than the preset importance score threshold, the area corresponding to the paragraph generation result is set as a semantically dense area, and the mark corresponding to the paragraph generation result is prohibited from being set as a mark to be compressed; If the target entropy value result is greater than the preset paragraph length threshold, the area corresponding to the paragraph generation result is set as a semantically dense area, and the mark corresponding to the paragraph generation result is prohibited from being set as a mark to be compressed; Or, determine whether the number of selection windows corresponding to the number of current tags generated by generating a preset number of tags is greater than the preset window number threshold; if the number of selection windows corresponding to the number of current tags generated by generating a preset number of tags is greater than the preset window number threshold, then set the area corresponding to the paragraph generation result as a semantically sparse area, and set the tag corresponding to the paragraph generation result as a tag to be compressed.

5. The code generation method based on a large language model according to claim 1, characterized in that The determining of a semantic block division strategy based on the importance score result and the semantic sparsity result, dividing the to-be-compressed tag sequence including each current tag into semantic blocks based on the semantic block division strategy, and then sorting the semantic blocks based on the importance scores corresponding to the semantic blocks to obtain a sorting result, including: Determining a first to-be-compressed token from each of the current tokens based on the importance score result and the semantic sparsity result, and constructing a first to-be-compressed sequence based on each of the first to-be-compressed tokens, so as to extract a semantic representation vector corresponding to each of the first to-be-compressed tokens from the first to-be-compressed sequence; the semantic representation vector being a context vector corresponding to the model layer and a weighted combination vector; Using a preset clustering algorithm, each of the semantic representation vectors is divided into a plurality of semantically related clusters, and labels are assigned to each of the first to-be-compressed tokens based on each cluster, so as to determine a semantic block based on the label assignment results; the arrangement order of each of the first to-be-compressed tokens in the semantic block is consistent with the arrangement order of each of the first to-be-compressed tokens in the first to-be-compressed sequence; Based on the importance scoring result, the overall importance of each semantic block is scored to obtain a corresponding importance score, and the semantic blocks are sorted in order from low to high according to the importance score to obtain a sorting result.

6. The code generation method based on a large language model according to claim 1, characterized in that The step of determining compression control parameters based on the model characteristics and inference requirements of the large language model, generating a target compression strategy based on the compression control parameters, and compressing an initial inference path corresponding to the large language model using the target compression strategy to obtain a target inference path includes: Determining compression control parameters based on model characteristics and inference requirements of the large language model, and generating a target compression strategy based on the compression control parameters; wherein the model characteristics include each model layer and each attention head; Determining the number of reserved semantic blocks based on the compression interval and the length of the semantic block corresponding to the semantic block, retaining the full key value cache corresponding to the semantic blocks of the first number of reserved semantic blocks in the sorting result, setting the current marks corresponding to the remaining semantic blocks as second to-be-compressed marks, and constructing a second to-be-compressed sequence based on each of the second to-be-compressed marks; Record the position information, importance score, and corresponding semantic block number corresponding to each second mark to be compressed, and use the target compression strategy and the second sequence to be compressed to compress the initial reasoning path corresponding to the large language model to obtain a target reasoning path.

7. The code generation method based on a large language model according to any one of claims 1 to 6, characterized in that: After generating code using the large language model and based on the target reasoning path, the method further includes: The process of generating the target inference path is detected and indicators are collected, and the compression interval, the selection window size, and the compression ratio in the compression control parameters are adjusted using the collected indicators to obtain new compression control parameters; the indicators include n-gram semantic entropy, average tag importance, full key-value cache growth rate, model output confidence fluctuation, and compression rollback rate; Recording the current number of marked steps, and determining a compression cooldown period based on the current number of marked steps and a target compression cooldown period using a preset compression suppression window mechanism, so as to suppress compression operations during the compression cooldown period; wherein the target compression cooldown period is a compression cooldown period determined based on a preset adjustment coefficient, a preset basic cooling period, a decrease in model output confidence, and the compression rollback rate; the preset adjustment coefficient is used to amplify the effect of the collected unstable feedback information on the compression cooldown period; An index list is constructed based on the target reasoning path to obtain a new target reasoning path by processing the target reasoning path using a preset weak context recovery mechanism based on the metadata corresponding to the target reasoning path and the cache pointer corresponding to the model layer; the preset weak context recovery mechanism is used to restore a preset amount of full key-value cache information in the full key-value cache area when it is detected that the reasoning quality corresponding to the target reasoning path has deteriorated; The meta-information corresponding to the target reasoning path is recorded in a structured manner in the compression log; the meta-information includes the position of the target reasoning path in the to-be-compressed mark sequence, the importance score, the semantic block number, the number of marking steps, and the soft deletion flag indicating whether recovery is allowed; Determine whether the target reasoning path meets the preset generation end conditions. If the target reasoning path meets the preset generation end conditions, end the current process of generating the target reasoning path; the target reasoning path includes the number of compression execution rounds, compression ratio, semantic entropy change, total number of compressed tags, number of rollback triggers and full key-value cache size; the preset generation end conditions include termination flag generation, the length corresponding to the target reasoning path reaches a preset length threshold, and the confidence corresponding to the large language model meets the preset confidence convergence conditions.

8. A code generation device based on a large language model, characterized in that: include: a selection window determination module, configured to generate each current token using a preset autoregressive method based on an initial inference path for code generation corresponding to the large language model, cache query vectors corresponding to each current token, and then select a preset number of most recently generated tokens to be processed from each current token to construct a selection window based on the query vectors corresponding to each token to be processed; a semantic sparsity result determination module, configured to obtain the sliding window attention corresponding to the target model layer and the target attention head respectively based on the selection window, and process the corresponding historical tags based on the sliding window attention to obtain an importance score result and a paragraph generation result, and then evaluate each of the paragraph generation results to obtain a corresponding semantic sparsity result; The target model layer is the model layer corresponding to the historical marker among the model layers of the large language model; the target attention head is the attention head corresponding to the historical marker among the attention heads of the large language model; a sorting result determination module, configured to determine a semantic block division strategy based on the importance score result and the semantic sparsity result, and to divide the sequence of to-be-compressed tags including each current tag into semantic blocks based on the semantic block division strategy, and then sort the semantic blocks based on the importance scores corresponding to the semantic blocks to obtain a sorting result; a compression result determination module, configured to determine compression control parameters based on the model characteristics and inference requirements of the large language model, generate a target compression strategy based on the compression control parameters, and compress the initial inference path corresponding to the large language model using the target compression strategy to obtain a target inference path; the compression control parameters include a compression interval, a selection window size, and a compression ratio; A code generation module is configured to generate code based on the target reasoning path using the large language model.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the code generation method based on a large language model according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program, wherein when the computer program is executed by a processor, the code generation method based on a large language model according to any one of claims 1 to 7 is implemented.

Images