Code generation method based on dynamic hierarchical sparse attention

Abstract

The invention discloses a code generation method based on dynamic hierarchical sparse attention, and belongs to the technical field of data processing, and the method comprises the following steps: S1, collecting an input data set; s2, constructing a dynamic hierarchical sparse attention model; s3, optimizing the dynamic layered sparse attention model by using the input data set, and generating a final dynamic layered sparse attention model; and S4, generating a code by using the final dynamic hierarchical sparse attention model. According to the method, the model can still keep high-quality code generation capacity at extremely low labeling cost, and a flexible and efficient solution is provided for agile development and automatic testing.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a code generation method based on dynamic hierarchical sparse attention. Background Art

[0002] With the rapid development of artificial intelligence (AI), large language models (LLMs) have demonstrated significant potential in the field of code generation. They can automatically generate functionally complete code snippets based on natural language descriptions, significantly improving development efficiency. The core of these models relies on the self-attention mechanism, which achieves contextual understanding by globally modeling the semantic relationships of input sequences. However, with the increasing complexity of code and the demand for long-term context support in industrial-grade tasks, the quadratic computational complexity of traditional attention mechanisms has become increasingly prominent. For example, when processing a 16KB code sequence, a standard Transformer model requires storing tens of GB of key-value cache (KV-Cache), significantly increasing the risk of GPU memory overflow. To alleviate this problem, sparse attention techniques reduce the computational load by limiting the attention range or dynamically pruning the attention field. However, these general solutions struggle to adapt to the multi-scale nature of grammatical structure and logical dependencies in code generation tasks, and they still face bottlenecks such as high GPU memory usage and high decoding latency in long sequence scenarios. Existing methods primarily focus on single-modal text input and lack targeted optimization for code semantic enhancement and small sample size scenarios, limiting their practical application in real-world development environments.

[0003] Current sparse attention techniques for code generation suffer from two core drawbacks: an imbalance between computational efficiency and semantic preservation, and insufficient generalization in small-sample scenarios. First, traditional static sparsity methods ignore the functional differences between different code layers—shallow layers need to capture local syntax, while deeper layers need to model global logic. A unified sparsity strategy results in loss of critical information or redundant computation. While dynamic sparsity schemes can adaptively adjust the attention range, they rely on a dense pre-population phase to explicitly compute the full attention matrix. This leads to memory overhead that grows with the square of the sequence length, making them difficult to deploy for very long code generation. Second, existing feedback-driven optimization frameworks rely heavily on large-scale annotated data, making them difficult to quickly adapt to new tasks in low-resource scenarios. These issues limit the practicality of these models in industrial-scale high-concurrency and multi-task scenarios. A new architecture that balances efficiency and adaptability is urgently needed: one that dynamically allocates computational resources through a hierarchical sparsity strategy and extracts essential task features from a small number of examples through meta-learning. This approach can reduce video memory usage while improving the generation quality of complex code logic and small-sample generalization.

[0004] A fixed-window sparse attention mechanism has been proposed in the prior art to reduce the computational complexity of long sequence processing. This technique uses a predefined sliding window to limit the context coverage of each attention head, for example, setting a fixed local window span of 512 tokens, while retaining a few initial tokens as global anchors to maintain baseline performance. During the decoding phase, a dynamic sliding window is used to update the key-value cache, reducing memory usage and improving inference throughput. Its core assumption is that all attention heads are functionally homogeneous, so a unified sparsity rule is applied to all heads and layers. This method is efficient in hardware implementation and supports real-time inference in streaming scenarios. The solution avoids dynamic decision-making overhead through the design of static sparse masks, such as fixing the window span parameter during pre-training, ensuring deterministic computation flow. Experiments show that this reduces video memory usage by 30% in a 16KB text summarization task. However, it does not optimize for the hierarchical semantics of code generation tasks. The main drawback of this technique is that the static sparsity strategy does not match the multi-scale requirements of code generation tasks. Fixed windows force all attention heads to follow the same span rule, ignoring the heterogeneity of the model, where shallow layers need to focus on local syntax while deep layers need to model global logic. For example, when generating complex algorithms, if deep attention heads are confined to a narrow window, they will not be able to capture long-range logical dependencies, resulting in loop condition errors or missing function calls in the generated code. Static windows do not adapt to the dynamic changes in input length, and in long sequence scenarios, there is still the problem of high video memory usage, making it difficult to support industrial-grade multi-task concurrency. This method lacks the design of a code semantic enhancement mechanism and cannot optimize generation quality through contrastive learning or feedback distillation, resulting in a significant increase in error rate in few-sample scenarios. These problems limit its practicality in complex code generation tasks.

[0005] Existing techniques also propose a dynamic sparse attention mechanism (H2O), which dynamically prunes a key-value cache by calculating importance scores of attention weights in real time. During the pre-population phase, this technique calculates a full attention matrix to assess token importance, screens for heavy hitters, and constructs a sparse cache. During the decoding phase, combined with a recent token retention strategy, the cache contents are dynamically adjusted to balance global and local information. Its advantage lies in adaptively adapting the importance score to the characteristics of the input sequence, avoiding the rigid limitations of a static window. For example, in code generation, H2O can automatically identify high-frequency API calls or recursive structures as key tokens, improving logical coherence. This method supports a hybrid dense-sparse computation mode, retaining full attention to maintain accuracy in short sequence scenarios. Experiments show that it can increase decoding throughput by five times in code completion tasks, but it lacks optimization strategies for small sample sizes. The core issue with this technique lies in the high computational and memory costs introduced by the dynamic sparsity mechanism. The explicit calculation of the full attention matrix during the pre-population phase results in over 24GB of video memory usage when processing a 16KB sequence, making it impossible to run large models on consumer-grade GPUs. Real-time importance scores must be updated token by token during the decoding phase, adding approximately 40% additional computational load and significantly reducing throughput. Dynamic logic relies on complex cache eviction strategies, making it difficult to integrate into standard inference engines and limiting deployment flexibility. More critically, the technology lacks a fast adaptation mechanism for low-sample scenarios, such as a meta-learning framework or a self-supervised contrast enhancement module, resulting in a sharp decline in performance in low-resource tasks. These issues severely limit its practicality in industrial-grade high-concurrency, multi-task environments. Summary of the Invention

[0006] In order to solve the above problems, the present invention proposes a code generation method based on dynamic hierarchical sparse attention.

[0007] The technical solution of the present invention is: a code generation method based on dynamic hierarchical sparse attention includes the following steps:

[0008] S1, collect input data set;

[0009] S2. Build a dynamic hierarchical sparse attention model;

[0010] S3. Optimize the dynamic hierarchical sparse attention model using the input data set to generate the final dynamic hierarchical sparse attention model;

[0011] S4. Generate code using the final dynamic hierarchical sparse attention model.

[0012] Furthermore, in S1, the input dataset includes a sample task dataset, a long-context calibration dataset, and a user feedback dataset;

[0013] Sample task dataset Dfew The expression is:

[0014] D few ={(t i ,u i )};

[0015] Where, t i represents the natural language task description, u i Represents a unit test suite;

[0016] User feedback dataset D fb The expression is:

[0017] D fb ={(x′ j ,f j )};

[0018] Where x′ j represents the jth generated code in the user feedback dataset, f j Indicates natural language feedback.

[0019] Furthermore, S2 includes the following sub-steps:

[0020] S21. Dynamically expand the sliding window using the local layer of the dynamic hierarchical sparse attention model;

[0021] S22. Use the global layer of the dynamic hierarchical sparse attention model to perform a low-rank approximation of the attention matrix;

[0022] S23. Use the bridge layer of the dynamic hierarchical sparse attention model to splice the local layer output and the global layer output to complete the construction of the dynamic hierarchical sparse attention model.

[0023] Furthermore, S21 includes the following sub-steps:

[0024] S211. Calculate the sliding window span according to the length of the input data set;

[0025] S212. Calculate the query vector and the key vector;

[0026] S213. Calculate the attention weight according to the sliding window span, the query vector, and the key vector;

[0027] S214. Based on the attention weight of the local layer, complete the dynamic expansion of the sliding window.

[0028] Furthermore, in S211, the sliding window span S local The calculation formula is:

[0029]

[0030] Where α represents the learnable parameter, N represents the length of the input dataset, log(·) represents the logarithmic function, Indicates rounding down;

[0031] In S212, the query vector Q i The expression is:

[0032] Q i =W q h i ;

[0033] Where W q represents the learnable weight matrix that maps the hidden state of position i to the query vector space, h i represents the hidden state of position i;

[0034] In S212, the key vector K j The expression is:

[0035] K j =W k h j ;

[0036] Where W k represents the learnable weight matrix used to transform the hidden state at position j into the key vector, h j represents the hidden state at position j;

[0037] In S213, the attention weight A local The expression of (i,j) is:

[0038]

[0039] Where, d k represents the dimension size used to scale the dot product result, T represents transpose, and Softmax(·) represents the activation function.

[0040] Furthermore, S22 includes the following sub-steps:

[0041] S221, calculating the global layer bond matrix;

[0042] S222, calculating the global layer value matrix;

[0043] S223. Based on the global layer key matrix and the global layer value matrix, the global layer low-rank attention matrix is ​​calculated using the low-rank approximation method.

[0044] Furthermore, in S221, the expression of the global layer bond matrix U is:

[0045] U=W u K;

[0046] Where Wu represents the first learnable parameter matrix, K represents the key matrix of the intermediate layer;

[0047] In S222, the global layer value matrix V proj The expression is:

[0048] V proj =W v V;

[0049] Where W v represents the second learnable parameter matrix, and V represents the value matrix of the intermediate layer;

[0050] In S223, the global layer low-rank attention matrix A global The expression is:

[0051]

[0052] Where Q represents the query matrix, T represents the transpose, Softmax(·) represents the activation function, and d k Indicates the dimension size used to scale the dot product result.

[0053] Furthermore, S23 includes the following sub-steps:

[0054] S231, using a bridge layer to splice the local layer output and the global layer output;

[0055] S232. Calculate the bridge layer gating weight according to the splicing result;

[0056] S233. According to the gating weight of the bridge layer, the output of the bridge layer is obtained to complete the construction of the dynamic hierarchical sparse attention model.

[0057] Furthermore, in S232, the calculation formula of the bridge layer gating weight γ is:

[0058] γ=σ(W g [h local ;h global ]);

[0059] Where h local represents the local layer output, h global represents the global layer output, σ(·) represents the Sigmoid function, W g represents the third learnable parameter matrix;

[0060] In S233, the output h of the bridge layer bridge The expression is:

[0061] h bridge =γ·h local +(1-γ)·h global ;

[0062] Where γ represents the gating weight of the bridge layer.

[0063] Furthermore, S3 includes the following sub-steps:

[0064] S31, mapping the natural language task description of the sample task dataset into a vector;

[0065] S32, encoding the unit test set of the sample task data set into a vector;

[0066] S33. Input the vector corresponding to the natural language task description and the vector corresponding to the unit test set into the graph neural network to generate a meta-feature vector;

[0067] S34, inputting the meta-feature vector into a multi-layer perceptron to obtain hierarchical configuration parameters;

[0068] S35. Collect positive samples and negative samples based on the layered configuration parameters;

[0069] S36, extracting the positive sample representation corresponding to the positive sample and the negative sample representation corresponding to the negative sample;

[0070] S37. Calculate the contrast loss function based on the positive sample representation and the negative sample representation;

[0071] S38, converting the natural language feedback of the user feedback dataset into a structured rule set;

[0072] S39. Based on the structured rule set, construct a rule alignment loss function;

[0073] S310, constructing a meta-learning loss function based on the sample task dataset;

[0074] S311. Calculate a total loss function based on the contrast loss function, the rule alignment loss function, and the meta-learning loss function;

[0075] S312. Based on the total loss function, update the model parameters and policy network parameters of the dynamic hierarchical sparse attention model to generate a final dynamic hierarchical sparse attention model;

[0076] In S31, the expression for mapping the natural language task description of the sample task dataset into a vector is:

[0077] e t =Embed(t i );

[0078] Where, e t represents the vector corresponding to the natural language task description, Embed(·) represents the function of embedding the input text through the encoder, t iRepresents a natural language task description;

[0079] In S32, the expression for encoding the unit test set of the sample task dataset into a vector is:

[0080] e u =Embed(u i );

[0081] Where, e u Represents the vector corresponding to the unit test set, u i Represents a unit test suite;

[0082] In S33, the element feature vector m t The expression is:

[0083] m t =GNN(e t ,e u );

[0084] Where GNN(·) represents graph neural network;

[0085] In S34, the expression of the hierarchical configuration parameter is:

[0086] {α,k,γ}=g φ (m t );

[0087] Where g φ ((·)) represents a multilayer perceptron, α represents a learnable parameter, γ represents the bridge layer gating weight, and k represents the low-rank projection dimension size in the global layer attention;

[0088] In S36, the positive sample represents z + The expression is:

[0089] z + =f θ (x + );

[0090] Where x + represents a positive sample, f θ (·) indicates encoder;

[0091] In S36, negative samples represent z - The expression is:

[0092] z - =f θ (x - );

[0093] Where x - represents negative samples;

[0094] In S37, the contrast loss function The expression is:

[0095]

[0096] Where log(·) represents the logarithmic function, e represents the exponent, and τ represents the temperature coefficient;

[0097] In S39, the rule alignment loss function The expression is:

[0098]

[0099] Where, represents the structured rule set, x′ represents the generated code of the current step, r represents the matching rule, CE(·) represents the cross entropy loss function, π θ (x′|t) represents the output probability distribution of the generated code for the current step under the model parameters θ under the condition of task t, and Match(·) represents the rule matching function;

[0100] In S310, the meta-learning loss function L meta The expression is:

[0101]

[0102] Where, Denotes the distribution D of the few-sample task few The task t extracted from the task is expected, KL(·) represents the KL divergence, represents the optimal strategy under task t, x represents the input of the model, and D few represents a sample task dataset;

[0103] In S311, the total loss function The expression is:

[0104]

[0105] Where λ1 represents the weight coefficient of the contrast loss function, and λ2 represents the weight coefficient of the rule alignment loss function.

[0106] The beneficial effects of the present invention are:

[0107] (1) The present invention significantly improves the computational efficiency and resource utilization of long code generation tasks through the collaborative design of dynamic hierarchical sparse attention mechanism and computing resource optimization strategy. The hierarchical architecture of the present invention fully adapts to the multi-scale characteristics of code semantics and differentially allocates attention resources according to functional requirements of different levels: shallow focus on the precise capture of fine-grained grammatical structure, deep strengthening of efficient modeling of global logical dependencies, and the organic integration of cross-level information through the dynamic fusion mechanism of the bridge layer; low-rank approximate calculation effectively compresses the dimension of the attention matrix, while ensuring the integrity of key semantic information and greatly reducing the computational complexity, solving the memory bottleneck problem caused by the explicit calculation of the complete attention matrix in traditional dynamic sparse methods; on the basis of maintaining the logical coherence of the generated code, the present invention significantly reduces the video memory usage and improves the decoding speed, providing reliable support for real-time reasoning in industrial-grade high-concurrency scenarios;

[0108] (2) The present invention uses task meta-feature extraction and parameterized strategy networks to quickly extract the essential laws of tasks from limited samples, dynamically generate adaptive sparse configuration rules, achieve rapid generalization of new tasks, strengthen the model's deep understanding of code structure and logical constraints, reduce semantic deviations caused by sample scarcity, and significantly improve error correction efficiency and generation accuracy in low-resource scenarios; the present invention enables the model to maintain high-quality code generation capabilities at extremely low annotation costs, providing a flexible and efficient solution for agile development and automated testing. BRIEF DESCRIPTION OF THE DRAWINGS

[0109] Figure 1 Flowchart of the code generation method based on dynamic hierarchical sparse attention. DETAILED DESCRIPTION

[0110] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0111] like Figure 1 As shown, the present invention provides a code generation method based on dynamic hierarchical sparse attention, comprising the following steps:

[0112] S1, collect input data set;

[0113] S2. Build a dynamic hierarchical sparse attention model;

[0114] S3. Optimize the dynamic hierarchical sparse attention model using the input data set to generate the final dynamic hierarchical sparse attention model;

[0115] S4. Generate code using the final dynamic hierarchical sparse attention model.

[0116] In this embodiment of the present invention, in S1, the input dataset includes a sample task dataset, a long context calibration dataset, and a user feedback dataset;

[0117] Sample task dataset D few The expression is:

[0118] D few ={(t i ,u i )};

[0119] Where, t i represents the natural language task description, u i Represents a unit test suite;

[0120] User feedback dataset D fb The expression is:

[0121] D fb ={(x′ j ,f j )};

[0122] Where x′ j represents the jth generated code in the user feedback dataset, f j Indicates natural language feedback.

[0123] In this embodiment of the present invention, S2 includes the following sub-steps:

[0124] S21. Dynamically expand the sliding window using the local layer of the dynamic hierarchical sparse attention model;

[0125] S22. Use the global layer of the dynamic hierarchical sparse attention model to perform a low-rank approximation of the attention matrix;

[0126] S23. Use the bridge layer of the dynamic hierarchical sparse attention model to splice the local layer output and the global layer output to complete the construction of the dynamic hierarchical sparse attention model.

[0127] In this embodiment of the present invention, S21 includes the following sub-steps:

[0128] S211. Calculate the sliding window span according to the length of the input data set;

[0129] S212. Calculate the query vector and the key vector;

[0130] S213. Calculate the attention weight according to the sliding window span, the query vector, and the key vector;

[0131] S214. Based on the attention weight of the local layer, complete the dynamic expansion of the sliding window.

[0132] In the embodiment of the present invention, in S211, the sliding window span S local The calculation formula is:

[0133]

[0134] Where α represents the learnable parameter, N represents the length of the input dataset, log(·) represents the logarithmic function, Indicates rounding down;

[0135] In S212, the query vector Q i The expression is:

[0136] Q i =W q h i ;

[0137] Where W q represents the learnable weight matrix that maps the hidden state of position i to the query vector space, h i represents the hidden state of position i;

[0138] In S212, the key vector K j The expression is:

[0139] K j =W k h j ;

[0140] Where W k represents the learnable weight matrix used to transform the hidden state at position j into the key vector, h j represents the hidden state at position j;

[0141] In S213, the attention weight A local The expression of (i,j) is:

[0142]

[0143] Where, d k represents the dimension size used to scale the dot product result, T represents transpose, and Softmax(·) represents the activation function.

[0144] In this embodiment of the present invention, S22 includes the following sub-steps:

[0145] S221, calculating the global layer bond matrix;

[0146] S222, calculating the global layer value matrix;

[0147] S223. Based on the global layer key matrix and the global layer value matrix, the global layer low-rank attention matrix is ​​calculated using the low-rank approximation method.

[0148] In the embodiment of the present invention, in S221, the expression of the global layer key matrix U is:

[0149] U=W u K;

[0150] Where W u represents the first learnable parameter matrix, K represents the key matrix of the intermediate layer;

[0151] In S222, the global layer value matrix V proj The expression is:

[0152] V proj =W v V;

[0153] Where W v represents the second learnable parameter matrix, and V represents the value matrix of the intermediate layer;

[0154] In S223, the global layer low-rank attention matrix A global The expression is:

[0155]

[0156] Where Q represents the query matrix, T represents the transpose, Softmax(·) represents the activation function, and d k Indicates the dimension size used to scale the dot product result.

[0157] In this embodiment of the present invention, S23 includes the following sub-steps:

[0158] S231, using a bridge layer to splice the local layer output and the global layer output;

[0159] S232. Calculate the bridge layer gating weight according to the splicing result;

[0160] S233. According to the gating weight of the bridge layer, the output of the bridge layer is obtained to complete the construction of the dynamic hierarchical sparse attention model.

[0161] In this embodiment of the present invention, in S232, the calculation formula of the bridge layer gating weight γ is:

[0162] γ=σ(W g [h local ;h global ]);

[0163] Where h local represents the local layer output, h global represents the global layer output, σ(·) represents the Sigmoid function, W g represents the third learnable parameter matrix;

[0164] In S233, the output h of the bridge layer bridge The expression is:

[0165] h bridge =γ·h local +(1-γ)·h global ;

[0166] Where γ represents the gating weight of the bridge layer.

[0167] In this embodiment of the present invention, S3 includes the following sub-steps:

[0168] S31, mapping the natural language task description of the sample task dataset into a vector;

[0169] S32, encoding the unit test set of the sample task data set into a vector;

[0170] S33. Input the vector corresponding to the natural language task description and the vector corresponding to the unit test set into the graph neural network to generate a meta-feature vector;

[0171] S34, inputting the meta-feature vector into a multi-layer perceptron to obtain hierarchical configuration parameters;

[0172] S35. Collect positive samples and negative samples based on the layered configuration parameters;

[0173] S36, extracting the positive sample representation corresponding to the positive sample and the negative sample representation corresponding to the negative sample;

[0174] S37. Calculate the contrast loss function based on the positive sample representation and the negative sample representation;

[0175] S38, converting the natural language feedback of the user feedback dataset into a structured rule set;

[0176] S39. Based on the structured rule set, construct a rule alignment loss function;

[0177] S310, constructing a meta-learning loss function based on the sample task dataset;

[0178] S311. Calculate a total loss function based on the contrast loss function, the rule alignment loss function, and the meta-learning loss function;

[0179] S312. Based on the total loss function, update the model parameters and policy network parameters of the dynamic hierarchical sparse attention model to generate a final dynamic hierarchical sparse attention model;

[0180] In S31, the expression for mapping the natural language task description of the sample task dataset into a vector is:

[0181] e t =Embed(t i );

[0182] Where, et represents the vector corresponding to the natural language task description, Embed(·) represents the function of embedding the input text through the encoder, t i Represents a natural language task description;

[0183] In S32, the expression for encoding the unit test set of the sample task dataset into a vector is:

[0184] e u =Embed(u i );

[0185] Where, e u Represents the vector corresponding to the unit test set, u i Represents a unit test suite;

[0186] In S33, the element feature vector m t The expression is:

[0187] m t =GNN(e t ,e u );

[0188] Where GNN(·) represents graph neural network;

[0189] In S34, the expression of the hierarchical configuration parameter is:

[0190] {α,k,γ}=g φ (m t );

[0191] Where g φ ((·)) represents a multilayer perceptron, α represents a learnable parameter, γ represents the bridge layer gating weight, and k represents the low-rank projection dimension size in the global layer attention;

[0192] In S36, the positive sample represents z + The expression is:

[0193] z + =f θ (x + );

[0194] Where x + represents a positive sample, f θ (·) indicates encoder;

[0195] In S36, negative samples represent z - The expression is:

[0196] z - =f θ (x - );

[0197] Where x - represents negative samples;

[0198] In S37, the contrast loss function The expression is:

[0199]

[0200] Where log(·) represents the logarithmic function, e represents the exponent, and τ represents the temperature coefficient;

[0201] In S39, the rule alignment loss function The expression is:

[0202]

[0203] Where, represents the structured rule set, x′ represents the generated code of the current step, r represents the matching rule, CE(·) represents the cross entropy loss function, π θ (x′|t) represents the output probability distribution of the generated code for the current step under the model parameters θ under the condition of task t, and Match(·) represents the rule matching function;

[0204] In S310, the meta-learning loss function L meta The expression is:

[0205]

[0206] Where, Denotes the distribution D of the few-sample task few The task t extracted from the task is expected, KL(·) represents the KL divergence, represents the optimal strategy under task t, x represents the input of the model, and D few represents a sample task dataset;

[0207] In S311, the total loss function The expression is:

[0208]

[0209] Where λ1 represents the weight coefficient of the contrast loss function, and λ2 represents the weight coefficient of the rule alignment loss function.

[0210] The core innovation of this paper lies in achieving precise adaptation of computing resources to the multi-scale requirements of code generation tasks through a functional layered dynamic attention mechanism. The model layers are divided into local, global, and bridge layers, and each layer adopts a differentiated sparse strategy.

[0211] The local layer dynamically expands the span through a progressive sliding window to ensure fine-grained capture of shallow grammar while avoiding the truncation of long-range grammar by fixed windows.

[0212] The global layer introduces low-rank approximate calculations to decompose the traditional attention matrix into a low-dimensional factor matrix, greatly reducing the computational complexity and solving the memory explosion problem in the pre-filling stage of dynamic pruning technology.

[0213] The bridge layer fuses local and global information through learnable gated weights, dynamically balancing the contribution of outputs from different layers to prevent semantic discontinuity and optimize logical coherence. Key protections in this architecture lie in the layered partitioning rules, the dynamic dimensionality adaptation algorithm for low-rank projection, and the joint optimization mechanism of gated fusion. These ensure that while reducing video memory usage, the generated code maintains syntactic accuracy and logical integrity, creating a technical barrier to existing static or single dynamic sparse solutions.

[0214] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A code generation method based on dynamic hierarchical sparse attention, characterized in that The following steps are involved: S1, collect input data set; S2. Build a dynamic hierarchical sparse attention model; S3. Optimize the dynamic hierarchical sparse attention model using the input data set to generate the final dynamic hierarchical sparse attention model; S4. Generate code using the final dynamic hierarchical sparse attention model.

2. The code generation method based on dynamic hierarchical sparse attention according to claim 1, characterized in that In S1, the input dataset includes a sample task dataset, a long context calibration dataset, and a user feedback dataset; The sample task dataset D few The expression is: D few ={(t i ,u i )}; Where, t i represents the natural language task description, u i Represents a unit test suite; The user feedback dataset D fb The expression is: D fb ={(x′ j ,f j )}; Where x′ j represents the jth generated code in the user feedback dataset, f j Indicates natural language feedback.

3. The code generation method based on dynamic hierarchical sparse attention according to claim 1, characterized in that The S2 includes the following sub-steps: S21. Dynamically expand the sliding window using the local layer of the dynamic hierarchical sparse attention model; S22. Use the global layer of the dynamic hierarchical sparse attention model to perform a low-rank approximation of the attention matrix; S23. Use the bridge layer of the dynamic hierarchical sparse attention model to splice the local layer output and the global layer output to complete the construction of the dynamic hierarchical sparse attention model.

4. The code generation method based on dynamic hierarchical sparse attention according to claim 3 is characterized in that The S21 includes the following sub-steps: S211. Calculate the sliding window span according to the length of the input data set; S212. Calculate the query vector and the key vector; S213. Calculate the attention weight according to the sliding window span, the query vector, and the key vector; S214. Based on the attention weight of the local layer, complete the dynamic expansion of the sliding window.

5. The code generation method based on dynamic hierarchical sparse attention according to claim 4 is characterized in that In S211, the sliding window span S local The calculation formula is: Where α represents the learnable parameter, N represents the length of the input dataset, log(·) represents the logarithmic function, Indicates rounding down; In S212, the query vector Q i The expression is: Q i =W q h i ; Where W q represents the learnable weight matrix that maps the hidden state of position i to the query vector space, h i represents the hidden state of position i; In S212, the key vector K j The expression is: K j =W k h j ; Where W k represents the learnable weight matrix used to transform the hidden state at position j into the key vector, h j represents the hidden state at position j; In S213, the attention weight A local The expression of (i,j) is: Where, d k represents the dimension size used to scale the dot product result, T represents transpose, and Softmax(·) represents the activation function.

6. The code generation method based on dynamic hierarchical sparse attention according to claim 3, characterized in that The S22 includes the following sub-steps: S221, calculating the global layer bond matrix; S222, calculating the global layer value matrix; S223. Based on the global layer key matrix and the global layer value matrix, the global layer low-rank attention matrix is ​​calculated using the low-rank approximation method.

7. The code generation method based on dynamic hierarchical sparse attention according to claim 6, characterized in that In S221, the expression of the global layer bond matrix U is: U=W u K; Where W u represents the first learnable parameter matrix, K represents the key matrix of the intermediate layer; In S222, the global layer value matrix V proj The expression is: V proj =W v V; Where W v represents the second learnable parameter matrix, and V represents the value matrix of the intermediate layer; In S223, the global layer low-rank attention matrix A global The expression is: Where Q represents the query matrix, T represents the transpose, Softmax(·) represents the activation function, and d k Indicates the dimension size used to scale the dot product result.

8. The code generation method based on dynamic hierarchical sparse attention according to claim 3, characterized in that The S23 includes the following sub-steps: S231, using a bridge layer to splice the local layer output and the global layer output; S232. Calculate the bridge layer gating weight according to the splicing result; S233. According to the gating weight of the bridge layer, the output of the bridge layer is obtained to complete the construction of the dynamic hierarchical sparse attention model.

9. The code generation method based on dynamic hierarchical sparse attention according to claim 8, characterized in that In the above S232, the calculation formula of the bridge layer gating weight γ is: γ=σ(W g [h local ;h global ]); Where h local represents the local layer output, h global represents the global layer output, σ(·) represents the Sigmoid function, W g represents the third learnable parameter matrix; In S233, the output h of the bridge layer bridge The expression is: h bridge =γ·h local +(1-γ)·h global ; Where γ represents the gating weight of the bridge layer.

10. The code generation method based on dynamic hierarchical sparse attention according to claim 1, characterized in that The S3 includes the following sub-steps: S31, mapping the natural language task description of the sample task dataset into a vector; S32, encoding the unit test set of the sample task data set into a vector; S33. Input the vector corresponding to the natural language task description and the vector corresponding to the unit test set into the graph neural network to generate a meta-feature vector; S34, inputting the meta-feature vector into a multi-layer perceptron to obtain hierarchical configuration parameters; S35. Collect positive samples and negative samples based on the layered configuration parameters; S36, extracting the positive sample representation corresponding to the positive sample and the negative sample representation corresponding to the negative sample; S37. Calculate the contrast loss function based on the positive sample representation and the negative sample representation; S38, converting the natural language feedback of the user feedback dataset into a structured rule set; S39. Based on the structured rule set, construct a rule alignment loss function; S310, constructing a meta-learning loss function based on the sample task dataset; S311. Calculate a total loss function based on the contrast loss function, the rule alignment loss function, and the meta-learning loss function; S312. Based on the total loss function, update the model parameters and policy network parameters of the dynamic hierarchical sparse attention model to generate a final dynamic hierarchical sparse attention model; In S31, the natural language task description of the sample task dataset is mapped into a vector expression: e t =Embed(t i ); Where, e t represents the vector corresponding to the natural language task description, Embed(·) represents the function of embedding the input text through the encoder, t i Represents a natural language task description; In S32, the expression for encoding the unit test set of the sample task data set into a vector is: e u =Embed(u i ); Where, e u Represents the vector corresponding to the unit test set, u i Represents a unit test suite; In S33, the element feature vector m t The expression is: m t =GNN(e t ,he u ); Where GNN(·) represents graph neural network; In the above S34, the expression of the hierarchical configuration parameter is: {a,k,g}=g φ (m t ); Where g φ ((·)) represents a multilayer perceptron, α represents a learnable parameter, γ represents the bridge layer gating weight, and k represents the low-rank projection dimension size in the global layer attention; In S36, the positive sample represents z + The expression is: z + =f θ (x + ); Where x + represents a positive sample, f θ (·) indicates encoder; In S36, the negative sample represents z - The expression is: z - =f θ (x - ); Where x - represents negative samples; In S37, the contrast loss function The expression is: Where log(·) represents the logarithmic function, e represents the exponent, and τ represents the temperature coefficient; In S39, the rule alignment loss function The expression is: Where, Represents a structured rule set, x ′ represents the code generated at the current step, r represents the matching rule, CE(·) represents the cross entropy loss function, π θ (x′|t) represents the output probability distribution of the generated code for the current step under the model parameters θ under the condition of task t, and Match(·) represents the rule matching function; In S310, the meta-learning loss function L meta The expression is: Where, Denotes the distribution D of the few-sample task few The task t extracted from the task is expected, KL(·) represents the KL divergence, represents the optimal strategy under task t, x represents the input of the model, and D few represents a sample task dataset; In S311, the total loss function The expression is: Where λ1 represents the weight coefficient of the contrast loss function, and λ2 represents the weight coefficient of the rule alignment loss function.

Images