Large model code generation method based on multi-feature improved Ptuning-V2

Abstract

The invention discloses a large model code generation method based on multi-feature improved Ptuning-V2, which comprises the following steps that: aiming at a source code, multi-source features are extracted and fused from three dimensions of text, consumption and image, and the text features comprise code vectors, grammar structures and execution process features; the consumption features comprise code annotation, time and space complexity features; and the image features are used for processing a structure chart and a flow chart of the code by utilizing a visual converter. And injecting final semantic features generated after weighted fusion into a P-tuning V2 fine tuning algorithm in each layer of the base large model, so that the final semantic features, learnable prompt vectors and an input sequence jointly participate in self-attention calculation. According to the method, rich priori knowledge is provided for the prompt vector by deeply fusing the multi-dimensional characteristics of the code, the understanding ability of the model for the code structure, logic and performance is remarkably enhanced, and the accuracy, logic completeness and practicability of the generated code are effectively improved while the advantage of efficient fine tuning of parameters is kept.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for generating large model code based on multi-feature improvement of Ptuning-V2. Background Technology

[0002] In recent years, pre-trained large language models (LLMs) based on the Transformer architecture have made groundbreaking progress in the field of natural language processing and have been successfully applied to intelligent code generation. Models such as Codex and CodeGen, through self-supervised learning on massive code corpora, can understand natural language requirements and generate program code corresponding to functions, greatly improving the efficiency of software development. To enhance the adaptability of large models to specific downstream tasks and reduce the high training costs, parameter-efficient fine-tuning (PEFT) techniques have emerged. Among them, P-tuning V2, by injecting learnable continuous prompts into each layer of the model, effectively guides the model's behavior while freezing most of the model parameters, becoming a highly regarded fine-tuning strategy.

[0003] However, when applied to code generation tasks, the standard P-tuning V2 method relies primarily on the general semantic information of the input for its suggestion vector optimization, failing to fully utilize the multi-dimensional features inherent in code as a formal language. Specifically, existing methods often treat code as a plain text sequence, ignoring its inherent Abstract Syntax Tree (AST) information, which accurately reflects the program structure, and the Control Flow Graph (CFG) information, which represents the program's execution logic. Furthermore, the functional descriptions contained in code comments, as well as key non-functional attributes such as time and space complexity that measure code performance, are not effectively integrated into the suggestion optimization process. This neglect of the multi-dimensional features of code limits the representational power of the suggestion vectors, making it difficult to provide precise and in-depth guidance to the model. The generated code may be syntactically correct, but it may lack logical completeness, runtime efficiency, or suitability for complex requirements, thus limiting its application potential in high-quality, high-reliability software development scenarios. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0005] In view of the aforementioned existing problems, this invention is proposed. Therefore, this invention provides a large model code generation method based on multi-feature improvements to Ptuning-V2 to address the problems mentioned in the background art.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a large model code generation method based on multi-feature improvement of Ptuning-V2, comprising: For the source code, extract and fuse its code vector features, syntax structure features, and execution flow features to generate the final text features; For the source code, extract and fuse its code comment vector features, time complexity vector features, and space complexity vector features to generate the final consumption features; For the source code, extract and fuse its graphical syntactic structure features and graphical execution flow features to generate the final image features; The final text features, final consumption features, and final image features are fused to generate the final semantic features; The final semantic features are incorporated into the Ptuning-V2 fine-tuning algorithm to construct an improved large model fine-tuning algorithm; Based on the improved large model fine-tuning algorithm and the base large model, a code generation large model is obtained by training using the source code dataset.

[0007] As a preferred embodiment of the large model code generation method based on multi-feature improved Ptuning-V2 described in this invention, wherein: the generation of final text features includes: The code vector features, syntax structure features, and execution flow features are weighted and fused together; The weighted fusion involves concatenating the three feature vectors and determining the weights of each feature vector through model learning to generate the final text features.

[0008] As a preferred embodiment of the large model code generation method based on multi-feature improved Ptuning-V2 described in this invention, the extraction of code vector features, syntax structure features, and execution flow features includes: The code vector features are extracted by inputting the source code into a preset vectorization model; The grammatical structure features are obtained by constructing an abstract syntax tree of the source code and processing the abstract syntax tree using a child node summation tree long short-term memory network. The processing of the child node summation tree long short-term memory network includes: recursive calculation from the leaf node to the root node, wherein the hidden state of each parent node is based on the hidden states of all its child nodes and is generated by controlling the information flow through the input gate, forget gate and output gate. The execution flow features are obtained by constructing the execution flow graph of the source code and processing the execution flow graph using a graph convolutional network. The processing using the graph convolutional network includes: inputting the adjacency matrix and node feature matrix of the execution flow graph into the graph convolutional network, aggregating the neighborhood node information through multi-layer graph convolution operations, and generating execution flow features representing the entire graph structure.

[0009] As a preferred embodiment of the large model code generation method based on multi-feature improved Ptuning-V2 described in this invention, wherein: the generation of the final consumed features includes: The code comment vector features, time complexity vector features, and space complexity vector features are weighted and fused together; The weighted fusion involves concatenating the three feature vectors, determining the weights of each feature vector through model learning, and generating the final consumed feature.

[0010] As a preferred embodiment of the large model code generation method based on multi-feature improved Ptuning-V2 described in this invention, the extraction methods of the code annotation vector features, time complexity vector features, and space complexity vector features include: The code annotation vector feature is obtained by segmenting the annotation text of the source code into sentences, inputting each sentence into a vectorization model to obtain sentence vectors, and then concatenating all sentence vectors. The time complexity vector feature and space complexity vector feature are obtained by calling a preset large language model to analyze the source code to predict its time complexity and space complexity respectively, and then mapping the predicted complexity results to the corresponding preset embedding vector.

[0011] As a preferred embodiment of the large model code generation method based on multi-feature improved Ptuning-V2 described in this invention, the method further includes a step of fine-tuning the vectorized model before extracting the code annotation vector features. This step includes: Using paired data consisting of source code text and corresponding comment content; The similarity between code and annotation is calculated using cosine similarity, and the result is used as the training target to fine-tune the vectorized model.

[0012] As a preferred embodiment of the large model code generation method based on multi-feature improved Ptuning-V2 described in this invention, wherein: the generation of final image features includes: The graphical syntactic structure features and graphical execution flow features of the code are weighted and fused. The weighted fusion involves concatenating the vectors representing the image-based grammatical structure features and the image-based execution flow features, and then determining the weights corresponding to each feature vector through model learning to generate the final image features.

[0013] As a preferred embodiment of the large model code generation method based on multi-feature improved Ptuning-V2 described in this invention, the extraction methods of the graphical syntactic structure features and graphical execution flow features include: The feature vectors of the syntax tree nodes and the feature vectors of the execution flow graph nodes extracted from the source code are organized into two-dimensional matrices to form the image syntax tree matrix and the image execution flow matrix. The image syntax tree matrix and the image execution flow matrix are respectively input into the visual transformer model, and processed by the multi-head self-attention mechanism and feedforward neural network layer inside the model to generate the image-based syntax structure features and image-based execution flow features respectively.

[0014] As a preferred embodiment of the large model code generation method based on multi-feature improved Ptuning-V2 described in this invention, wherein: the generation of final semantic features includes: The final text features, final consumption features, and final image features are weighted and concatenated to form a fused feature vector; A fully connected layer is used to reduce the dimensionality of the fused feature vector to generate the final semantic features.

[0015] As a preferred embodiment of the large model code generation method based on multi-feature improved Ptuning-V2 described in this invention, wherein: the final semantic features are incorporated into the Ptuning-V2 fine-tuning algorithm, including: In each layer of the base model, the learnable cue vector, the embedding vector of the input sequence, and the final semantic features of that layer are concatenated to form a new combined input vector. The combined input vector is fed into the self-attention mechanism of this layer for calculation, thus completing the fine-tuning.

[0016] Compared with existing technologies, the beneficial effects of the invention are: 1. This invention provides a comprehensive analysis of source code from three dimensions: text, computation, and image. It not only extracts traditional code vectors but also delves into the Abstract Syntax Tree (AST) reflecting program structure and the Execution Flow Graph (CFG) embodying operational logic. Furthermore, it is the first to integrate non-functional attributes such as time / space complexity and code comments, as well as the visual representation of code structure (processed through Vision Transformer), into the feature system. This multi-dimensional, multi-modal feature fusion approach constructs an unprecedentedly comprehensive and profound semantic understanding of the code, overcoming the information loss problem caused by existing technologies that treat code merely as a pure text sequence.

[0017] 2. This invention injects the fused "final semantic features" as strong prior knowledge into every layer of the P-tuning V2 framework. This allows the learnable prompt vectors to no longer blindly learn from the general context, but rather to be optimized under the precise guidance of multiple information such as code structure, logic, and performance. This greatly enhances the representational power and guidance accuracy of the prompt vectors, enabling the model to more deeply understand complex generation tasks and effectively solving the problem of insufficient guidance capability in the code generation domain of standard P-tuning V2.

[0018] 3. Because the model of this invention fully learns the internal structure, execution logic, and performance consumption of the code during the fine-tuning stage, the generated code not only has higher accuracy at the syntactic level (as shown by the BLEU metric), but also performs better in terms of logical completeness, structural rationality, and semantic consistency (as shown by the CodeBLEU metric). In particular, the consideration of time / space complexity enables the model to generate code that better meets the performance requirements of actual engineering, significantly improving the overall quality of the generated results and their practical value in real-world development scenarios.

[0019] 4. Furthermore, this invention improves upon the P-tuning V2 framework, a highly efficient parameter fine-tuning framework. While freezing most of the base model parameters, it empowers a small number of trainable cue parameters by introducing highly informative multi-dimensional features. This achieves code generation performance surpassing many advanced models without significantly increasing training costs and computational resource consumption, achieving an ideal balance between model performance and training efficiency. Attached Figure Description

[0020] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein: Figure 1This is a flowchart illustrating the overall process of a large model code generation method based on multi-feature improved Ptuning-V2 according to an embodiment of the present invention. Detailed Implementation

[0021] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0022] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0023] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0024] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0025] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0026] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0027] Example 1 Reference Figure 1 This is the first embodiment of the present invention, which provides a method for generating large model code based on multi-feature improved Ptuning-V2, including: S1. For the source code, extract and merge its code vector features, syntax structure features, and execution flow features to generate the final text features.

[0028] Furthermore, regarding the source code set Extract the corresponding code vector features from each code segment. syntactic structural features Execution process characteristics The above features are then weighted, learned, and fused to obtain the final text features for each code. .

[0029] Furthermore, obtain the vector features corresponding to the source code. The source code collection Each piece of code text is input into the Bge-m3 model, which extracts the vector features of the code. .

[0030] Furthermore, obtain the corresponding syntactic structure features of the source code. The steps to obtain it are as follows: S101.1 Construct source code syntax structure tree data (built from each word node) using an abstract syntax tree. Then, use the FastText model to encode word vectors for each node (index: 1~K) of the syntax structure tree to obtain the word vector features of each word node. The final grammatical structure features are obtained through Child-Sum Tree-LSTM; the grammatical structure tree data is as follows: in, , This represents the final syntactic structure feature node (i.e., the root node). Represents any word node, and The set of child nodes of a node is: in, , The The child node is , No. Nodes The child node is .

[0031] S101.2, For any node in the set Calculate all nodes in its child node set, i.e., the input gate feature. Features of the forgetting gate and output gate features .

[0032] Specifically, input gate features The calculation formula is as follows: in, For the first The offset top of the node input gate. For the first Each child node of a node The weights of the input gates, To sum the hidden states of all child nodes, It is the Sigmoid activation function. For the first Each child node of a node The weights of the hidden states, For the current number Node input gate features.

[0033] Specifically, forgetting gate characteristics The calculation formula is as follows: in, For the first The offset top of the node forget gate, For the first Each child node of a node The weights of the input gates, For the first Each child node of a node The hidden state, It is the Sigmoid activation function. For the first Each child node of a node The weights of the hidden states, For the current number Node input gate features.

[0034] Specifically, output gate The calculation formula is as follows: in, For the first The offset top of the node output gate For the first Each child node of a node The weights of the output gates To sum the hidden states of all child nodes, It is the Sigmoid activation function. For the first Each child node of a node The weights of the hidden states, For the current number Node input gate features.

[0035] S101.3, For any node in the set The input gate features of the child nodes of the current parent node are calculated according to S101.2. Features of the forgetting gate Based on the output gate features, calculate the candidate memory unit features of the current parent node. and memory unit characteristics .

[0036] Specifically, candidate memory unit features The calculation formula is as follows: in, The output range is [-1, 1]. For the first The bias top of the candidate memory cell of the node. For the first Each child node of a node The weights of candidate memory units, It is the Sigmoid activation function. For the first Each child node of a node The weights of candidate memory units, For the current number Features of candidate memory units in nodes.

[0037] Specifically, characteristics of memory units The calculation formula is as follows: in, This indicates that the information of the current node has been updated. A single cell represents the memory information of all its child nodes. Through their respective forgetting gates Update the information.

[0038] S101.4, Based on the obtained output gate and characteristics of memory units Calculate the first Hidden features of nodes The specific formula is as follows: in, This represents the hidden features of the current node. The output gate feature of the current node, This represents the cellular unit characteristics of the current node; S101.5. Based on the aforementioned syntax tree structure, starting from the first... The recursive calculation begins with the leaf node, and its features are calculated in steps S101.2 and S101.3. Features of the forgetting gate Output gate features Candidate memory unit characteristics and memory unit characteristics According to step S101.4, the hidden layer features of the current node are calculated. Repeat steps S101.2, S101.3, and S101.4 sequentially to obtain each node. Hidden features Hidden features of the root node This represents the syntactic structural features of the entire source code. The calculation expression is as follows: Furthermore, obtain the execution flow characteristics corresponding to the source code. The steps to obtain it are as follows: S102.1 Construct the execution flow graph structure data corresponding to the source code based on the execution flow and logical relationship of the source code. Construct the source code syntax structure tree data (built from each word node) using an abstract syntax tree. Use the FastText model to encode word vectors for each node (index: 1~K) of the syntax structure tree to obtain the word vector features of each word node. The final syntactic structure features are obtained through GCN, where the execution flow graph structure data is as follows: in, Let be the set of nodes (i.e., vertices) for each word in the graph. For each set of edges in the graph (i.e., the edges between vertices), the expression is as follows: in, , Represents any word node (i.e., vertex). , Represents any edge; S102.2, will Word vector features of each vertex Build as a dimensional matrix The relationship between edges between vertices Form a adjacency matrix ; S102.3, Modeling is performed using a Graph Neural Network (GCN) model. and Input the model architecture group, and obtain the results through model training and iterative computation. Features after convolution The calculation expression is as follows: Wherein, the initial input is , , It is the identity matrix. yes The degree matrix, For the features of each layer, For ReLU functions; for conduct The vector obtained after computation by the graph neural network; For the first The weight matrix is ​​calculated in this step; S102.4. Based on step S102.3, use the GCN neural network model to calculate the entire execution flow graph structure data. ,pass Learning from layered graph convolutional neural networks to obtain execution flow features corresponding to the source code. The calculation expression is as follows: It should be noted that in vector semantic feature extraction, the Bge-m3 model is used to encode the entire source code into dense vectors, which can effectively capture the global functional semantic information of the code and overcome the limitations of traditional bag-of-words models or simple token embedding in understanding the context. Furthermore, in terms of syntactic structure feature modeling, an Abstract Syntax Tree (AST) is used to construct the hierarchical structure of the code, and FastText is used to encode the word vectors of nodes. Then, the child node information is recursively aggregated through Child-Sum Tree-LSTM, which can accurately characterize the syntactic composition and nested logic of the code and enhance the model's deep understanding of the code structure. Secondly, in execution flow feature extraction, an execution flow graph (such as a control flow graph) is constructed, and the node word vectors are combined with the graph structure relationships. Graph Convolutional Networks (GCNs) are used for multi-layer propagation and aggregation, which can effectively capture dynamic execution characteristics such as control dependencies and data flow paths in the code, enabling the model to have the ability to reason about program behavior. Finally, the three heterogeneous features are linearly concatenated, and learnable weight parameters are introduced. Through training, the contribution ratio of each feature is automatically optimized, achieving adaptive fusion of semantic, structural, and execution information to generate more representative unified code features. Compared with single-feature or simple concatenation methods, the present invention fully explores the multimodal characteristics of code, enhances the context awareness and generation quality of pre-trained models in tasks such as code completion and function generation, and provides strong content support for improving lightweight fine-tuning algorithms such as Ptuning-V2.

[0039] Furthermore, the extracted vector features syntactic structural features and execution process characteristics Linear concatenation is performed, and weight ratios are assigned between features. Through learning and fusion, the final text features are obtained. The specific formula is as follows: in, This indicates the concatenation of the above vectors. Vector features The concatenated weights are then integrated and learned to obtain the optimal weight ratio. For syntactic structural features The concatenated weights are then integrated and learned to obtain the optimal weight ratio. For execution process characteristics The concatenated weights are then integrated and learned to obtain the optimal weight ratio. This is an adjustable parameter.

[0040] S2. For the source code, extract and merge its code comment vector features, time complexity vector features, and space complexity vector features to generate the final consumption features.

[0041] Furthermore, regarding the source code set Extract the corresponding code comment vector features from each code segment. Time complexity vector features Space complexity vector characteristics The above features are weighted, learned, and fused to obtain the final consumption features of each code. .

[0042] Furthermore, obtain the annotation vector features corresponding to the source code. The process is as follows: S201.1, Collect the comment text of the source code. Each code segment in the code is divided into sentences by periods, and each sentence is a collection of sentences in the code segment. S201.2. Input the SenCollection of sentences from each text segment into the Bge-m3 vectorization model, and obtain the text vector features of each sentence through the Bge-m3 model. .

[0043] S201.3. Linearly concatenate the vector features of each sentence to obtain the vector features of each text segment, which are the annotation vector features corresponding to the source code. The calculation expression is as follows: Furthermore, obtain the time complexity vector features corresponding to the source code. The process is as follows: S202.1, Call the Qianwen 3-32B large model, input the source code into the large model, and use specific prompts to guide the large model to calculate the time complexity of the source code. .

[0044] S202.2, List all time complexities as follows: Number each time complexity category in Otimeset as {0, 2, 4, 6, 8, 10, 5, 9}, and initialize a vector table (i.e., an embedding matrix), with each row corresponding to a complexity and each row being an embedding vector.

[0045] S202.3. Input a time complexity, look up the vector in the table, input the vector into the neural network for learning and optimization, and obtain the vector features of each source code, i.e., the time complexity vector features. .

[0046] Furthermore, obtain the space complexity vector features corresponding to the source code. The process is as follows: S203.1, Call the Qianwen 3-32B large model, input the source code into the large model, and use specific prompts to guide the large model to calculate the source code. The time complexity is... ; S203.2 List all time complexity sets, number each time complexity category in Oplaceset as {0, 2, 4, 6, 8, 10, 5, 9}, initialize the vector table (i.e. the embedding matrix), with each row corresponding to a complexity and each row being an embedding vector; S203.3. Input a time complexity, look up the vector in the table, input the vector into the neural network for learning and optimization, and obtain the vector features of each source code, i.e., the time complexity vector features. .

[0047] Furthermore, in step S202.2, the sentence set SenCollection of each text segment is input into the Bge-m3 vectorization model, and the text vector features of each sentence are obtained through the Bge-m3 model. The processing flow is as follows: 2000 lines of source code text and comments were randomly selected. Each line of code was treated as a sentence, and comments were divided into sentences according to periods. Cosine similarity is used to calculate the result, and the result is used as training data to input into the initial Bge-m3 vectorization model for secondary fine-tuning.

[0048] Furthermore, the extracted code annotation vector features Time complexity vector features Space complexity vector characteristics Linear concatenation is performed, and weight ratios are assigned between features. Through learning and fusion, the final text features are obtained. The specific formula is as follows: in, This indicates the concatenation of the above vectors. Vector features The concatenated weights are then integrated and learned to obtain the optimal weight ratio. For syntactic structural features The concatenated weights are then integrated and learned to obtain the optimal weight ratio. For execution process characteristics The concatenated weights are then integrated and learned to obtain the optimal weight ratio. This is an adjustable parameter.

[0049] It should be noted that by integrating the semantics of code annotations, as well as time and space complexity information, to construct a performance-aware "consumption feature" representation, the model's ability to understand code efficiency can be enhanced. Its core advantages are: First, by using the Bge-m3 model to segment and vectorize annotations, and by introducing real code-annotation pairs for secondary fine-tuning, the model's ability to accurately capture the semantics of code functional descriptions is improved; second, by innovatively introducing a large model (Qianwen 3-32B) to automatically infer time and space complexity and map it into learnable embedding vectors, the model can perceive the algorithmic efficiency characteristics of the code; and third, by weighting and fusing the three types of features with learnable weights, collaborative modeling of semantic description and resource consumption information is achieved. This mechanism ensures that the generated code is not only functionally correct but also better meets performance optimization requirements, possessing greater practicality and intelligence in code generation tasks.

[0050] S3. For the source code, extract and merge its graphical syntax structure features and graphical execution flow features to generate the final image features.

[0051] Furthermore, regarding the source code set Extract the corresponding code image syntax structure from each code segment. Image execution process characteristics The above features are weighted, learned, and fused to obtain the final image features for each code. .

[0052] Furthermore, obtain the image syntax structure features corresponding to the source code. The process is as follows: S301.1, Based on syntactic structure features Obtain the syntax tree feature vector of each code word node. .

[0053] S301.2. Based on the visual image form of the tree structure, analyze all the feature vectors of the normal tree. Perform concatenation to obtain the image syntax tree image matrix. The input image syntax tree image matrix The data is divided into multiple groups and input into the Vision Transformers neural network for attention calculation, resulting in multiple attention mechanism vectors. The specific formula is as follows: in, The key after linear transformation of the input, and . The value after linear transformation of the input, and . The query after linear transformation of the input, and . The dimension of the word vector is the image syntax tree image matrix. Specifically: Among them, the image syntax tree image matrix Before being input into the Vision Transformers neural network, an image preprocessing module is used to process the data, resulting in multiple attention mechanism vectors for the image syntax tree. .

[0054] S301.3, Combine multiple attention mechanism vectors The multi-head attention mechanism vector is obtained by concatenating the vectors, and the image grammatical structure features are calculated based on the multi-head attention mechanism vector. The specific formula is as follows: in, Let be the multi-head attention mechanism vector corresponding to the image syntax tree image, and , The linear transformation matrix of the image syntax tree. To perform feedforward neural network calculations, For layer normalization processing, This is for residual join processing.

[0055] Furthermore, obtain the image execution flow characteristics corresponding to the source code. The process is as follows: S302.1, Based on execution process characteristics Obtain the execution flow feature vector of each code word node. ; S302.2. Based on the visual image form of the execution flow graph structure, analyze the feature vectors of all execution flows. Perform image stitching to obtain the image execution logic matrix. The input image execution flow matrix is ​​divided into multiple groups and fed into the Vision Transformers neural network for attention calculation, resulting in multiple attention mechanism vectors. The calculation expression is as follows: in, The key after linear transformation of the input, and . The value after linear transformation of the input, and . The query after linear transformation of the input, and .

[0056] Specifically, the image execution flow matrix Specifically: Among them, the image execution flow matrix Before being input into the Vision Transformers neural network, an image preprocessing module is used to process the image, thereby obtaining multiple sets of attention mechanism vectors for image execution flow features. ; S302.3, Combine multiple attention mechanism vectors The multi-head attention mechanism vector is obtained by concatenating the vectors. The image execution flow features are calculated using the formula. The specific formula is as follows: in, This is the multi-head attention mechanism vector corresponding to the image execution process features, and , The linear transformation matrix is ​​a feature of the image execution process. For feedforward neural network calculations, For layer normalization processing, This is for residual join processing.

[0057] Furthermore, the extracted code image syntactic structure features Image execution process characteristics Linear concatenation is performed, and weight ratios are assigned between features. Through learning and fusion, the final image features are obtained. The specific formula is as follows: in, This indicates the concatenation of the above vectors. For code image syntax structure The concatenated weights are then integrated and learned to obtain the optimal weight ratio. Image execution process features The concatenated weights are then integrated and learned to obtain the optimal weight ratio. This is an adjustable parameter.

[0058] It should be noted that the present invention significantly improves the depth and accuracy of the model's semantic understanding of source code by fusing the image syntactic structure features and image execution flow features of the code. Its core advantages are: First, by constructing a syntax tree and transforming it into an image syntax tree matrix, combined with multi-head attention computation using Vision Transformers, it effectively captures the hierarchical syntactic structure information of the code while preserving its static structural features. Second, starting from the program execution path, it constructs a visual matrix of the execution flow graph, using the same Transformer architecture to extract the control flow and data flow features during dynamic runtime, enhancing the model's ability to understand program behavior logic. Furthermore, the present invention adaptively fuses two heterogeneous features—static syntactic structure and dynamic execution flow—through linear concatenation and a learnable weighted fusion mechanism, enabling the model to automatically adjust the contribution weights of the two types of features according to task requirements, thereby obtaining more representative unified image features. Simultaneously, the entire process fully utilizes the powerful capabilities of Vision Transformers in visual semantic modeling, treating code as an "image" for processing, breaking through the limitations of traditional text sequence modeling. Meanwhile, by introducing structures such as residual connections, layer normalization, and feedforward networks, the stability and expressive power of feature extraction are ensured. This method not only enhances the ability of large models to model complex structures and logics in code generation tasks, but also improves the syntactic correctness and functional rationality of the generated code, providing high-quality input representations for efficient parameter fine-tuning of Ptuning-V2, and significantly improving the overall quality and reliability of code generation.

[0059] S4. Integrate the final text features, final consumption features, and final image features to generate the final semantic features.

[0060] Furthermore, regarding the source code set Text features corresponding to each code segment Consumption characteristics and image features We perform weighted learning and fusion to obtain the final semantic features of each code. .

[0061] S401.1, Text Features Consumption characteristics and image features The features are concatenated linearly to obtain a new feature vector. The calculation expression is as follows: in, Text features The splicing weight, Consumption characteristics Weighting Consumption characteristics Weighting This is the final semantic vector feature.

[0062] S401.2, Using a fully connected layer on the new feature vector Perform dimensionality reduction to obtain the best features.

[0063] It should be noted that by integrating three types of heterogeneous features—text, consumption, and images—a comprehensive semantic representation of the code is constructed. Learnable weights are used to weight and concatenate multimodal features, adaptively adjusting the contributions of various features. Furthermore, dimensionality reduction through fully connected layers optimizes feature representation, effectively enhancing the richness and compactness of the semantic representation and strengthening the model's understanding of code functionality and performance requirements.

[0064] S5. Integrate the final semantic features into the Ptuning-V2 fine-tuning algorithm to build an improved large model fine-tuning algorithm.

[0065] Furthermore, multiple prompt word templates are obtained, and the extracted semantic features are... By incorporating the Ptuning-V2 algorithm, a complete large-scale model fine-tuning algorithm is constructed.

[0066] Furthermore, 10 prompt templates are constructed to guide the large model in generating code. For example, the prompt templates are as follows: Template 1: "You are a large language model focused on code; please help me generate code about {input}", Template 2: "Please help me implement a piece of code with the following requirements: {input content}", Template 3: "Please help me generate the source code for {input content}", Template 4: "I have some code that I need your help to implement; the specific information is as follows: {input content}", Template 5: "Please generate the corresponding code based on {input content}", Template 6: "Now, given a piece of text, please quickly generate code that meets the requirements based on the text content; the text is: {input content}", Template 7: "What is the code generated by {input content}?", Template 8: "As a code assistant, please generate the correct source code; the requirement is: {input content}", Template 9: "Given a piece of information value and output the corresponding code; the content is as follows: {input content}", Template 10: "The code comment information is {input content}; please write the corresponding code."

[0067] Furthermore, the fused features Working together with prompt to fine-tune the model's training, the fused multi-granular features will be integrated. The embedding layers of cue vectors and input sequences are combined and participate in the subsequent self-attention mechanism. For the input sequence, For the first Layer cue vectors, This refers to the semantic feature vectors added to each layer in this paper, and each semantic vector... Embedding table of cue vectors and input sequences The following expression is calculated by concatenating the expressions: It should be noted that by constructing diverse prompt word templates and deeply integrating them with the fused semantic features into the Ptuning-V2 fine-tuning framework, the guidance and adaptability of large models in code generation tasks are significantly improved. Its core advantage lies in the design of 10 semantically rich and sentence-structure-diverse prompt templates to cover different expression habits and instruction forms, enhancing the model's robustness in understanding user intent and improving the readability and usability of the generated results. More importantly, the deep semantic features obtained through multi-stage weighted fusion are used as prior knowledge and embedded into the input representation of each layer of the model. These features, along with the prompt vector and the original input sequence, participate in self-attention calculation, achieving efficient parameter fine-tuning under semantic guidance. Furthermore, this invention not only retains the training advantages of Ptuning-V2 in low-resource scenarios but also enhances the context-awareness of prompt words through semantic enhancement mechanisms. This effectively guides the model to generate high-quality code that is grammatically correct, logically sound, and meets functional requirements, achieving synergy between external knowledge injection and internal structure optimization, significantly improving the accuracy and generalization performance of code generation.

[0068] S6. Based on the improved large model fine-tuning algorithm and the base large model, a code generation large model is obtained by training using the source code dataset.

[0069] Furthermore, the source code dataset is obtained, the data is labeled, and divided into training and testing sets. Based on the complete large model fine-tuning method and the base large model, the code generation large model is trained, and the code generation result is obtained based on the trained large model. This includes the following steps: Set the ratio of training set to test set to 7:3; Remove low-quality code (such as automatically generated code, empty functions, and test stubs), filter sensitive information (API keys, personal information in comments), standardize indentation, and remove redundant blank lines; The Qianwen 3-14B model is selected as the base model. The code input format is input(code, comment, output), where code is the code body content, comment is the corresponding comment content, and output is the expected output code. It should be noted that the present invention ensures the high quality and security of training data by partitioning the dataset and conducting rigorous data cleaning, effectively improving the model's generalization ability. Furthermore, by using the Qianwen 3-14B large model as a base and combining it with a multimodal input format that includes code, comments, and expected outputs, the model's understanding of programming semantics is enhanced, resulting in more accurate and structurally sound generated code, significantly improving the overall performance and usability of code generation.

[0070] Example 2 Referring to Table 1, the second embodiment of the present invention provides a method for generating large model code based on multi-feature improved Ptuning-V2, including: To verify the effectiveness and reliability of this invention in generating code for large models, BLEU and codeBLEU metrics were used for evaluation, and comparative experiments were conducted. The experimental data are recorded below: On BLEU-1 to BLEU-4 metrics, the proposed solution achieved the highest scores, at 0.879, 0.843, 0.819, and 0.801 respectively, significantly higher than codeGeeX's 0.778, 0.742, 0.711, and 0.688. This indicates that the code generated by the proposed solution has a significant advantage in n-gram matching accuracy, especially in the prediction accuracy of individual tokens. Compared with ChatGPT4.0 and Tongyi Qianwen 3, the proposed solution also maintains a slight lead in various BLEU metrics, indicating its superior ability to reconstruct syntactic structures and common code patterns. On the CodeBLEU metric, which focuses more on the semantic and structural similarity of codes, the proposed solution scored 0.833, also ranking first, exceeding ChatGPT 4.0 and Tongyi Qianwen 3 by approximately 2.1% and 2.4% respectively, significantly outperforming WizardCode and codeGeeX. This indicates that the solution of the present invention not only performs well in surface word order matching, but also achieves higher accuracy in semantic aspects such as variable usage, control flow structure, and function calls, and can generate high-quality code that better conforms to expected logic.

[0071] In summary, the code generation method proposed in this invention outperforms existing mainstream models in terms of syntactic correctness, code reproducibility, and semantic consistency, validating its effectiveness and reliability in practical applications. Further experiments can be conducted in scenarios such as diverse programming languages ​​and complex project-level code generation to comprehensively evaluate its generalization ability and engineering applicability.

[0072] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0073] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0074] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0075] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0076] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0077] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for generating large model code based on multi-feature improvements to Ptuning-V2, characterized in that, include: For the source code, extract and fuse its code vector features, syntax structure features, and execution flow features to generate the final text features; For the source code, extract and fuse its code comment vector features, time complexity vector features, and space complexity vector features to generate the final consumption features; For the source code, extract and fuse its graphical syntactic structure features and graphical execution flow features to generate the final image features; The final text features, final consumption features, and final image features are fused to generate the final semantic features; The final semantic features are incorporated into the Ptuning-V2 fine-tuning algorithm to construct an improved large model fine-tuning algorithm; Based on the improved large model fine-tuning algorithm and the base large model, a code generation large model is obtained by training using the source code dataset.

2. The large model code generation method based on multi-feature improved Ptuning-V2 as described in claim 1, characterized in that, The generation of final text features includes: The code vector features, syntax structure features, and execution flow features are weighted and fused together; The weighted fusion involves concatenating the three feature vectors and determining the weights of each feature vector through model learning to generate the final text features.

3. The large model code generation method based on multi-feature improved Ptuning-V2 as described in claim 2, characterized in that, The extraction of code vector features, syntax structure features, and execution flow features includes: The code vector features are extracted by inputting the source code into a preset vectorization model; The grammatical structure features are obtained by constructing an abstract syntax tree of the source code and processing the abstract syntax tree using a child node summation tree long short-term memory network. The processing of the child node summation tree long short-term memory network includes: recursive calculation from the leaf node to the root node, wherein the hidden state of each parent node is based on the hidden states of all its child nodes and is generated by controlling the information flow through the input gate, forget gate and output gate. The execution flow features are obtained by constructing the execution flow graph of the source code and processing the execution flow graph using a graph convolutional network. The processing using the graph convolutional network includes: inputting the adjacency matrix and node feature matrix of the execution flow graph into the graph convolutional network, aggregating the neighborhood node information through multi-layer graph convolution operations, and generating execution flow features representing the entire graph structure.

4. The large model code generation method based on multi-feature improved Ptuning-V2 as described in claim 1, characterized in that, The generation of final consumption features includes: The code comment vector features, time complexity vector features, and space complexity vector features are weighted and fused together; The weighted fusion involves concatenating the three feature vectors, determining the weights of each feature vector through model learning, and generating the final consumed feature.

5. The large model code generation method based on multi-feature improved Ptuning-V2 as described in claim 4, characterized in that, The extraction methods for the code comment vector features, time complexity vector features, and space complexity vector features include: The code annotation vector feature is obtained by segmenting the annotation text of the source code into sentences, inputting each sentence into a vectorization model to obtain sentence vectors, and then concatenating all sentence vectors. The time complexity vector feature and space complexity vector feature are obtained by calling a preset large language model to analyze the source code to predict its time complexity and space complexity respectively, and then mapping the predicted complexity results to the corresponding preset embedding vector.

6. The large model code generation method based on multi-feature improved Ptuning-V2 as described in claim 5, characterized in that, Before extracting the code annotation vector features, the process also includes a step of fine-tuning the vectorization model, which includes: Using paired data consisting of source code text and corresponding comment content; The similarity between code and annotation is calculated using cosine similarity, and the result is used as the training target to fine-tune the vectorized model.

7. The large model code generation method based on multi-feature improved Ptuning-V2 as described in claim 1, characterized in that, The generated final image features include: The graphical syntactic structure features and graphical execution flow features of the code are weighted and fused. The weighted fusion involves concatenating the vectors representing the image-based grammatical structure features and the image-based execution flow features, and then determining the weights corresponding to each feature vector through model learning to generate the final image features.

8. The large model code generation method based on multi-feature improved Ptuning-V2 as described in claim 7, characterized in that, The extraction methods for the graphical syntax structure features and graphical execution flow features include: The feature vectors of the syntax tree nodes and the feature vectors of the execution flow graph nodes extracted from the source code are organized into two-dimensional matrices to form the image syntax tree matrix and the image execution flow matrix. The image syntax tree matrix and the image execution flow matrix are respectively input into the visual transformer model, and processed by the multi-head self-attention mechanism and feedforward neural network layer inside the model to generate the image-based syntax structure features and image-based execution flow features respectively.

9. The large model code generation method based on multi-feature improved Ptuning-V2 as described in claim 1, characterized in that, The generation of the final semantic features includes: The final text features, final consumption features, and final image features are weighted and concatenated to form a fused feature vector; A fully connected layer is used to reduce the dimensionality of the fused feature vector to generate the final semantic features.

10. The large model code generation method based on multi-feature improved Ptuning-V2 as described in claim 1, characterized in that, The final semantic features are incorporated into the Ptuning-V2 fine-tuning algorithm, including: In each layer of the base model, the learnable cue vector, the embedding vector of the input sequence, and the final semantic features of that layer are concatenated to form a new combined input vector. The combined input vector is fed into the self-attention mechanism of this layer for calculation, thus completing the fine-tuning.

Images