Method and system for integrating code structure into large model based on multiple modes

Abstract

The invention belongs to the technical field of computers, and particularly discloses a method and system for integrating a code structure into a large model based on multiple modalities, and the method comprises the steps: obtaining a code attribute graph corresponding to a code text; inputting the code attribute graph to a graph encoder, and obtaining a graph structure representation output by the graph encoder; the code text and the graph structure representation are input to a large language model, a code processing result output by the large language model is obtained, and the large language model is used for processing the code according to the code processing task based on the code text and the graph structure representation; wherein the graph structure representation and the serialization language representation are kept aligned in a representation space, and the serialization language representation is obtained by performing serialization processing on a code text by a text encoder in a large language model. According to the method, the multi-modal encoder is introduced, so that different modal information, such as grammar, semantics, control flow and data flow, of the code can be fused and processed, and the understanding ability of a large model to a code structure can be remarkably improved.

Description

Technical Field

[0001] The present application belongs to the field of computer technology, and more specifically, to a method and system for integrating a multimodal code structure into a large model. Background Art

[0002] With the widespread use of large-scale language models (abbreviated as large language models or large models), they have also shown good results in the field of code. At present, the most common way to use large models is often to directly ask the large language model by constructing prompt words, and put the code with structural information in the input text prompt words. When the large model processes the task input, the code will become a serialized text. Despite this, it has also shown considerable results in some downstream tasks of software engineering (vulnerability detection, automatic program repair, and vulnerability location, etc.). However, when the large model is directly used to query the data flow and control flow of the code structure, some false positives will still occur, which shows that there is still room for improvement in the large model's ability to understand the code structure. How to effectively improve the large model's ability to understand the code structure is a technical problem that needs to be solved urgently in this field. Summary of the invention

[0003] In view of the defects of the prior art, the purpose of this application is to effectively improve the ability of large models to understand code structure.

[0004] To achieve the above objectives, in a first aspect, the present application provides a method for integrating a code structure into a large model based on multimodality, the method comprising: obtaining a code attribute graph corresponding to a code text; Input the code attribute graph to the graph encoder and obtain the graph structure representation output by the graph encoder; Input code text and graph structure representation to the large language model, and obtain the code processing result output by the large language model. The large language model is used to process the code according to the code processing task based on the code text and graph structure representation; Among them, the graph structure representation and the serialized language representation are aligned in the representation space, and the serialized language representation is obtained by serializing the code text by the text encoder in the large language model.

[0005] In a possible implementation, the above-mentioned obtaining the code attribute graph corresponding to the code text includes: The code text is parsed through the code parsing tool to generate multiple code graph structure information corresponding to the code text; Based on multiple pieces of code graph structure information corresponding to the code text, a code attribute graph corresponding to the code text is generated.

[0006] In a possible implementation, the multiple pieces of code graph structure information include: an abstract syntax tree, a control flow graph, and a data flow graph.

[0007] In a possible implementation, the code attribute graph corresponding to the code text is generated based on the multiple pieces of code graph structure information corresponding to the code text, including: Based on the abstract syntax tree, control flow graph and data flow graph corresponding to the code text, fusion processing is performed to generate a code attribute graph corresponding to the code text.

[0008] In one possible implementation, the model structure of the graph encoder adopts a graph neural network.

[0009] Exemplarily, the graph neural network can specifically be a graph convolutional network (GCN), a graph attention network (GAT), a graph autoencoder (GAE) or a graph transformer, etc.

[0010] In a possible implementation, the graph encoder is obtained by the following steps: Get code samples by querying open source code repositories; Obtain the code attribute graph corresponding to the code sample through the code parsing tool; Based on the code attribute graph and target label corresponding to the code sample, the graph neural network is trained to obtain the trained graph neural network, and the trained graph neural network is used as a graph encoder.

[0011] Optionally, the above process of obtaining a graph encoder can also be referred to as graph encoder pre-training. Graph encoder pre-training can adopt tasks such as code structure understanding, code structure completion, node prediction, and edge prediction. Code structure understanding: output the structural representation of the code and generate a code summary. Code structure completion: randomly mask the information in the structure and let the encoder complete it. Node prediction: includes the prediction of the parent / child nodes of a specified node, such as given a graph and a specified node, predicting the parent / child nodes of the specified node in the graph. Edge prediction: given a graph and two nodes in the graph, predict whether there is a directly connected edge between the two nodes and other tasks.

[0012] In a possible implementation, the method further includes: adjusting model parameters of the graph encoder and model parameters of the large language model through two-stage training, so that the graph structure representation and the serialized language representation are aligned in the representation space; Among them, the first stage of training is to fix the large language model parameters and adjust the model parameters of the graph encoder, and the second stage of training is to fix the graph encoder parameters and adjust the model parameters of the large language model.

[0013] In a second aspect, the present application provides a multi-modal code structure integrated into a large model system, including: A graph structure information acquisition module, used to obtain a code attribute graph corresponding to a code text; A graph structure information processing module, used to input a code attribute graph into a graph encoder and obtain a graph structure representation output by the graph encoder; A code processing task execution module is used to input code text and graph structure representation into the large language model and obtain the code processing result output by the large language model. The large language model is used to process the code according to the code processing task based on the code text and graph structure representation; Among them, the graph structure representation and the serialized language representation are aligned in the representation space, and the serialized language representation is obtained by serializing the code text by the text encoder in the large language model.

[0014] In a third aspect, the present application provides an electronic device comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method described in the first aspect or any possible implementation of the first aspect.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method described in the first aspect or any possible implementation of the first aspect.

[0016] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art: The textual representation of the code is uniformly encoded with its graph structure (such as abstract syntax tree, control flow graph, data dependency graph, etc.), and the grammatical information and structured features of the code are deeply modeled with the help of the characteristics of the multimodal encoder. Through this process, the model can not only capture the surface semantics of the code, but also understand its inherent logic and structural relationships, thereby improving its performance in tasks such as code analysis, vulnerability detection, and automatic repair. Ultimately, this application can enhance the large model's ability to understand code structure while enhancing its cross-modal processing capabilities, providing strong support for intelligent analysis in complex code scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of a method for integrating a multimodal code structure into a large model provided in an embodiment of the present application; Figure 2 It is a schematic diagram of extracting a code property graph using a code analysis tool provided in an embodiment of the present application; Figure 3 is a schematic diagram of encoding using a graph attention neural network provided in an embodiment of the present application; Figure 4 is a schematic diagram of modal alignment of text and graph structure provided in an embodiment of the present application; Figure 5 It is a structural diagram of integrating a multi-modal code structure into a large model system provided by an embodiment of the present application; Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to facilitate a clearer understanding of the embodiments of the present application, some relevant background knowledge is first introduced as follows.

[0019] If the structural information of the code can be encoded at the stage of inputting into the big model, the big model's ability to understand the input code can be enhanced, thereby improving the performance of downstream tasks that require understanding of the code. Multimodal big models are an emerging deep learning method that can fuse data from other modalities when the pre-training data only has text and serialized data, so as to achieve the ability to complete more complex and diverse downstream tasks using the understanding of the big model. Based on this, this application proposes a method and system for integrating multimodal code structure into a big model through alignment fusion training.

[0020] The upstream data for pre-training of large-scale language models are all serialized data. How to enable the model that has only been trained on serialized data to obtain and understand the structural information of the code graph is a key problem in multimodal fusion. This not only involves extracting the logic and grammatical structure of the code from the linear sequence, but also requires the model to be able to capture the structural information of higher-dimensional graphs such as function calls, control flows, and data flows in the code. The graph structure of the code (such as abstract syntax trees, control flow graphs, dependency graphs, etc.) is usually used to represent the relationship between program elements, while serialized data can only show the linear order of these elements and it is difficult to directly reflect complex dependencies. Therefore, how to combine the structural representation of the graph with the traditional serialized language representation in a multimodal environment, fully integrate the features of these different data forms, and then improve the model's understanding and reasoning ability of the code is a problem that needs to be solved urgently in the existing technology. This not only requires effective expression of multimodal information at the input level, but also requires the model to have the ability to collaboratively process different modal information to achieve a comprehensive understanding and accurate inference of code semantics. This application encodes the code graph structure through a multimodal fusion alignment method to integrate it into the large model, and aligns different modal information so that the module used for encoding and the large model module can collaborate in reasoning to complete the task.

[0021] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0022] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0023] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0024] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0025] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0026] Figure 1 is a flow chart of a method for integrating a multimodal code structure into a large model provided in an embodiment of the present application, such as Figure 1 As shown, the method includes the following steps S101, S102 and S103.

[0027] Step S101, obtaining a code attribute graph corresponding to the code text; Step S102, inputting the code attribute graph into the graph encoder, and obtaining the graph structure representation output by the graph encoder; Step S103, inputting the code text and the graph structure representation into the large language model, obtaining the code processing result output by the large language model, the large language model is used to process the code according to the code processing task based on the code text and the graph structure representation; Among them, the graph structure representation and the serialized language representation are aligned in the representation space, and the serialized language representation is obtained by serializing the code text by the text encoder in the large language model.

[0028] Here, the code processing task is exemplified, and the code processing task may be code analysis, vulnerability detection, automatic repair and other tasks. In the case where the code processing task is a code analysis task, the large language model performs corresponding analysis on the code based on the code text and graph structure representation, and the code processing result output by the large model is specifically the code analysis result. In the case where the code processing task is a vulnerability detection task, the large language model performs vulnerability detection on the code based on the code text and graph structure representation, and the code processing result output by the large model is specifically the vulnerability detection result. In the case where the code processing task is an automatic repair task, the large language model repairs the code based on the code text and graph structure representation, and the code processing result output by the large model is specifically the repaired code.

[0029] By introducing multimodal encoders (text encoders and graph encoders), different modal information of the code (such as syntax, semantics, control flow, data flow, etc.) can be fused and processed, which can significantly improve the large model's ability to understand the code structure.

[0030] Specifically, this application uniformly encodes the textual representation of the code and its graph structure (such as abstract syntax tree, control flow graph, data dependency graph, etc.), and uses the characteristics of multimodal encoders to deeply model the grammatical information and structured features of the code. Through this process, the model can not only capture the surface semantics of the code, but also understand its inherent logic and structural relationships, thereby improving performance in tasks such as code analysis, vulnerability detection, and automatic repair. Ultimately, this application can enhance the large model's ability to understand code structure while enhancing its cross-modal processing capabilities, providing strong support for intelligent analysis in complex code scenarios.

[0031] The following is an illustrative description of the method for integrating multimodal code structure into a large model provided by the present application with several examples. In this method, first, a static code parsing tool processes the code extraction to obtain a structural representation of the graph (such as an abstract syntax tree, a control flow graph, a dependency graph, etc.). Then, an independent graph encoder is designed to process the structural information of the graph and encode it. Finally, an alignment module is designed to align the encoding result obtained by the graph encoder to the representation space of the large model. The method mainly includes a data sorting stage, an encoding stage, and an alignment stage.

[0032] Figure 2 is a schematic diagram of extracting a code attribute graph using a code analysis tool provided in an embodiment of the present application, such as Figure 2 As shown, the data collation stage includes the following steps: data collection and code graph structure extraction.

[0033] Regarding the data collection phase, specifically, a large amount of single-function programming codes are collected through open source code library websites (such as Github, etc.), including programming language, function name, function parameters and other information.

[0034] For the code graph structure extraction phase, we use static code parsing tools (such as Joern, etc.) to parse the collected function codes to obtain a code property graph (CPG) containing the code graph structure information. The code property graph unifies multiple code representations such as the abstract syntax tree (AST), control flow graph (CFG) and data flow graph (DFG) in one graph structure, so that the analysis tool can have a deeper understanding and processing of the code.

[0035] The steps of code graph structure extraction include the following sub-steps: (1) parsing the code into an abstract syntax tree (AST), which helps capture the basic structure and syntax elements of the code; (2) further constructing a control flow graph (CFG) to represent the execution order of statements in the program, and a data flow graph (DFG) to track the flow of variables and data in the program; (3) merging these graph structures into a unified code property graph (CPG), which can effectively represent the semantics and logical structure of the program.

[0036] Here we explain the Code Property Graph (CPG). CPG is a comprehensive graph structure used to represent program code. It combines the features of the Abstract Syntax Tree (AST), Control Flow Graph (CFG), and Data Flow Graph (DFG), and can provide rich information for code analysis, static analysis, and security auditing.

[0037] Here we explain the Abstract Syntax Tree (AST), which is a tree structure used to represent the grammatical structure of source code. Each node represents a construct in the source code (such as a variable, operator, statement, etc.), and the hierarchical structure of the tree reflects the grammatical relationship between these constructs.

[0038] Here we explain the control flow graph (CFG), which is a directed graph used to represent the path of control flow in a program. The nodes in the graph represent basic blocks (i.e., a continuous piece of code) in the program, and the edges represent the transfer of control flow (such as conditional branches, loops, etc.).

[0039] Here, a data flow graph (DFG) is explained. DFG is a graphical representation used to describe the flow and dependencies of data in a program. The nodes in the DFG represent computational operations or variables, while the edges represent data dependencies.

[0040] Here, the multiple code graph structure information corresponding to the above-mentioned generated code text is exemplified. (1) Parse the source code to generate AST: Use the code parsing tool to parse the source code into an abstract syntax tree (AST). The nodes of the AST represent the syntactic elements in the code, such as variables, functions, control structures, etc. (2) Build a control flow graph (CFG): Extract control flow information from the AST and build a control flow graph (CFG). Basic blocks (i.e., a continuous section of code) are identified as nodes of the CFG, while control flow transfers (such as conditional branches, loops, etc.) are represented as edges. This step usually involves analyzing the control structure of the program to determine the execution path between different code blocks. (3) Build a data flow graph (DFG): Analyze the AST and CFG, extract data dependencies, and build a data flow graph (DFG).

[0041] Here, the above-mentioned abstract syntax tree, control flow graph and data flow graph corresponding to the code text are fused to generate the code attribute graph corresponding to the code text. (1) Node fusion: The nodes of CPG can simultaneously represent the syntax elements in AST, the basic blocks in CFG and the data operations in DFG. For example, a function node can contain its control flow information and data dependency information at the same time. (2) Edge definition: The edges in CPG can represent multiple relationships, such as control flow edges (from CFG), data dependency edges (from DFG) and syntax structure edges (from AST). This definition of multiple edges enables CPG to capture the complex interactions of the code. (3) Attribute annotation: Add attributes to the nodes and edges in CPG to facilitate subsequent analysis and query. For example, nodes can contain attributes such as type information, scope, definition location, etc., while edges can contain information such as conditional expressions and data flow direction.

[0042] Figure 3 is a schematic diagram of encoding using a graph attention neural network provided in an embodiment of the present application, such as Figure 3 As shown, the encoding stage mainly encodes the structural information through the designed graph encoder (graph attention neural network), so that the graph information that cannot be input into the large model can be input in the form of vectors. The encoding stage includes the following steps S301, S302 and S303.

[0043] Step S301: Graph preprocessing and feature extraction, preprocessing and feature extraction of the graph. Taking the Graph Attention Network (GAT) as an example, the Graph Attention Network (GAT) receives a graph structure with node features and edges, so it is necessary to generate a feature vector for each node and use the edge type as the connection information of the graph; Graph preprocessing and feature extraction include the following sub-steps. (1) Node feature extraction: Feature representation can be generated for each node based on the attributes of code elements (such as variable name, data type, operator type, etc.), usually through word embedding. (2) Edge information representation: The edges in the graph represent different types of relationships, such as grammatical connections, data dependencies, or control flows. These edges need to be incorporated into the model through specific graph representations.

[0044] Step S302: Construct a graph attention network (GAT) model. GAT is a neural network that can automatically learn the relationship between nodes in a graph structure. It realizes information propagation by calculating the attention weights of neighbor nodes for each node on the graph. GAT consists of multiple graph attention layers. Each layer aggregates the features of neighbor nodes through the attention mechanism and generates new node representations.

[0045] The construction of the graph attention network consists of the following sub-steps. (1) Initialize node features and edges. (2) In each layer of the network, calculate the attention weights between the target node and its neighboring nodes. These weights are based on the similarity of the node features. (3) Use these weights to weighted sum the features of neighboring nodes and update the representation of each node. (4) Multi-layer GAT can capture deeper structural information.

[0046] Step S303: Use the constructed GAT model for training. The training data usually includes CPG and its corresponding target label (such as function classification, vulnerability detection label, etc.). During the training process, the model will gradually optimize the attention mechanism and the representation of node features to minimize the loss function.

[0047] The alignment stage mainly aligns the output of the graph attention neural network independent of the large model to the representation space of the large model through training, so that the two models can work better together.

[0048] The specific approach is to achieve this through two-step training (or two-stage training). First, the parameters of the large model are fixed, and only the parameters of the graph encoder are trained. During the training process, contrastive learning, transfer learning and other methods can be used to optimize the parameters of the graph encoder so that the output of the graph encoder is as close as possible to the output of the text encoder under the specified metric method. The purpose is to align the output of the graph encoder with the output of the text encoder in the large model, so that the output of the graph encoder is more consistent with the representation and understanding of the large model. Then the parameters of the graph encoder are fixed, and the parameters of the large model are continued to be trained. The purpose is to combine the input of the graph structure to tap the potential capabilities of the large model.

[0049] Figure 4 is a schematic diagram of modal alignment of text and graph structure provided in an embodiment of the present application, such as Figure 4 As shown, the output of the graph encoder is aligned with the output of the text encoder in the large model.

[0050] This application introduces multimodal fusion technology to inject the structured information of the code as input into the large model, so that the large model that originally only accepted pure text sequence data can expand its perception range and pay attention to more complex modal information. This technology breakthrough combines the multi-dimensional representation of the code (such as grammatical structure, control flow, data flow, dependencies, etc.) with the language processing capabilities of the large model, so that the model not only relies on traditional linear serialization input, but also can capture the deep structure and logical associations in the code. In this way, the model's code understanding ability is significantly improved, and it can more accurately identify logical patterns, potential problems, and functional implementation details in the code, thereby showing a stronger level of intelligence and processing capabilities in complex tasks such as code auditing, vulnerability detection, and automatic program repair. At the same time, this method enhances the large model's ability to integrate cross-modal information, giving it a wider range of adaptability and application scenarios, and further promoting the advancement of intelligent programming technology.

[0051] The following is a description of the multimodal-based code structure integration into a large model system provided by the present application. The multimodal-based code structure integration into a large model system described below and the multimodal-based code structure integration into a large model method described above can be referenced to each other.

[0052] Figure 5 It is a schematic diagram of the structure of integrating a multi-modal code structure into a large model system provided by an embodiment of the present application, such as Figure 5 As shown, the system includes: a graph structure information acquisition module 10, a graph structure information processing module 20 and a code processing task execution module 30. Among them: A graph structure information acquisition module 10, used to acquire a code attribute graph corresponding to the code text; A graph structure information processing module 20, used to input a code attribute graph to a graph encoder and obtain a graph structure representation output by the graph encoder; A code processing task execution module 30 is used to input code text and graph structure representation to a large language model, and obtain a code processing result output by the large language model. The large language model is used to process the code according to the code processing task based on the code text and graph structure representation; Among them, the graph structure representation and the serialized language representation are aligned in the representation space, and the serialized language representation is obtained by serializing the code text by the text encoder in the large language model.

[0053] It can be understood that the detailed functional implementation of each of the above-mentioned units / modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0054] It should be understood that the above-mentioned system is used to execute the methods in the above-mentioned embodiments. The implementation principles and technical effects of the corresponding program modules in the system are similar to those described in the above-mentioned methods. The working process of the system can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0055] Based on the method in the above embodiment, an embodiment of the present application provides an electronic device, Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the electronic device may include: a processor (Processor) 810, a communication interface (Communications Interface) 820, a memory (Memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logic instructions in the memory 830 to execute the method in the above embodiment.

[0056] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0057] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0058] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0059] It is understandable that the processor in the embodiment of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0060] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0061] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0062] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0063] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for integrating multimodal code structure into a large model, characterized in that: include: Get the code attribute graph corresponding to the code text; Input the code attribute graph to the graph encoder and obtain the graph structure representation output by the graph encoder; Input code text and graph structure representation to the large language model, and obtain the code processing result output by the large language model. The large language model is used to process the code according to the code processing task based on the code text and graph structure representation; Among them, the graph structure representation and the serialized language representation are aligned in the representation space, and the serialized language representation is obtained by serializing the code text by the text encoder in the large language model.

2. The method for integrating multimodal code structure into a large model according to claim 1, characterized in that: The step of obtaining a code attribute graph corresponding to the code text includes: The code text is parsed through the code parsing tool to generate multiple code graph structure information corresponding to the code text; Based on multiple pieces of code graph structure information corresponding to the code text, a code attribute graph corresponding to the code text is generated.

3. The method for integrating multimodal code structure into a large model according to claim 2, characterized in that: The multiple pieces of code graph structure information include: an abstract syntax tree, a control flow graph, and a data flow graph.

4. The method for integrating multimodal code structure into a large model according to claim 3, characterized in that: The generating of the code attribute graph corresponding to the code text based on the multiple code graph structure information corresponding to the code text includes: Based on the abstract syntax tree, control flow graph and data flow graph corresponding to the code text, fusion processing is performed to generate a code attribute graph corresponding to the code text.

5. The method for integrating multimodal code structure into a large model according to any one of claims 1 to 4, characterized in that: The model structure of the graph encoder adopts a graph neural network.

6. The method for integrating multimodal code structure into a large model according to claim 5, characterized in that: The graph encoder is obtained by the following steps: Get code samples by querying open source code repositories; Obtain the code attribute graph corresponding to the code sample through the code parsing tool; Based on the code attribute graph and target label corresponding to the code sample, the graph neural network is trained to obtain the trained graph neural network, and the trained graph neural network is used as a graph encoder.

7. The method for integrating multimodal code structure into a large model according to claim 6, characterized in that: Also includes: Through two-stage training, the model parameters of the graph encoder and the model parameters of the large language model are adjusted to align the graph structure representation with the serialized language representation in the representation space; Among them, the first stage of training is to fix the large language model parameters and adjust the model parameters of the graph encoder, and the second stage of training is to fix the graph encoder parameters and adjust the model parameters of the large language model.

8. A multi-modal code structure integrated into a large model system, characterized in that: include: A graph structure information acquisition module, used to obtain a code attribute graph corresponding to a code text; A graph structure information processing module, used to input a code attribute graph into a graph encoder and obtain a graph structure representation output by the graph encoder; A code processing task execution module is used to input code text and graph structure representation into the large language model and obtain the code processing result output by the large language model. The large language model is used to process the code according to the code processing task based on the code text and graph structure representation; Among them, the graph structure representation and the serialized language representation are aligned in the representation space, and the serialized language representation is obtained by serializing the code text by the text encoder in the large language model.

9. An electronic device, characterized in that: include: at least one memory for storing a computer program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program runs on a processor, the processor is caused to execute the method according to any one of claims 1 to 7.

Images