Training method and device of code representation model based on multi-task learning

Abstract

The application is suitable for the field of artificial intelligence technology, and provides a training method and device of a code representation model based on multi-task learning, the code representation model comprising a preprocessing module, a code task encoder and a code task decoder, the method comprising: performing sequence processing on code samples of a plurality of target code tasks by using the preprocessing module to obtain code token sequences, linear AST structure sequences and NL token sequences of the code samples, and generating code representation sequences; performing feature extraction on the code representation sequences by using the code task encoder to obtain deep semantic representations; decoding the deep semantic representations by using the code task decoder to obtain corresponding decoding sequences; calculating task learning losses and dynamic weights of each target code task, further calculating a total learning loss of the current training, updating parameters of the code representation model according to the total learning loss, continuing to train the code representation model until the training is completed.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence, and particularly relates to a training method and device of a code representation model based on multi-task learning. BACKGROUND

[0002] With the continuous progress of computer technology and the continuous expansion of software application scenarios, the scale of software projects is growing at an explosive rate, and the system structure is becoming increasingly complex. Modern software not only has numerous functions, but also has higher requirements for performance, reliability and security, which makes developers face more severe challenges in the process of code understanding, writing and maintenance. Especially in large projects, the code volume is large, the dependency relationship between modules is complex, and the abstraction and variability of the code further increase the difficulty of development and maintenance. Therefore, how to efficiently and accurately realize code understanding and generation has become a core problem that needs to be tackled in the field of software engineering and intelligent development.

[0003] The rapid development of deep learning technology, especially the successful application of the Transformer model and its variant models in natural language processing, has further promoted the innovation in the field of code intelligence. Researchers have gradually introduced deep learning methods into code representation and generation tasks, using models to automatically learn the underlying patterns in code sequences, abstract syntax trees (AST) and natural language annotations, thereby achieving significant results in various code understanding and generation tasks. However, the current mainstream Transformer models mostly use a single-task learning paradigm, i.e., model training and optimization are performed for a single task. Although this approach can achieve good results on a specific task, when the task is transferred or faces a new related task, the model often shows a significant performance decline. Even if a code representation model performs excellently on a single task, its performance may drop significantly when transferred to a related task, which reveals the natural limitations of single-task training in terms of representation universality and transferability.

[0004] On the other hand, although code representation models based on multi-task learning have begun to try to jointly optimize multiple tasks, they generally rely on a static weight mechanism to allocate task losses. This approach cannot fully consider the convergence speed and optimization difficulty differences of different tasks during training. As a result, the tasks that converge quickly dominate the training process, while the optimization of slow-converging tasks is suppressed, leading to unbalanced model training and making it difficult to achieve true multi-task collaborative optimization. This not only limits the performance of the model in a multi-task environment, but also affects the generalization ability of code representation, making the model perform poorly when facing complex code understanding and generation tasks. SUMMARY

[0005] The application aims to provide a code representation model training method and device based on multi-task learning, aiming to solve the problem of poor multi-task processing effect of existing code representation models.

[0006] In a first aspect, the application provides a code representation model training method based on multi-task learning, the code representation model comprising a preprocessing module, a code task encoder, and a code task decoder, the code task encoder being a Transformer encoder, and the code task decoder being a Transformer decoder, the method comprising the following steps:

[0007] The preprocessing module is used to perform serialization processing on code samples of multiple target code tasks, to obtain code token sequences, linearized abstract syntax tree structure sequences, and natural language token sequences of the code samples, and to generate code representation sequences of each target code task based on the code token sequences, the linearized abstract syntax tree structure sequences, and the natural language token sequences;

[0008] The code representation sequences of each target code task are input into the corresponding code task encoder, the code task encoder is used to extract features from the code representation sequences, and deep semantic representations are obtained;

[0009] The deep semantic representations output by each code task encoder are input into the corresponding code task decoder, the code task decoder is used to decode the deep semantic representations, and corresponding decoding sequences are obtained;

[0010] The task learning loss and the dynamic weight of each target code task are calculated based on the code samples of the multiple target code tasks and the target code, the total learning loss of the current training is calculated according to the task learning loss and the dynamic weight of each target code task, the parameters of the code task encoder and the code task decoder are updated according to the total learning loss, the code representation model is continuously trained until the training is completed.

[0011] In some embodiments, the step of performing serialization processing on code samples of multiple target code tasks by using the preprocessing module to obtain code token sequences, linearized abstract syntax tree structure sequences, and natural language token sequences of the code samples comprises:

[0012] The code samples are subjected to lexical analysis to obtain code token sequences of the code samples;

[0013] The abstract syntax tree and the natural language token sequence of the code samples are obtained, and the abstract syntax tree is subjected to linearization processing to obtain linearized abstract syntax tree structure sequences of the code samples.

[0014] In some embodiments, the number of code task encoders and code task decoders is the same, and all the code task encoders share parameters.

[0015] In some embodiments, the step of inputting the deep semantic representation output by each code task encoder into a corresponding code task decoder to decode the deep semantic representation using the code task decoder to obtain a corresponding decoding sequence comprises:

[0016] The deep semantic representation is input into the code task decoder of the target code task associated with the deep semantic representation, and the deep semantic representation is decoded using the code task decoder to obtain a corresponding decoding sequence.

[0017] In some embodiments, the step of calculating the task learning loss and the dynamic weight of each target code task based on the code samples and target code of the plurality of target code tasks comprises:

[0018] The original task learning loss of each target code task is calculated based on the code samples and target code of the plurality of target code tasks.

[0019] The original task learning loss of each target code task is smoothed, and the smoothed original task learning loss is set as the task learning loss of each target code task.

[0020] Based on the task learning loss of each target code task, the learning speed of each target code task is calculated, and based on the calculated learning speed and the total number of target code tasks, the dynamic weight of each target code task is calculated.

[0021] In some embodiments, the learning speed of each target code task is calculated by the following formula:

[0022]

[0023] The learning speed of each target code task is calculated by the following formula: denotes the learning speed of target code task k at time t, denotes the task learning loss of target code task k at time t, denotes the task learning loss of target code task k at the previous time of time t, denotes a preset minimum constant;

[0024] The dynamic weight of each target code task is calculated by the following formula:

[0025]

[0026] The dynamic weight of each target code task is calculated by the following formula: denotes the dynamic weight of target code task k at time t, represents the total number of target code tasks, represents a preset temperature coefficient.

[0027] In a second aspect, the present application provides a processing method of a target code task, the method comprising the following steps:

[0028] receiving a user-inputted source code to be processed and a corresponding target code task;

[0029] inputting the source code to be processed and the target code task into a trained code representation model to obtain a target code of the source code to be processed, the code representation model being a code representation model trained by the training method in any one of the above.

[0030] In a third aspect, the present application provides a training device of a code representation model based on multi-task learning, the code representation model comprising a preprocessing module, a code task encoder and a code task decoder, the code task encoder being a Transformer encoder, the code task decoder being a Transformer decoder, the training device comprising:

[0031] a preprocessing unit configured to perform sequence processing on code samples of a plurality of target code tasks by using the preprocessing module to obtain code token sequences, linearized abstract syntax tree structure sequences and natural language token sequences of the code samples, and generate code representation sequences of each target code task based on the code token sequences, the linearized abstract syntax tree structure sequences and the natural language token sequences;

[0032] a feature extraction unit configured to input the code representation sequences of each target code task into a corresponding code task encoder, and perform feature extraction on the code representation sequences by using the code task encoder to obtain deep semantic representations;

[0033] a sequence decoding unit configured to input the deep semantic representations output by each code task encoder into a corresponding code task decoder, and perform decoding on the deep semantic representations by using the code task decoder to obtain corresponding decoding sequences; and

[0034] a parameter updating unit configured to calculate task learning losses and dynamic weights of each target code task based on the code samples and target codes of the plurality of target code tasks, calculate a total learning loss of the current training according to the task learning losses and the dynamic weights of each target code task, update parameters of the code task encoder and the code task decoder according to the total learning loss, continue training the code representation model until the training is completed.

[0035] In a fourth aspect, the present application also provides a computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method as described above when executing the computer program.

[0036] In a fifth aspect, the present application also provides a computer readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the method as described above.

[0037] After obtaining the code representation sequence of each target code task, the code representation sequence is input to the corresponding code task encoder, the code task encoder is used for feature extraction of the code representation sequence to obtain deep semantic representation, the corresponding deep semantic representation is decoded by using the code task decoder processing the same task to obtain a decoding sequence, the task learning loss and the dynamic weight of each target code task are calculated based on the code samples and the target code of multiple target code tasks, the total learning loss of the current training is calculated according to the task learning loss and the dynamic weight of each target code task, the parameters of the code task encoder and the code task decoder are updated according to the total learning loss, then the code representation model is continuously trained, and finally the trained code representation model is obtained, so that multiple code tasks are optimized in the training process of the code representation model, and the weight proportion of each code task in the total learning loss is dynamically adjusted through the dynamic weight, so that the model maintains an optimized balance among different tasks, and the processing efficiency of the code representation model multi-task is improved. BRIEF DESCRIPTION OF DRAWINGS

[0038] Figure 1 FIG. 1 is a flowchart of a training method of a code representation model based on multi-task learning provided by an embodiment of the present application;

[0039] Figure 2 FIG. 2 is a flowchart of step S104 in the embodiment one provided by an embodiment of the present application;

[0040] Figure 3 FIG. 3 is a flowchart of a training method of a code representation model based on multi-task learning provided by an embodiment of the present application;

[0041] Figure 4 FIG. 4 is a flowchart of a processing method of a target code task provided by an embodiment of the present application;

[0042] Figure 5 FIG. 5 is a structural diagram of a training device of a code representation model based on multi-task learning provided by an embodiment of the present application;

[0043] Figure 6is a structural schematic diagram of a computing device provided by an embodiment nine of the present application. DETAILED DESCRIPTION

[0044] In order to make the objects, technical solutions and advantages of the present application clearer and more apparent, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0045] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. And the terms used in the present application specification are only for the purpose of describing specific embodiments and are not intended to limit the present application. As used in the present application specification and the appended claims, the singular forms "a", "an" and "the" are intended to include the plural forms unless the context clearly indicates otherwise. The terms "first", "second" and similar words do not indicate any order, number or importance, but are only used to distinguish different components. The terms "connected" or "connected" and the like do not limit to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right", and the like are only used to indicate relative positional relationships, which may change accordingly when the absolute positions of the described objects change.

[0046] In order to keep the following description of the embodiments of the present application clear and concise, the detailed description of some known functions and known components is omitted in the present specification.

[0047] The specific implementation of the present application is described in detail below in combination with specific embodiments:

[0048] Embodiment one:

[0049] The embodiment of the present application provides a training method of a code representation model based on multi-task learning, the code representation model can process multiple code tasks, for example, code completion, code translation and code vulnerability repair, the code representation model includes a preprocessing module, a code task encoder and a code task decoder, of course, it can also include some other processing modules. Among them, the code task encoder is a Transformer encoder, and the code task decoder is a Transformer decoder.

[0050] As shown in the figure, Figure 1 The implementation process of the training method of the code representation model is shown, only the part related to the embodiment of the present application is shown for convenience, and the details are as follows.

[0051] In step S101, the code samples of the plurality of target code tasks are serialized by using a preprocessing module to obtain code token sequences, linearized abstract syntax tree structure sequences and natural language token sequences of the code samples, and code representation sequences of each target code task are generated based on the code token sequences, the linearized abstract syntax tree structure sequences and the natural language token sequences.

[0052] In the embodiment of the present application, the target code task is an operation performed on the input source code, for example, code completion, code translation and code vulnerability repair, and the code sample includes a plurality of source codes, each of which corresponds to a target code task, so as to realize multi-task learning of the code representation model through the code sample.

[0053] In step S102, the code representation sequence of each target code task is input into the corresponding code task encoder, and the code task encoder is used to extract features from the code representation sequence to obtain a deep semantic representation.

[0054] In step S103, the deep semantic representation output by each code task encoder is input into the corresponding code task decoder, and the code task decoder is used to decode the deep semantic representation to obtain a corresponding decoding sequence.

[0055] In the embodiment of the present application, the number of code task encoders and code task decoders is the same, and all code task encoders share parameters, each code task encoder corresponds to a code task and is used to extract features from the input code representation sequence to obtain a deep semantic representation, and similarly, each code task decoder also corresponds to a code task and is used to decode the deep semantic representation. One code task encoder is associated with one code task decoder, and both of them process the same code task.

[0056] In step S104, based on the code samples of the plurality of target code tasks and the target code, the task learning loss and the dynamic weight of each target code task are calculated, the total learning loss of the current training is calculated according to the task learning loss and the dynamic weight of each target code task, the parameters of the code task encoder and the code task decoder are updated according to the total learning loss, the code representation model is continuously trained until the training is completed.

[0057] In the embodiment of the present application, the target code is the corresponding target code (answer) obtained after the code sample (question) is processed by the code task, and the task learning loss of each target code task can be calculated through the code sample and the target code of the plurality of target code tasks. Based on the calculated task learning loss of each target code task, the dynamic weight of each target code task can be further calculated, so that the weight of the code task with fast convergence will be reduced, and the weight of the code task with slow convergence will be increased, thereby preventing individual code tasks from dominating the parameter update of the code representation model and ensuring the training stability of the code representation model.

[0058] Embodiment two

[0059] In the embodiment of the present application, when the pre-processing module is used to serialize the code sample of the plurality of target code tasks in step S101 of embodiment one, the code sample can be subjected to lexical analysis to obtain the code token sequence of the code sample, obtain the abstract syntax tree (Linearized AST) and the natural language token sequence (NL tokens) of the code sample, linearize the abstract syntax tree to obtain the linearized abstract syntax tree structure sequence of the code sample, and finally obtain the code token sequence, the linearized abstract syntax tree structure sequence and the natural language token sequence of the code sample.

[0060] Specifically, when obtaining the code token sequence of the code sample, the code sample of the plurality of target code tasks is subjected to lexical analysis by a tokenizer to extract the code token sequence of the code sample to reflect the basic syntax composition of the code. When obtaining the linearized abstract syntax tree structure sequence of the code sample, the code sample is parsed by an abstract syntax tree parser to obtain the abstract syntax tree (AST) of the code sample, and then the abstract syntax tree is linearized based on the traversal mode of the XML structure to obtain the linearized abstract syntax tree structure sequence to depict the structural layer information of the code. When obtaining the natural language token sequence of the code sample, the natural language token sequence of the code sample is generated by named entity recognition and call expression recognition to express the semantic intent implied by the code.

[0061] After obtaining the code token sequence, the linearized abstract syntax tree structure sequence and the natural language token sequence, the code representation sequence of each target code task is generated based on the code token sequence, the linearized abstract syntax tree structure sequence and the natural language token sequence. Specifically, the above three types of sequences can be spliced by a special delimiter to obtain the code representation sequence of the target code task, thereby realizing comprehensive modeling of the code sample features of the plurality of target code tasks.

[0062] Embodiment three

[0063] In the embodiments of the present application, step S102 in embodiment one can be realized by the following method:

[0064] (1) The input code representation sequence is vectorized to map discrete code tokens to continuous low-dimensional semantic vectors through a word embedding matrix, and position encoding is introduced to preserve the order information of the code statement, obtaining a code initial representation with semantic and position information;

[0065] (2) Context-dependent modeling is performed on the code initial representation, and the attention weight between different sequences is calculated to capture the semantic relationship in the global range, thereby obtaining a code feature representation containing structure and context information;

[0066] (3) The obtained code feature representation is subjected to nonlinear mapping and dimension transformation to further refine high-order semantic information and enhance the model's ability to abstract complex code semantics, obtaining an intermediate feature representation after feature transformation;

[0067] (4) The intermediate feature representation is normalized, and residual connection is used to retain the original feature information to prevent gradient disappearance and improve training stability, obtaining a high-dimensional context feature, which is the output of the code task encoder (deep semantic representation).

[0068] Embodiment four

[0069] In the embodiments of the present application, step S103 in embodiment one can be realized by the following method: inputting the deep semantic representation into the code task decoder of the target code task associated with the deep semantic representation, and decoding the deep semantic representation using the code task decoder to obtain the corresponding decoding sequence.

[0070] Specifically, the deep semantic representation is input into the code task decoder of the target code task associated with the deep semantic representation, and the deep semantic representation is decoded using the code task decoder, including:

[0071] (1) The target code of the code sample is vectorized to map discrete symbols to continuous semantic vectors through an embedding matrix, and position encoding is combined to preserve sequence position information, obtaining an initial input representation for subsequent decoding;

[0072] (2) Internal dependency modeling is performed on the initial input representation, the attention relationship between each sequence in the initial input representation is calculated to capture the semantic association within the generated context, and effective modeling of the target code is realized;

[0073] (3) The deep semantic representation output by the encoder is interacted with the current state of the decoder, the attention weight between the decoder query vector and the encoder key value vector is calculated, and the semantic alignment and information fusion between the code sample and the target code are calculated to obtain the fused context feature, thereby guiding the decoder to generate an output conforming to the semantic logic;

[0074] (4) The fused context feature is subjected to nonlinear mapping and semantic transformation to obtain the intermediate feature representation after feature extraction.

[0075] (5) The intermediate feature representation is subjected to normalization operation, and residual connection is used to maintain gradient flow and semantic continuity to obtain a high-dimensional vector representation, which is the output (decoding sequence) of the code task decoder.

[0076] Embodiment five

[0077] In the embodiment of the application, as shown in Figure 2 The implementation of step S104 in embodiment one is shown in the figure, and is specifically described as follows:

[0078] In step S201, based on the code samples and the target code of the plurality of target code tasks, the original task learning loss of each target code task is calculated.

[0079] In step S202, the original task learning loss of each target code task is smoothed, and the smoothed original task learning loss is set as the task learning loss of each target code task.

[0080] In step S203, based on the task learning loss of each target code task, the learning speed of each target code task is calculated, and according to the calculated learning speed and the total number of target code tasks, the dynamic weight of each target code task is calculated.

[0081] In step S204, according to the task learning loss and the dynamic weight of each target code task, the total learning loss of the current training is calculated, and according to the total learning loss, the parameters of the code task encoder and the code task decoder are updated, and the training of the code representation model is continued until the training is completed.

[0082] In the embodiment of the application, when calculating the total learning loss of the code representation model, based on the code samples and the target code of the plurality of target code tasks, the task learning loss and the dynamic weight of each target code task are calculated, and according to the task learning loss and the dynamic weight of each target code task, the total learning loss of the current training is calculated, and according to the total learning loss, the weight proportion of each code task in the total learning loss is dynamically adjusted by the dynamic weight, so that the model maintains an optimized balance between different tasks, and the multi-task processing efficiency of the code representation model is improved.

[0083] In some embodiments, the original task learning loss of each target code task is smoothed by the following formula:

[0084]

[0085] wherein, denotes the target code task the original task learning loss of the target code task at time t, is a smoothing coefficient for controlling the decay rate of the historical loss, denotes the target code task the original task learning loss of the target code task at time t, denotes the target code task the original task learning loss of the target code task at time t, denotes the smoothed original task learning loss of the target code task at time t.

[0086] In the embodiments of the present application, by smoothing the original task learning loss of each target code task, the noise influence in the training process of the code representation model is weakened, and the training efficiency of the code representation model is improved.

[0087] In some embodiments, the learning speed of each target code task is calculated by the following formula:

[0088]

[0089] wherein, denotes the learning speed of the target code task k at time t, denotes the task learning loss of the target code task k at time t, denotes the task learning loss of the target code task k at the previous time of time t, denotes a preset minimum constant.

[0090] In the embodiments of the present application, the learning speed or difficulty of each target code task can be accurately represented by the above formula, thereby improving the accuracy of subsequent dynamic weight adjustment.

[0091] In some embodiments, the dynamic weight of each target code task is calculated by the following formula:

[0092]

[0093] wherein, denotes the dynamic weight of the target code task k at time t, denotes the total number of target code tasks, denotes a preset temperature coefficient. ​

[0094] In the embodiments of the present application, the dynamic adjustment of the weight proportion of each code task in the total loss function is realized by the above formula, so that the model maintains an optimized balance between different tasks and improves the multi-task processing efficiency of the code representation model.

[0095] In some embodiments, after obtaining the task learning loss of each code task and the corresponding weight of each code task, the total learning loss of the current training can be calculated by the following formula:

[0096]

[0097] The total learning loss serves as the optimization target of the code representation model in each step of training, guiding the updating process of the code representation model parameters, thereby realizing the collaborative learning of multiple code tasks.

[0098] When determining whether the training of the code representation model is terminated, specifically, the model is trained for a maximum of 50 iterations, and an early stopping strategy is adopted: if the performance of the validation set does not improve for 20 consecutive iterations, the training is terminated in advance to reduce the computational overhead and avoid overfitting. After each training iteration, the total learning loss is calculated and the training termination condition is determined. When the iteration upper limit is reached or the early stopping condition is met, the training is ended, and the trained code representation model is obtained, otherwise the training of the code representation model is continued.

[0099] Embodiment six

[0100] In the embodiments of the present application, as an example, Figure 3 a flowchart of a training method of a code representation model based on multi-task learning is shown.

[0101] As shown in the figure, in the preprocessing stage, the code samples of the plurality of target code tasks are serialized to obtain code token sequences, linearized abstract syntax tree structure sequences and natural language token sequences of the code samples, and then the code token sequences, the linearized abstract syntax tree structure sequences and the natural language token sequences are concatenated (Concat) to generate the code representation sequence of each target code task. In the encoding stage, the code representation sequence of each target code task is input into the corresponding code task encoder (for example, the first, second and third task encoders), and the code task encoder is used to extract features from the code representation sequence to obtain a deep semantic representation. In the decoding stage, the deep semantic representation output by each code task encoder is input into the code task decoder (for example, the first, second and third task decoders) for processing the same task, and the code task decoder is used to decode the deep semantic representation to obtain a corresponding decoding sequence. In the learning loss calculation stage, based on the code samples of the plurality of target code tasks and the target code, the task learning loss (for example, L 1、 L 2、 L 3) and a dynamic weight (for example, 、 、 ) of each target code task are calculated, and the total learning loss of the current training is calculated according to the task learning loss and the dynamic weight of each target code task. According to the total learning loss, the parameters of the code task encoder and the code task decoder are updated through back propagation, and the code representation model is continuously trained until the training is completed.

[0102] In the embodiment of the application, the first, second and third task decoders are all Transformer encoders, and the parameters of the first, second and third task decoders are the same. The first, second and third task decoders are all Transformer decoders.

[0103] Embodiment seven

[0104] Figure 4 The implementation process of the target code task processing method provided by the embodiment of the application is shown. For ease of illustration, only the parts related to the embodiment of the application are shown, and the details are as follows.

[0105] In step S401, the user inputs the source code to be processed and the corresponding target code task;

[0106] In step S402, the source code to be processed and the target code task are input into the trained code representation model to obtain the target code of the source code to be processed. The code representation model is a code representation model trained according to the training method described in any of the above embodiments.

[0107] Embodiment Eight

[0108] Figure 5 The structure of the training device of the code representation model based on multi-task learning provided by Embodiment Eight of the present application is shown, and only the parts related to the embodiments of the present application are shown for the convenience of illustration.

[0109] In the embodiments of the present application, the code representation model includes a preprocessing module, a code task encoder, and a code task decoder, the code task encoder is a Transformer encoder, and the code task decoder is a Transformer decoder. The training device includes:

[0110] The preprocessing unit 51 is configured to perform sequence processing on the code samples of the plurality of target code tasks by using the preprocessing module, to obtain code token sequences, linearized abstract syntax tree structure sequences, and natural language token sequences of the code samples, and to generate code representation sequences of each target code task based on the code token sequences, the linearized abstract syntax tree structure sequences, and the natural language token sequences.

[0111] The feature extraction unit 52 is configured to input the code representation sequences of each target code task into the corresponding code task encoder, to perform feature extraction on the code representation sequences by using the code task encoder, and to obtain deep semantic representations.

[0112] The sequence decoding unit 53 is configured to input the deep semantic representations output by each code task encoder into the corresponding code task decoder, to perform decoding on the deep semantic representations by using the code task decoder, and to obtain corresponding decoding sequences.

[0113] The parameter updating unit 54 is configured to calculate the task learning loss and the dynamic weight of each target code task based on the code samples of the plurality of target code tasks and the target code, to calculate the total learning loss of the current training according to the task learning loss and the dynamic weight of each target code task, to update the parameters of the code task encoder and the code task decoder according to the total learning loss, to continue training the code representation model until the training is completed.

[0114] In the embodiments of the present application, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be implemented by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to realize all or part of the above-described functions. Each unit and module of the device can be realized by a corresponding hardware or software unit, and each unit and module can be an independent software and hardware unit, or can be integrated into a software and hardware unit, which is not used to limit the present application. In addition, the specific names of each functional unit and module are only for easy distinction from each other, and do not limit the protection scope of the present application. The specific working process of the units and modules in the device can be referred to the corresponding description in the foregoing method embodiments, which will not be described here.

[0115] Embodiment Nine

[0116] Figure 6 The structure of the computing device provided by the ninth embodiment of the present application is shown, and only the parts related to the embodiments of the present application are shown for the convenience of description.

[0117] The computing device 6 of the embodiment of the present application includes a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60. The processor 60 implements the steps in the training method embodiments of the code representation model when executing the computer program 62, such as Figure 1 the steps S101 to S104 shown. Alternatively, the processor 60 implements the functions of the units in the device embodiments when executing the computer program 62, such as Figure 5 the functions of the units 51 to 54 shown.

[0118] The computing device 6 of the embodiment of the present application can be a personal computer, a server, etc. The steps implemented when the processor 60 in the computing device 6 implements the training method of the code representation model when executing the computer program 62 can refer to the description of the foregoing method embodiments, which will not be described here.

[0119] Embodiment Ten

[0120] In the embodiments of the present application, a computer readable storage medium is provided, which stores a computer program. The computer program is executed by a processor to implement the steps in the training method embodiments of the code representation model, such as Figure 1 the steps S101 to S104 shown. Alternatively, the computer program is executed by a processor to implement the functions of the units in the device embodiments, such as Figure 5 the functions of the units 51 to 54 shown.

[0121] The computer readable storage medium of embodiments of the present application may, for example, be but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In embodiments of the present application, the computer readable storage medium can be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0122] The above embodiments are merely intended for describing the technical solutions of the present application, not to limit the present application. Although the present application is described in detail with reference to the above embodiments, those skilled in the art should understand that the disclosed scope of the above embodiments is not limited to the technical solutions formed by the specific combination of the technical features, and should also cover other technical solutions formed by any combination of the above technical features or equivalent features without departing from the disclosed concept. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

[0123] Further, although operations are depicted in a particular order, this should not be understood as requiring the operations to be performed in the particular order shown or in sequential order, and that certain operations can be performed in parallel or in any order. Also, although specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. Certain features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented separately or in any suitable subcombination.

Claims

1. A training method for a code representation model based on multi-task learning, characterized in that, The code representation model includes a preprocessing module, a code task encoder, and a code task decoder. The code task encoder is a Transformer encoder, and the code task decoder is a Transformer decoder. The method includes the following steps: The preprocessing module is used to serialize code samples from multiple target code tasks to obtain code tag sequences, linearized abstract syntax tree structure sequences, and natural language tag sequences. Based on the code tag sequences, linearized abstract syntax tree structure sequences, and natural language tag sequences, a code representation sequence for each target code task is generated. The code samples include multiple source codes, each source code corresponding to a target code task. The target code task is an operation performed on the input source code, including code completion, code translation, and code vulnerability repair. The code representation sequence of each target code task is input into the corresponding code task encoder, and the code task encoder is used to extract features from the code representation sequence to obtain deep semantic representation. The deep semantic representation output by each code task encoder is input into the corresponding code task decoder, and the deep semantic representation is decoded by the code task decoder to obtain the corresponding decoding sequence. Based on the code samples and target code of the multiple target code tasks, the task learning loss and dynamic weights of each target code task are calculated. According to the task learning loss and dynamic weights of each target code task, the total learning loss of the current training is calculated. According to the total learning loss, the parameters of the code task encoder and code task decoder are updated, and the code representation model is trained until the training ends to obtain a trained code representation model. The trained code representation model is used to process multiple code tasks. The steps of using the preprocessing module to serialize code samples from multiple target code tasks to obtain a code tag sequence, a linearized abstract syntax tree structure sequence, and a natural language tag sequence include: Lexical analysis is performed on the code sample to obtain the code tag sequence of the code sample; Obtain the abstract syntax tree and natural language tag sequence of the code sample, and perform linearization on the abstract syntax tree to obtain the linearized abstract syntax tree structure sequence of the code sample.

2. The method as described in claim 1, characterized in that, The number of code task encoders and code task decoders is the same, and all code task encoders share parameters.

3. The method as described in claim 1, characterized in that, The steps of inputting the deep semantic representation output by each code task encoder into the corresponding code task decoder, and using the code task decoder to decode the deep semantic representation to obtain the corresponding decoded sequence include: The deep semantic representation is input into the code task decoder of the target code task associated with the deep semantic representation, and the code task decoder is used to decode the deep semantic representation to obtain the corresponding decoded sequence.

4. The method as described in claim 1, characterized in that, Based on the code samples and target code of the multiple target code tasks, the steps for calculating the task learning loss and dynamic weights for each target code task include: Based on the code samples and target code of the multiple target code tasks, calculate the original task learning loss for each target code task; The original task learning loss for each target code task is smoothed, and the smoothed original task learning loss is set as the task learning loss for each target code task. Based on the task learning loss for each target code task, the learning speed for each target code task is calculated. Then, based on the calculated learning speed and the total number of target code tasks, the dynamic weight for each target code task is calculated.

5. The method as described in claim 4, characterized in that, Using the following formula: Calculate the learning speed for each target code task, where, This represents the learning speed of the target code task k at time t. This represents the task learning loss of target code task k at time t. Let represent the task learning loss of target code task k at time t in the previous time step. This represents a pre-defined minimum constant; Using the following formula: Calculate the dynamic weights for each target code task, where, This represents the dynamic weight of the target code task k at time t. Indicates the total number of target code tasks. This indicates the preset temperature coefficient.

6. A method for processing target code tasks, characterized in that, The method includes the following steps: Receive the source code to be processed and the corresponding target code task from the user; The source code to be processed and the target code task are input into the trained code representation model to obtain the target code of the source code to be processed. The code representation model is a code representation model trained according to any one of the training methods described in claims 1 to 5.

7. A training device for a code representation model based on multi-task learning, characterized in that, The code representation model includes a preprocessing module, a code task encoder, and a code task decoder. The code task encoder is a Transformer encoder, and the code task decoder is a Transformer decoder. The training device includes: The preprocessing unit is used to serialize code samples of multiple target code tasks using the preprocessing module to obtain code tag sequences, linearized abstract syntax tree structure sequences, and natural language tag sequences of the code samples. Based on the code tag sequences, linearized abstract syntax tree structure sequences, and natural language tag sequences, a code representation sequence for each target code task is generated. The code samples include multiple source codes, each source code corresponding to a target code task. The target code task is an operation performed on the input source code, including code completion, code translation, and code vulnerability repair. The feature extraction unit is used to input the code representation sequence of each target code task into the corresponding code task encoder, and use the code task encoder to extract features from the code representation sequence to obtain deep semantic representation. A sequence decoding unit is used to input the deep semantic representation output by each code task encoder into the corresponding code task decoder, and use the code task decoder to decode the deep semantic representation to obtain the corresponding decoded sequence; and The parameter update unit is used to calculate the task learning loss and dynamic weights for each target code task based on the code samples and target code of the multiple target code tasks, calculate the total learning loss of the current training based on the task learning loss and dynamic weights of each target code task, update the parameters of the code task encoder and code task decoder based on the total learning loss, and continue to train the code representation model until the training ends to obtain a trained code representation model, which is used to process multiple code tasks. The preprocessing unit, when using the preprocessing module to serialize code samples from multiple target code tasks to obtain a code tag sequence, a linearized abstract syntax tree structure sequence, and a natural language tag sequence, includes: Lexical analysis is performed on the code sample to obtain the code tag sequence of the code sample; Obtain the abstract syntax tree and natural language tag sequence of the code sample, and perform linearization on the abstract syntax tree to obtain the linearized abstract syntax tree structure sequence of the code sample.

8. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images