A cross-training method for code search

Abstract

The application discloses a cross-training method for code search, comprising the following steps: feature coding is performed on a natural language query and a code snippet to generate a semantic representation vector; cross-training is performed based on the semantic representation vector to obtain a contrastive learning loss of an In-Policy stage and a Non-Policy stage, respectively; in the In-Policy stage, negative examples are generated based on code snippets stored in a historical training process; in the Non-Policy stage, code snippets in a current batch are used as negative examples; model parameters are optimized based on the contrastive learning loss, and a code snippet matched with the semantic of the natural language query is output. The application significantly expands the diversity of negative examples without increasing the online computing burden, and improves the retrieval accuracy and robustness.

Description

Technical Field

[0001] This invention belongs to the technical field of code representation learning, and in particular relates to a cross-training method for code search. Background Technology

[0002] Code search is a core task in modern software development and maintenance, aiming to retrieve semantically relevant code snippets from open-source code repositories or internal enterprise code repositories based on user-described requirements in natural language. With the rapid growth of code size and the diversification of programming languages, efficient code search methods are crucial for improving development efficiency and promoting code reuse. In recent years, deep learning-based code representation learning techniques have significantly propelled the development of code search, resulting in various pre-trained models such as CodeBERT, GraphCodeBERT, and UniXcoder. These models achieve significant performance improvements by mapping natural language queries and source code to a shared semantic representation space and performing retrieval based on vector similarity. For example, CodeBERT utilizes Masked Language Modeling (MLM) and Replacement Token Detection (RTD) tasks for pre-training on large-scale code-text pairs; GraphCodeBERT enhances its understanding of code structure by incorporating data flow information; and UniXcoder combines multimodal inputs of source code and Abstract Syntax Trees (ASTs) to support cross-modal semantic modeling. In addition, contrastive learning techniques are widely used in code representation learning, which improves the model's ability to discriminate semantically similar code by narrowing the representation distance between positive sample pairs and widening the representation distance between negative sample pairs. However, existing code search methods still have several shortcomings in the fine-tuning stage: (1) Traditional fine-tuning methods mainly rely on samples in the current batch to construct negative examples, and the diversity of negative examples is limited, making it difficult to fully explore the semantic discrimination potential of the pre-trained model; (2) Negative example generation usually adopts mechanisms such as momentum encoders or dropout perturbation, which are computationally complex and may lead to training instability; (3) Existing methods lack flexible training strategy scheduling, making it difficult to adapt to the needs of multilingual or complex datasets (such as real user queries or obfuscated code); (4) When dealing with multilingual code or low-quality code data, the generalization and robustness of the model still need to be further improved. In response to the above problems, an efficient fine-tuning method is urgently needed. Summary of the Invention

[0003] This invention proposes a cross-training method for code search to address the problems existing in the prior art.

[0004] To achieve the above objectives, this invention provides a cross-training method for code search, comprising the following steps: Natural language queries and code snippets are feature-encoded to generate semantic representation vectors; Based on the semantic representation vector, cross-training is performed to obtain the contrastive learning loss for the In-Policy stage and the Non-Policy stage, respectively. In the In-Policy phase, negative examples are generated based on code snippets stored during the historical training process; In the Non-Policy phase, code snippets within the current batch are used as negative examples; The model parameters are optimized based on the contrastive learning loss, and a code snippet that matches the semantics of a natural language query is output.

[0005] Optionally, the feature encoding includes: The input sequences of the natural language queries and code fragments are subject to length restrictions. The length-limited natural language queries and code snippets are input into a pre-trained model based on the Transformer architecture; Low-dimensional semantic representation vectors containing semantic information are generated by pre-training models.

[0006] Optionally, the cross-training includes: A Cross strategy is adopted, which executes the In-Policy phase and the Non-Policy phase in a periodic alternation manner; The type of the current execution phase is determined based on the parity of the training cycle index.

[0007] Optionally, the periodic alternation method includes: A periodic scheduling function is defined, which is used to execute the In-Policy training phase in even-numbered training cycles and the Non-Policy training phase in odd-numbered training cycles. The execution order of the In-Policy training phase and the Non-Policy training phase is switched according to the periodic scheduling function.

[0008] Optionally, during the In-Policy phase, the historical semantic vector and the current query vector are compared to calculate the similarity to obtain the historical negative sample similarity. Based on the historical negative sample similarity, a first contrastive learning loss function is constructed to update the model parameters.

[0009] Optionally, the expression for the first contrastive learning loss function is: ; In the formula, Represents the annotation vector. This indicates the positive sample code representation. This represents the negative sample code. The cosine similarity between vectors is represented by... This indicates temperature hyperparameters. Indicates the first in the batch The positive sample code corresponding to each data point represents, Indicates batch size, Indicates the size of the cache queue.

[0010] Optionally, during the Non-Policy phase, code snippets from the current batch are input into the current model to obtain the current semantic vector; The similarity between the current semantic vector and the current query vector is calculated to obtain the similarity of the current negative sample. Based on the current negative sample similarity, a second contrastive learning loss function is constructed to update the model parameters.

[0011] Optionally, the expression for the second contrastive learning loss function is: ; In the formula, Represents the annotation vector. This indicates the positive sample code representation. The cosine similarity between vectors is represented by... This indicates temperature hyperparameters. Indicates the first in the batch The positive sample code corresponding to each data point represents, Indicates the batch size.

[0012] Optional, model optimization and querying include: Based on the current training stage, select the appropriate contrastive learning loss to optimize the model; Use cosine similarity to calculate the semantic distance between the query and the candidate code snippet; Based on the semantic distance, the code fragment most relevant to the query is retrieved from the candidate code set.

[0013] Compared with the prior art, the present invention has the following advantages and technical effects: This invention significantly improves the retrieval performance and robustness of the model by alternately executing In-Policy and Non-Policy training phases, combining the storage of historical code fragments as negative examples in a cache queue, and using a historical best checkpoint model for negative example encoding, thus providing a new technical solution for code search tasks. Attached Figure Description

[0014] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a method framework diagram of an embodiment of the present invention; Figure 2 This is a comparison chart of temperature parameter experiments in an embodiment of the present invention; Figure 3 This is a comparison chart of the learning rate parameter experiments in an embodiment of the present invention. Detailed Implementation

[0015] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0016] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0017] Example 1 like Figure 1 As shown, this embodiment provides a cross-training method for code search, including the following steps: Natural language queries and code snippets are feature-encoded to generate semantic representation vectors; Based on semantic representation vectors, cross-training is performed to obtain the contrastive learning loss for the In-Policy stage and the Non-Policy stage, respectively. In the In-Policy phase, negative examples are generated based on code snippets stored during historical training. In the Non-Policy phase, the code snippet within the current batch is used as the negative example; The model parameters are optimized based on contrastive learning loss, and a code snippet that matches the semantics of a natural language query is output.

[0018] Specifically, the following steps are included: 1. Feature encoding includes: Length limits are applied to the input sequences of natural language queries and code snippets; The length-limited natural language queries and code snippets are input into a pre-trained model based on the Transformer architecture; Low-dimensional semantic representation vectors containing semantic information are generated by pre-training models.

[0019] Specifically, a pre-trained model (such as CodeBERT or UniXcoder) is used to encode natural language queries and code snippets into semantic representation vectors. The input includes the natural language query and the code snippet. Through the last hidden state of the pre-trained model, a low-dimensional dense feature vector is generated, which provides input for subsequent contrastive learning.

[0020] Let the current training batch be... Contains a sequence of annotations and its corresponding code snippets The cache queue is denoted as This invention denotes the current training model as... And introduce the checkpoint model that achieved the best performance on the validation set during the historical training process, denoted as It is used to encode samples in the cache queue.

[0021] The current sample is from Encoding yields a representation vector: in, Indicates batch size. Cache negatives are based on the historical best model. The encoding is represented as: in This represents the number of samples in the cache queue.

[0022] 2. Cross-training includes: The In-Policy phase and the Non-Policy phase are executed in a periodic alternation manner; The type of the current execution phase is determined based on the parity of the training cycle index.

[0023] The periodic alternation method includes: A periodic scheduling function is defined, which is used to execute the In-Policy training phase in even-numbered training cycles and the Non-Policy training phase in odd-numbered training cycles. The execution order of the In-Policy training phase and the Non-Policy training phase is switched according to the periodic scheduling function.

[0024] The In-Policy phase aims to introduce more semantically perturbative negative examples to enhance the model's discriminative ability. To this end, this invention designs a cache queue to store code snippets that appeared in previous training. These historical code samples participate in the current training as "difficult negative examples," compared to the current... The random negative examples within the dataset are typically more diverse, enabling the construction of more challenging training contrasts.

[0025] After encoding all samples, this invention will assign each annotation vector The code representation paired with it Treated as positive sample pairs All representations in the cache queue As negative samples Based on this, the contrastive learning loss function for the In-Policy phase is defined as follows: in, The cosine similarity between vectors is represented by... Temperature is a hyperparameter used to adjust the smoothness of the similarity distribution. The core idea of ​​contrastive learning is to maximize the semantic similarity of positive samples while minimizing the similarity between positive and negative samples. Indicates the first in the batch The positive sample code corresponding to each data point represents, Indicates batch size, Indicates the size of the cache queue.

[0026] Unlike the In-Policy phase, which uses cached negatives for enhancement, the Non-Policy phase is a standard code search and fine-tuning process, a common training method for most pre-trained models in downstream tasks. In this phase, training depends only on the current... The annotations in the code construct positive and negative samples with their paired code snippets without introducing additional information or structure.

[0027] Specifically, given the current annotation vectors in and its corresponding positive samples This invention treats the code representations in the remaining samples as negative samples. The contrastive learning loss function constructed based on this is shown below: 3. Negative example generation: During the In-Policy training phase, a cache queue is used to store code snippets from the historical training process. The code snippets in the cache queue are encoded using the checkpoint model that performed best on the validation set during the historical training process, generating negative example representations, as shown in Equation (2). By encoding the contents of the cache queue using the historical best checkpoint model, this process is only performed during the In-Policy phase. This avoids the current model directly participating in negative example generation, reducing the burden of parameter updates and training instability.

[0028] 4. Training strategies: To fully leverage the advantages of both In-Policy and Non-Policy training phases, this invention proposes four training strategies to schedule the frequency and order of the two phases throughout the training cycle. These four strategies are: Cross, Increase, Decrease, and Divide. Figure 1 As shown on the right. Each strategy embodies a different design philosophy regarding training pacing, suitable for the training needs of different models and tasks. Let the total training time be... For each epoch, this invention defines a function. Indicates the first The strategy phase adopted for each epoch is as follows: In this context, Non stands for Non-Policy and In stands for In-Policy.

[0029] This invention employs a Cross strategy, which alternates between In-Policy and Non-Policy phases throughout the training cycle. The Cross strategy balances negative example diversity and training stability through periodic alternation, ensuring optimal performance of the model in multilingual code search tasks. Its policy scheduling function can be formalized as: in This represents the modulo operation. This indicates the epoch index.

[0030] 5. Model optimization and prediction: By optimizing the contrastive learning loss function, the model uses cosine similarity to narrow the representation distance between positive sample pairs and widen the representation distance between negative sample pairs. Finally, the optimized model retrieves the code snippets most semantically relevant to the natural language query from the candidate code set, outputting MRR and Top-k (k=1,5,10) retrieval results.

[0031] This embodiment also underwent experimental verification, the verification content of which is as follows: Dataset and Experiment Setup: This invention uses four widely used public datasets—CodeSearchNet (CSN), AdvTest, CosQA, and GPD—to conduct experiments and verify the performance of the proposed method in various code search scenarios. Statistical information for the datasets is shown in Table 1. Specific dataset descriptions are as follows: (1) CSN: Based on the CodeSearchNet corpus, covering data from six mainstream programming languages: Ruby, JavaScript, Go, Python, Java, and PHP. To improve query quality, low-quality natural language queries were filtered using manual rules to ensure that the queries used in the retrieval task have high accuracy and semantic relevance. The training set contains 251,820 samples, the validation set contains 13,914 samples, the test set contains 14,918 samples, and there are 43,827 candidate code snippets.

[0032] (2) AdvTest: Based on CSN, function names and variable names are obfuscated and replaced with semantically meaningless placeholders. This aims to reduce the model's dependence on surface identifiers and focus on the semantic understanding of the code to evaluate the model's generalization and robustness. The training set contains 251,820 samples, the validation set contains 9,604 samples, and the test set contains 19,210 samples.

[0033] (3) CosQA: Queries are derived from real search logs of Microsoft Bing search engine, and code-query pairs are manually labeled. Because the query content closely resembles actual user behavior, this dataset better reflects the application effect of the model in real-world code search tasks. The training set contains 19,604 samples, the validation set and the test set each contain 500 samples, and there are 6,268 candidate code snippets.

[0034] (4) GPD: A new dataset constructed in this invention, which comes from a real project on GitHub. After normalization and cleaning, about 100,000 high-quality samples are retained as a supplement to the existing dataset and are used to further verify the stability and generalization ability of the model under real corpus.

[0035] In all experiments, the maximum input sequence length for annotations and code snippets was set to 128 and 256, respectively. The optimizer used was AdamW, with an initial learning rate of 2e-5, a temperature hyperparameter of 0.05, and a batch size of 128. The number of training epochs on different datasets was set as follows: 10 epochs for CSN, CosQA, and GPD, and 4 epochs for AdvTest. All experiments were conducted on a server equipped with 128GB of RAM and a single 48GB A6000 GPU.

[0036] Table 1

[0037] Evaluation indicators: This invention uses two evaluation metrics—Mean Reciprocal Rank (MRR) and Top-k accuracy (k=1, 5, 10)—to assess the model's performance in code search tasks. The metrics are defined as follows: (1) MRR: Measured by the reciprocal rank of correct results in a code search task, reflecting the model's ability to rank the best matching results. Let the query set be... , For query The ranking of the matching codes is determined by the following formula: (2) Top-k accuracy: measures the proportion of correct codes in the top k results, and evaluates the probability that the model contains the correct answer in the top k returned results. It is an indicator function, when The value is 0 if it is 0, otherwise it is 1. The formula is: In the above formula, Indicates the total number of queries. Indicates the relationship with the first The ranking of the correct code matched by each query in the candidate list. MRR and Top-k accuracy are used together to comprehensively evaluate the model's performance in code search tasks.

[0038] Comparison method: In this section, the method proposed in this invention is applied to several pre-trained models for comparison to verify its effectiveness. The comparison method includes seven encoder models and three encoder-decoder models: (1) RoBERTa: Based on a multi-layer Transformer encoder, it is pre-trained on a natural language corpus using an MLM task.

[0039] (2) CodeBERT: Pre-trained on six programming languages ​​of CodeSearchNet, it learns code semantics through MLM and RTD tasks.

[0040] (3) GraphCodeBERT: Pre-trained on CodeSearchNet, it introduces data flow information and enhances the code structure representation through MLM, edge prediction and node alignment tasks.

[0041] (4) StarEncoder: Pre-trained on 86 programming languages ​​of The Stack, using MLM and next sentence prediction (NSP) tasks.

[0042] (5) SynCoBERT: It integrates source code, AST and annotations, and improves the ability of code syntax and semantic representation through identifier prediction, AST edge prediction and cross-modal contrastive learning tasks.

[0043] (6) UniXcoder: Using source code and simplified AST bimodal input, it enhances the semantic and structural representation of code through MLM, one-way language modeling, denoising and contrastive learning tasks.

[0044] (7) CoCoSoDa: Through soft data augmentation and momentum contrastive learning, it dynamically constructs positive samples and maintains a negative sample queue, optimizes cross-modal representation, and improves code search performance.

[0045] (8) CodeT5: Based on the T5 architecture, design identifier-aware pre-training tasks (including MLM and denoising) to optimize code understanding and generation.

[0046] (9) PLBART: Based on denoising autoencoders, it is pre-trained on Java and Python function datasets and employs noise strategies such as label masking, deletion and padding.

[0047] (10) SPT-Code: Based on sequence-to-sequence architecture, it combines code sequence, AST structure and natural language input, and enhances code semantics and structural representation through MASS, AST structure prediction and method name generation tasks, adapting to code search tasks.

[0048] Experimental results: Table 2 presents the experimental results of the proposed cross-training code search method (CTM) applied to code search tasks on various baseline models. Based on the experimental results, the following observations and analyses can be drawn.

[0049] The CTM method demonstrated superior performance across multiple evaluation metrics. Specifically, after introducing CTM, GraphCodeBERT's MRR improved from 71.2% to 72.6%, and its Top-1 accuracy increased by approximately 1.7 percentage points. PLBART and UniXcoder achieved performance gains of approximately 0.2% and 0.5% respectively in the Top-1 metric. Notably, even with the already high-performing CoCoSoDa model, CTM still delivered significant improvements, increasing Top-1 accuracy from 65.6% to 66.2%, achieving an MRR of 75.5%, and improving Top-10 accuracy to 91.1%. It is worth noting that despite slight fluctuations in performance across different dataset distributions, the CTM method consistently maintained a steady improvement trend. Systematic experiments on multiple datasets, including CSN, AdvTest, CosQA, and the self-constructed GPD, show that CTM has good adaptability to mainstream models such as RoBERTa, CodeBERT, and UniXcoder, and can significantly improve code search performance across multiple languages ​​and scenarios. In summary, the CTM method can effectively enhance the model's ability to express feature semantics, has strong generalization ability, and can significantly improve the overall performance of code search tasks.

[0050] Table 2

[0051] To further verify the effectiveness of the CTM method under different data distributions and complex application scenarios, this invention systematically evaluated the performance of various mainstream models on the self-constructed GPD dataset and two more challenging public datasets, AdvTest and CosQA. Compared with CSN, AdvTest uses obfuscation of function names and variables to examine the robustness and generalization ability of the model; CosQA provides query-code pairs that are closer to real-world development scenarios, better reflecting the model's adaptability in actual code search.

[0052] The experimental results are shown in Table 3. After introducing the CTM method, the performance of most models on the three datasets mentioned above was improved. Specifically, on the GPD dataset, the average MRR of the models improved by 1.4%, and the Top-1 accuracy improved by 1.8%. Taking GraphCodeBERT as an example, its MRR increased from 71.2% to 72.6%, and its Top-1 accuracy improved by 1.7 percentage points; PLBART and UniXcoder improved by 0.2% and 0.5% respectively in the Top-1 metric. Even on the CoCoSoDa model, whose original performance was already quite high, CTM still brought significant performance improvements, with its Top-1 accuracy increasing from 65.6% to 66.2%, MRR increasing to 75.5%, and Top-10 accuracy reaching 91.1%.

[0053] On the AdvTest dataset, UniXcoder achieved improvements of 2.5%, 2.2%, and 2.9% in MRR, Top-1, and Top-5 metrics, respectively, surpassing CoCoSoDa in overall performance. PLBART showed even more significant improvements, reaching 3.0%, 2.7%, and 3.4%, respectively. On the CosQA dataset, RoBERTa (CTM), GraphCodeBERT (CTM), and UniXcoder (CTM) all significantly outperformed their respective original versions. It should be noted that CodeBERT's MRR decreased slightly by 0.1% on CosQA, possibly due to the smaller dataset size, which did not fully realize the potential of CTM in pre-trained representation optimization.

[0054] In summary, the CTM method demonstrates stable and significant performance advantages in multilingual, multi-distribution, and multi-task scenarios, effectively enhancing the model's ability to model high-order and low-order semantic relationships, and further verifying its versatility and practical value in real-world complex applications.

[0055] Table 3

[0056] To verify the effectiveness of the CTM method in the pre-training stage, this invention uses CoCoSoDa as the base model and introduces the Cross approach from the CTM strategy for pre-training on six programming language subsets of the CSN dataset. Model evaluation is then conducted on the same dataset. Experimental results are shown in Table 4. CTM pre-training significantly improves MRR performance across languages. Although the overall gain is relatively limited, it still exhibits stable consistency, further demonstrating the transferability and effectiveness of CTM in the pre-training stage.

[0057] In evaluations on three other datasets, the CTM pre-trained model also demonstrated superior performance compared to the original model: an MRR of 70.0% on the CosQA dataset; 41.9% on the AdvTest dataset; and 75.9% on the GPD dataset, representing a 0.9 percentage point improvement over the original CoCoSoDa model. These results outperform several existing baseline models, further demonstrating that the CTM method can bring stable performance gains during the pre-training phase.

[0058] Table 4

[0059] Furthermore, this invention also evaluated the average MRR of the CTM pre-trained model in six languages ​​in a zero-shot scenario. The results are shown in Table 5, reaching 77.7%, significantly exceeding all comparative models and surpassing CoCoSoDa (73.8%) without CTM. This result demonstrates that CTM can still significantly improve the model's semantic generalization ability and cross-language retrieval performance without any downstream fine-tuning.

[0060] In summary, CTM is not only effective in the fine-tuning stage, but can also be successfully applied to the pre-training stage, demonstrating good versatility and cross-task transferability, and verifying its wide applicability and significant performance advantages as a unified training strategy in code search tasks.

[0061] Table 5

[0062] This embodiment also includes the following analysis: 1) Ablation studies: To verify whether CTM truly delivers a performance improvement, this invention designed a series of ablation experiments. Based on the conventional method (Normal), the core functionality of CTM is gradually introduced, and each step is evaluated separately to analyze the impact of each strategy on model performance. All non-Normal settings employ a cross-validation approach for negative example expansion to ensure the uniformity and comparability of the experiments.

[0063] Specifically, this invention divides the gradual evolution process of CTM into three settings: Extend: Directly extends the negative examples (i.e., code snippets) in contrastive learning on top of Normal. These negative examples are generated from the current model, rather than being obtained from historical checkpoints; w / o gradient: An improvement on Extend, the source of negative examples is still the current model, but gradients are not accumulated when generating negative examples, thus avoiding interference with the training path of the main model; CTM (the method in this embodiment): The main difference from w / o gradient is that CTM uses the best checkpoint model saved in the previous epoch to generate negative examples, thereby achieving model state decoupling and improving the stability and diversity of negative examples.

[0064] Table 6

[0065] As shown in Table 6, this invention systematically evaluated the impact of different negative example generation methods on model performance in the multilingual code search task of the GraphCodeBERT model. Experimental results show that the Extend method, which directly expands negative examples, exhibits performance degradation on all six programming languages, with the average MRR decreasing from 71.3% to 70.0%. When gradient backpropagation (without gradient) is disabled, model performance recovers, with the average MRR increasing to 72.9%. Furthermore, after adopting the CTM configuration that generates negative examples based on historical best checkpoints, the model achieves optimal MRR on all languages, with the average increasing to 73.3%, a 2.0 percentage point improvement compared to the original model. These results indicate that the effectiveness of the negative example expansion strategy is highly dependent on its generation method. Directly introducing additional negative examples may cause training perturbations, leading to performance degradation; while stable negative examples generated based on historical best models have better semantic quality and can fully leverage the effect of negative example expansion, thereby achieving a more significant performance improvement. This finding further verifies the key role and robustness of the CTM mechanism in the negative example construction process.

[0066] 2) Hyperparameter analysis To evaluate the sensitivity of the CTM method to key hyperparameters, this invention focuses on the impact of two hyperparameters, temperature coefficient and learning rate, on model performance. Figure 2 and Figure 3 The performance changes of the model in four metrics—MRR, Top-1, Top-5, and Top-10—are shown under different settings.

[0067] In terms of temperature coefficient ( Figure 2 This invention tested the temperature coefficient. Multiple values ​​ranging from 0.01 to 0.10. The results show that when When the MRR is in the range of [0.03, 0.05], the model performance is most stable, with the average MRR remaining above 73%; while when When the temperature coefficient exceeds 0.05, the model performance begins to decline significantly, with the Top-1 and MRR indices showing the largest decreases, reaching a maximum drop of 2.6% and 2.3%, respectively. This trend indicates that a smaller temperature coefficient helps to enhance the discrimination between positive and negative samples and improve the ability to distinguish between semantically similar but different code segments.

[0068] Regarding the learning rate ( Figure 3This invention tested multiple configurations ranging from 5e-6 to 1e-4. The results show that when the learning rate is set within the range of [1e-5, 2e-5], the model exhibits stable performance across all metrics, with an average MRR remaining above 72%, consistent with existing best practices. However, when the learning rate exceeds 2e-5, model performance significantly degrades, with both Top-1 and Top-5 accuracies decreasing by more than 3%, possibly due to unstable training or overfitting caused by excessively large parameter updates.

[0069] In summary, the CTM method is highly sensitive to both the temperature coefficient and the learning rate; appropriately setting these two parameters can further unlock the model's potential. Based on the experimental results above, this invention fixes the temperature coefficient in subsequent experiments. The learning rate is set to It is used as the default configuration for unified evaluation.

[0070] This invention proposes a cross-training strategy for the fine-tuning stage of code search tasks. This method introduces an alternating optimization mechanism to further explore the potential of the pre-trained model during fine-tuning, thereby fully releasing its representational capabilities in code search scenarios. Fine-tuning experiments were conducted on mainstream code models across multiple datasets. Results show that CTM can significantly improve code search performance under different model structures and task settings, demonstrating good adaptability and versatility. Furthermore, this invention integrates CTM into the pre-training stage, performing further pre-training on CoCoSoDa. Experimental results show that CTM not only effectively enhances the code synthesis capabilities of CoCoSoDa but also achieves superior performance in zero-shot code search tasks. This indicates that CTM also has significant potential in improving the multilingual generalization ability of models. In summary, CTM, as a simple and effective training strategy, can bring performance improvements in both the fine-tuning and pre-training stages, providing new ideas and methods for further optimizing code representation learning.

[0071] This invention proposes a cross-training method that combines in-policy and non-policy training mechanisms during the fine-tuning phase. It utilizes a cache queue to store historical code snippets and employs a historical best-checkpoint model to generate diverse negative examples, significantly improving the model's ability to discriminate semantically similar code and optimizing code search performance. Simultaneously, a cross-training strategy is used for training scheduling, reducing training complexity and instability and enhancing the model's robustness. Comprehensive experiments were conducted on multiple public datasets (including CodeSearchNet, CosQA, and AdvTest). Experimental results show that the model using this method exhibits significant improvements in mean reciprocal rank (MRR) and Top-k accuracy, especially demonstrating good generalization ability in multilingual and complex scenarios (such as real user queries and obfuscated code). Furthermore, this method can be extended to the pre-training phase to further enhance the model's performance in zero-shot code search tasks, providing an efficient and robust solution for code search tasks.

[0072] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A cross-training method for code search, characterized in that, Includes the following steps: Natural language queries and code snippets are feature-encoded to generate semantic representation vectors; Based on the semantic representation vector, cross-training is performed to obtain the contrastive learning loss for the In-Policy stage and the Non-Policy stage, respectively. Cross-training includes: A Cross strategy is adopted, which executes the In-Policy phase and the Non-Policy phase in a periodic alternation manner; The current execution phase type is determined based on the parity of the training cycle index; A periodic scheduling function is defined, which is used to execute the In-Policy training phase in even-numbered training cycles and the Non-Policy training phase in odd-numbered training cycles. According to the periodic scheduling function, the execution order of the In-Policy training phase and the Non-Policy training phase is switched. In the In-Policy phase, negative examples are generated based on code snippets stored during the historical training process; During the In-Policy phase, the similarity between the historical semantic vector and the current query vector is calculated to obtain the similarity of historical negative samples. Based on the historical negative sample similarity, a first contrastive learning loss function is constructed to update the model parameters; In the Non-Policy phase, code snippets within the current batch are used as negative examples; During the Non-Policy phase, the code snippets from the current batch are input into the current model to obtain the current semantic vector; The similarity between the current semantic vector and the current query vector is calculated to obtain the similarity of the current negative sample. Based on the current negative sample similarity, a second contrastive learning loss function is constructed to update the model parameters; The model parameters are optimized based on the contrastive learning loss, and a code snippet that matches the semantics of a natural language query is output.

2. The method according to claim 1, characterized in that, The feature encoding process includes: The input sequences of the natural language queries and code fragments are subject to length restrictions. The length-limited natural language queries and code snippets are input into a pre-trained model based on the Transformer architecture; Low-dimensional semantic representation vectors containing semantic information are generated by pre-training models.

3. The method according to claim 1, characterized in that, The expression for the first contrastive learning loss function is: ； In the formula, Represents the annotation vector. This indicates the positive sample code representation. This represents the negative sample code. The cosine similarity between vectors is represented by... This indicates temperature hyperparameters. Indicates the first in the batch The positive sample code corresponding to each data point represents, Indicates batch size, Indicates the size of the cache queue.

4. The method according to claim 1, characterized in that, The expression for the second contrastive learning loss function is: ； In the formula, Represents the annotation vector. This indicates the positive sample code representation. The cosine similarity between vectors is represented by... This indicates temperature hyperparameters. Indicates the first in the batch The positive sample code corresponding to each data point represents, Indicates the batch size.

5. The method according to claim 1, characterized in that, Model optimization and querying include: Based on the current training stage, select the appropriate contrastive learning loss to optimize the model; Use cosine similarity to calculate the semantic distance between the query and the candidate code snippet; Based on the semantic distance, the code fragment most relevant to the query is retrieved from the candidate code set.

Images