Retrieval apparatus, retrieval method, and program
The search device leverages a large-scale language model to generate and compare explanatory text features for source codes, addressing the accuracy issues in repository-level code tasks by focusing on meaningful content rather than structure, thereby improving code search and generation precision.
Patent Information
- Application Number
- JP2024030985
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2024-03-01
- Publication Date
- 2025-09-11
AI Technical Summary
Existing large-scale language models (LLMs) face challenges in accurately identifying related code at the repository level due to input length limitations and low accuracy in repository-level code generation and search tasks, such as code generation, correction, and comprehension, particularly in tasks requiring referencing multiple files.
A search device utilizing a large-scale language model (LLM) to generate explanatory text for source codes, calculate features, evaluate similarity, and output related code candidates based on deep learning techniques, focusing on the meaning rather than structural similarity.
Improves the accuracy of identifying related source code by comparing meaningful explanations generated by the LLM, enhancing the precision of code search and generation tasks.
Smart Images
Figure 2025133195000001_ABST
Abstract
Description
[Technical Field]
[0001] The present invention relates to a search device, a search method, and a program. [Background technology]
[0002] Japan faces a shortage of IT human resources. One solution is to support software development using large-scale language models (LLMs). Currently, LLMs' code generation capabilities have dramatically improved, achieving high accuracy in understanding and generating single source code (hereafter simply referred to as "code"). However, there has been little research at the repository level, which is closer to actual software development, and LLMs' repository-level code generation capabilities are very low. Here, repository-level code refers to, for example, a large collection of code related to software with multiple functions developed by multiple people. The bug resolution rate of the SWE-bench, a repository-level bug fix benchmark, is only 1.96%. Repository-level code-related tasks (such as code generation, code correction, code comprehension, code search, and other code-related tasks typically performed in programming) require referencing many files.
[0003] On the other hand, most LLMs are Transformer-based models, which means they have input length limitations and cannot accept many files. Typically, bug fixes and feature changes require modifying only a portion of the code related to the bug or feature change. Therefore, inputting only a portion of the code into an LLM, rather than all of the code, is likely to address the LLM's input length limitations. [Prior art documents] [Non-patent literature]
[0004] [Non-Patent Document 1] Tianyang Liu, Canwen Xu, and Julian McAuley, "Repobench: Benchmarking repository-level code autocompletion systems", arXiv preprint arXiv:2306.03091, 2023 Summary of the Invention [Problem to be solved by the invention]
[0005] However, it is not easy to manually identify multiple pieces of code (hereinafter referred to as "related code") related to a certain bug fix or function change from a large code collection. Therefore, it is possible to identify related code by code search based on a portion of code that can be manually identified.
[0006] In existing research on code search, features (such as embedded representations of language models) of the code to be generated using LLM and the code candidates that may be related to it are obtained, and similarity is calculated based on the features.
[0007] However, the accuracy of code search is not high even in SWE-bench or RepoBench (Non-Patent Document 1), a repository-level code completion and search benchmark. Therefore, even with existing code search technologies, it is difficult to identify related code with high accuracy.
[0008] The present invention has been made in view of the above points, and has as its object to improve the accuracy of identifying source code related to a certain source code. [Means for solving the problem]
[0009] In order to solve the above problem, the search device has an explanatory text generation unit configured to generate explanatory text for each of a first source code related to a certain software and multiple second source codes related to the software using a large-scale language model, a feature calculation unit configured to calculate features of each of the explanatory texts, a similarity evaluation unit configured to evaluate the similarity between each of the features related to the second source code and the features related to the first source code, and an output unit configured to output information based on the results of the similarity evaluation. [Effects of the Invention]
[0010] The accuracy of identifying source code related to a certain source code can be improved. [Brief explanation of the drawings]
[0011] [Figure 1] FIG. 10 is a diagram illustrating a change to be made to an application. [Figure 2] FIG. 10 is a diagram illustrating an example of client-side source code of an application before modification. [Figure 3] FIG. 10 is a diagram illustrating an example of server-side source code of an application before modification. [Figure 4] FIG. 10 is a diagram illustrating an example of client-side source code of an application after modification. [Figure 5] FIG. 10 is a diagram illustrating an example of server-side source code of an application after modification. [Figure 6] FIG. 1 is a diagram illustrating a situation assumed in the present embodiment. [Figure 7] 1 is a diagram illustrating an example of a hardware configuration of a search device 10 according to an embodiment of the present invention. [Figure 8] 1 is a diagram illustrating an example of a functional configuration of a search device 10 according to an embodiment of the present invention. [Figure 9] 10 is a flowchart illustrating an example of a processing procedure executed by the search device 10. [Figure 10] FIG. 10 is a diagram illustrating an example of a prompt. [Figure 11] 10 is a diagram showing a specific example of processing executed by the search device 10. FIG. DETAILED DESCRIPTION OF THE INVENTION
[0012] In conventional chord search technology, when acquiring the features of each chord candidate, the original chord information is used as is, which means that even the chord structure tends to be converted into a feature vector. The inventors of this application believe that one of the reasons for the low accuracy of conventional chord search technology is that it is unable to properly evaluate the similarity between the meanings of each chord.
[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[0014] For example, consider a case where you want to make a change to only the member registration function in a piece of software (hereinafter referred to as an "app") that has a login function that accepts logins via a login screen and a member registration function that accepts member registration via a member registration screen, as shown in Figure 1. In the example of Figure 1, a functional change is envisioned, such as adding a confirmation password to the member registration function.
[0015] FIG. 2 is a diagram showing an example of the client-side source code of the application before modification, and FIG. 3 is a diagram showing an example of the server-side source code of the application before modification. As is clear from FIGS. 2 and 3, the source code for the login function and member registration function of the application consists of four files: login.html and register.html on the client side, and form.py and views.py on the server side. In this example, the client-side source code is an HTML file that defines the screen configuration, and the server-side source code is a file that defines processing based on values entered on the screen. views.py has been omitted for convenience.
[0016] When making the changes to the member registration function as explained in Figure 1, the client-side source code will be changed as shown in Figure 4, and the server-side source code will be changed as shown in Figure 5. In Figures 4 and 5, the parts surrounded by dashed lines correspond to the changes. In other words, the changes will occur to register.html on the client side, and to form.py on the server side.
[0017] While register.html, which is the source code for the screen, can be easily identified manually as a target for modification, it is difficult to identify form.py, which is a server-side program. Therefore, in this embodiment, as shown in Figure 6, an example will be described in which the ResigterForm::clean function in form.py, which needs to be modified in conjunction with the modification of register.html, is identified (searched) as source code related to register.html (hereinafter referred to as "related code").
[0018] Fig. 7 is a diagram showing an example of the hardware configuration of a search device 10 according to an embodiment of the present invention. The search device 10 in Fig. 7 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, which are all interconnected via a bus B.
[0019] A program that realizes processing in the search device 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.
[0020] When an instruction to start a program is received, the memory device 103 reads and stores the program from the auxiliary storage device 102. The processor 104 is a CPU or a GPU (Graphics Processing Unit), or a CPU and a GPU, and executes functions related to the search device 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.
[0021] FIG. 8 is a diagram illustrating an example of a functional configuration of a search device 10 according to an embodiment of the present invention. In FIG. 8, the search device 10 includes an input unit 11, an explanation generation unit 12, a feature calculation unit 13, a similarity evaluation unit 14, and an output unit 15. The similarity evaluation unit 14 utilizes a large-scale language model (LLM 121), which is a machine learning model that has previously trained a large-scale language model. That is, the LLM 121 is previously trained to input text and output text corresponding to the input text. These units and the LLM 121 are realized by a process executed by the processor 104 of one or more programs installed in the search device 10. Note that the units and the LLM 121 in FIG. 8 may be distributed and arranged across multiple computers. That is, the search device 10 may be configured by multiple computers.
[0022] The following describes the processing procedure executed by the search device 10. Fig. 9 is a flowchart for explaining an example of the processing procedure executed by the search device 10. Fig. 11 is a diagram showing a specific example of processing executed by the search device 10.
[0023] In step S101, the input unit 11 inputs a query code and a related file candidate group. The query code refers to source code from which related code is searched. In the example of FIG. 6, the source code written in register.html (FIG. 4) modified in accordance with the above-described functional change of the application corresponds to the query code. The related file candidate group refers to a group of files storing source code that is a candidate for related code. In the example of FIG. 6, register.html, login.html, form.py, and views.py correspond to the related file candidate group. Note that register.html, which stores the query code, is included in the related code candidate group in consideration of the case where source code of a portion of a certain file is input as the query code. In this case, code related to that portion in the file can also be related code. Note that FIG. 6 illustrates an example in which all source code in the modified register.html is input as the query code.
[0024] Next, the input unit 11 divides the source code stored in each related file candidate into source code for each predetermined processing unit (S102). The predetermined processing unit is, for example, a function or a method. Hereinafter, each divided source code will be referred to as a "related code candidate." Note that source code that does not include a predetermined processing unit (here, a function or a method), such as source code stored in an HTML file, may be treated as a related code candidate on a file-by-file basis without being divided.
[0025] Next, the explanation generator 12 generates a prompt requesting the LLM 121 to generate an explanation of the contents of the query code (S103).
[0026] An example of a prompt is shown in Figure 10. In the prompt in Figure 10, the query code is transcribed as is in the <code content> section.
[0027] Next, the explanation generation unit 12 inputs the generated prompt to the LLM 121 and acquires the text (explanation) output from the LLM 121 (S104). In Fig. 11, an example is shown in which explanation d1 is generated as the explanation of register.html.
[0028] Next, the feature amount calculation unit 13 calculates a feature vector (feature amount) for the description using an embedding technique based on a deep learning model (S105).
[0029] Next, loop processing L1 is executed. In loop processing L1, steps S106 to S109 are executed for each related code candidate. The source code that is the processing target in loop processing L1 will be referred to as "target code" hereinafter.
[0030] In step S106, the explanation generator 12 generates a prompt requesting the LLM 121 to generate an explanation of the contents of the target code. The generated prompt may have the same format as that shown in FIG.
[0031] Next, the explanatory sentence generation unit 12 inputs the generated prompt to the LLM 121 and acquires the text (explanatory sentence) output from the LLM 121 (S107). Note that Fig. 11 shows an example in which explanatory sentence d2 is generated when the source code of the ResigterForm::clean function in form.py is the target code.
[0032] Next, the feature amount calculation unit 13 calculates a feature vector (feature amount) for the description using an embedding technique based on a deep learning model (S108).
[0033] Next, the similarity evaluation unit 14 evaluates the similarity between the feature vector related to the query code and the feature vector related to the target code (S109). For example, the similarity evaluation unit 14 calculates the cosine similarity between the two feature vectors.
[0034] When the loop process L1 is executed for all source codes (related code candidates) included in the related code candidate group, the output unit 15 outputs information based on the results of the evaluation of the similarity between each related code candidate and the query code (S110). The output information is not limited to a predetermined format as long as the user can identify the related code. For example, information indicating the related code candidate with the highest similarity may be output as the related code. Alternatively, information indicating some of the related code candidates with the highest similarity (up to Nth) may be output. Alternatively, information in which all related code candidates are sorted in descending order of similarity may be output.
[0035] In the above, in order to show an example of searching for related code on a function or method basis, an example was given in which each related file candidate was divided into specified processing units and related code candidates were generated, but if it is sufficient to be able to search for related code on a file basis, related code candidates may also be generated on a related file candidate basis.
[0036] Furthermore, in the above example, the query code is the source code after correction, but the source code before correction may be used as the query code.
[0037] In addition, in the conventional method, since the feature values of each source code are compared rather than the feature values of the description, the structure of the source code tends to be converted into a feature vector. As a result, in the example of this embodiment, it is possible that the similarity between the query code register.html and login.html, which is an HTML file similar to the query code register.html, will be high.
[0038] As described above, according to this embodiment, the LLM 121 converts code into a concise description and then converts it into a feature vector, which allows comparison based on the meaning of the code rather than its structure. As a result, the accuracy of identifying source code related to a certain source code can be improved.
[0039] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and variations are possible within the scope of the gist of the present invention as described in the claims. [Explanation of symbols]
[0040] 10 Search Device 11 Input section 12 Description generation section 13 Feature calculation unit 14 Similarity Evaluation Section 15 Output section 100 Drive device 101 Recording media 102 Auxiliary storage device 103 Memory Device 104 processors 105 Interface Device 121 LLM B Bus
Claims
1. an explanatory sentence generation unit configured to generate explanatory sentences for a first source code related to a piece of software and a plurality of second source codes related to the software using a large-scale language model; a feature calculation unit configured to calculate a feature of each of the descriptions; a similarity evaluation unit configured to evaluate a similarity between each of the features related to the second source code and a feature related to the first source code; an output unit configured to output information based on the result of the similarity evaluation; A search device comprising:
2. the explanatory text generation unit is configured to input, for each of the first source code and the second source code, the source code and text requesting generation of an explanatory text for the source code into the large-scale language model.
2. The search device according to claim 1.
3. the second source code is source code obtained by dividing source code related to the software into predetermined processing units; characterized in that The search device according to claim 1.
4. the first source code is source code to which modifications corresponding to functional changes related to the software have been applied; 2. The search device according to claim 1.
5. an explanation generation step of generating explanations for a first source code related to a certain piece of software and a plurality of second source codes related to the software using a large-scale language model; a feature calculation step of calculating feature amounts of each of the descriptions; a similarity evaluation step of evaluating a similarity between each of the features related to the second source code and the features related to the first source code; an output step of outputting information based on the result of the similarity evaluation; A search method characterized by being executed by a computer.
6. an explanation generation step of generating explanations for a first source code related to a certain piece of software and a plurality of second source codes related to the software using a large-scale language model; a feature calculation step of calculating feature amounts of each of the descriptions; a similarity evaluation step of evaluating a similarity between each of the features related to the second source code and the features related to the first source code; an output step of outputting information based on the result of the similarity evaluation; A program characterized by causing a computer to execute the above.