Software analysis device, software analysis method, and program

Abstract

This software analysis device comprises: a first calculation unit configured to calculate, for each of one or more first character strings representing, in a natural language, the content of a change performed on a source code of software, the degree of association with a second character string representing, in a natural language, the content of a change to be performed on the software; a second calculation unit configured to calculate, for each location of the source code having been changed in relation to the content of the change represented by the first character string, the total sum of the degrees of association related to all of the first character strings corresponding to the location; and an output unit configured to output information indicating the location in a mode corresponding to the total sum, thereby enabling acquisition of a portion related to a character string expressed in a natural language from a source code.

Description

Software analysis device, software analysis method and program

[0001] The present invention relates to a software analysis device, a software analysis method, and a program.

[0002] To quickly respond to diverse user needs, there is a need to shorten software development times. However, because software development involves a large amount of non-routine tasks, shortening development times through automation is difficult. Large Language Models (LLMs) are a technology that has the potential to solve these challenges. By training large amounts of data using large-scale models, LLMs enable AI chat and coding assistance.

[0003] To introduce LLM into software development, it is necessary to provide information about the software under development to the LLM. However, since LLM has a token length limit, it is difficult to provide all the information about a large piece of software. Therefore, it is necessary to appropriately extract relevant information according to the task to be achieved with LLM.

[0004] For example, consider a case where you want to make a change only to the member registration function in an app that has a login function that accepts logins via a login screen and a member registration function that accepts member registration via a member registration screen, as shown in Figure 1. In the example of Figure 1, a change is expected to be made to the member registration function, such as adding a confirmation password.

[0005] FIG. 2 shows an example of the client-side source code of the application before modification, and FIG. 3 shows an example of the server-side source code of the application before modification. As is clear from FIGS. 2 and 3, the source code related to the login function and member registration function of the application consists of four files: login.html and register.html on the client side, and form.py and views.py on the server side. In this example, the client-side source code is an HTML file that defines the screen configuration, and the server-side source code is a file that defines processing based on values ​​entered on the screen. views.py has been omitted for convenience.

[0006] When making the changes to the member registration function as described in Figure 1, the client-side source code is changed as shown in Figure 4, and the server-side source code is changed as shown in Figure 5. In Figures 4 and 5, the parts surrounded by dashed lines correspond to the changes. In other words, the changes occur to register.html on the client side, and to form.py on the server side.

[0007] To make such changes to the LLM, instructions to the LLM (hereinafter referred to as "prompts") and the source code before the changes must be input to the LLM. For example, a prompt could be, "Please add a password confirmation function to the membership registration screen." However, since source code is usually large, if the source code is input to the LLM as is, there is a possibility that the token length limit of the LLM will be exceeded.

[0008] Therefore, it is possible to provide only the portion of the source code related to the prompt (hereinafter referred to as "context") to the LLM, rather than the entire source code, as shown in Figure 6. Figure 6 shows how a portion of register.html and a portion of form.py are extracted as context from login.html, register.html, form.py, and views.py, and the context is provided to the LLM along with the prompt, and the modified register.html and modified form.py are output from the LLM.

[0009] Since proper extraction of context from a huge amount of source code requires knowledge and experience of the source code, automation of this task is being considered.

[0010] A conventional technique for acquiring context is disclosed in Non-Patent Document 1. In this technique, a document containing information related to a prompt is divided into segments of a certain length, the relevance of each segment to the prompt is measured, and the segment with the highest relevance is acquired as the context.

[0011] Furthermore, although it is not a technology aimed at acquiring context, it is possible to use a program search technology, which is a related technology for acquiring related code based on natural language. In Non-Patent Document 2, the function description and the function body are converted into feature vectors using separate neural network models, and learning is performed based on the presence or absence of correlation.

[0012] "LlamaIndex - Data Framework for LLM Applications", LlamaIndex Inc., [online], Internet <URL: https: / / www.llamaindex.ai / >. Xiaodong Gu, Hongyu Zhang, and Sunghun Kim, "Deep code search", In Proceedings of the 40th International Conference on Software Engineering (ICSE '18). Association for Computing Machinery, New York, NY, USA, 933-944, 2018, [online], Internet <URL: https: / / doi.org / 10.1145 / 3180155.<3180167>

[0013] However, in source code, since the context is likely to be distributed among various files, the method of splitting into a certain length as in Non-Patent Document 1 is not appropriate. For example, in a web application, the login function is composed of a view for entering the login ID and password, a controller for distributing requests, and a model for checking with the database whether the ID and password exist, etc. These functions exist distributed in separate files.

[0014] Also, since the method of Non-Patent Document 2 requires a large amount of learning data, it is difficult to use in a project domain where the function descriptions are insufficient. The present invention has been made in view of the above points, and an object thereof is to enable acquisition of portions related to character strings expressed in natural language from source code.

[0015] In order to solve the above problem, the software analysis device has a first calculation unit configured to calculate, for each of one or more first character strings that express in natural language the content of changes made to the source code of the software, the degree of relevance between the first character string and a second character string that expresses in natural language the content of changes to be made to the software; a second calculation unit configured to calculate, for each location in the source code that has been changed in relation to the content of the changes expressed by the first character string, the sum of the degrees of relevance for all of the first character strings that correspond to that location; and an output unit configured to output information indicating the location in a manner corresponding to the sum.

[0016] It is possible to obtain portions related to strings expressed in natural language from source code.

[0017] 1 is a diagram for explaining changes desired to be made to a certain application. FIG. 2 is a diagram illustrating an example of client-side source code of an application before the change. FIG. 3 is a diagram illustrating an example of server-side source code of an application before the change. FIG. 4 is a diagram illustrating an example of client-side source code of an application after the change. FIG. 5 is a diagram illustrating an example of server-side source code of an application after the change. FIG. 6 is a diagram illustrating how context is extracted from source code and given to an LLM. FIG. 7 is a diagram illustrating an example hardware configuration of a software analysis device 10 in an embodiment of the present invention. FIG. 8 is a diagram illustrating an example functional configuration of a software analysis device 10 in an embodiment of the present invention. FIG. 9 is a diagram illustrating an example configuration of a commit history storage unit 15. FIG. 10 is a flowchart illustrating an example of processing procedures executed by the software analysis device 10. FIG. 11 is a diagram illustrating an example of the contents of dictionary D. FIG. 12 is a diagram illustrating an example of the contents of dictionary R.

[0018] An embodiment of the present invention will now be described with reference to the accompanying drawings. Fig. 7 is a diagram showing an example of the hardware configuration of a software analysis device 10 according to an embodiment of the present invention. The software analysis device 10 shown in Fig. 7 includes a drive device 100, an auxiliary storage device 102, a memory device 103, a processor 104, and an interface device 105, all of which are interconnected via a bus B.

[0019] A program that realizes processing in the software analysis device 10 is provided by a recording medium 101 such as a CD-ROM. When the recording medium 101 storing the program is set in the drive device 100, the program is installed from the recording medium 101 to the auxiliary storage device 102 via the drive device 100. However, the program does not necessarily have to be installed from the recording medium 101, but may be downloaded from another computer via a network. The auxiliary storage device 102 stores the installed program as well as necessary files, data, etc.

[0020] When an instruction to start a program is received, the memory device 103 reads and stores the program from the auxiliary storage device 102. The processor 104 is a CPU or a GPU (Graphics Processing Unit), or a CPU and a GPU, and executes functions related to the software analysis device 10 in accordance with the program stored in the memory device 103. The interface device 105 is used as an interface for connecting to a network.

[0021] 8 is a diagram showing an example of the functional configuration of a software analysis device 10 according to an embodiment of the present invention. In FIG. 8, the software analysis device 10 includes a change information extraction unit 11, a commit-specific relevance calculation unit 12, a relevance score calculation unit 13, and an output unit 14. Each of these units is implemented by a processor 104 executing one or more programs installed in the software analysis device 10. The software analysis device 10 also uses a commit history storage unit 15. The commit history storage unit 15 can be implemented using, for example, an auxiliary storage device 102 or a storage device connectable to the software analysis device 10 via a network.

[0022] 9 is a diagram showing an example of the configuration of the commit history storage unit 15. As shown in FIG. 9, the commit history storage unit 15 stores a commit message and changes for each commit or commits to a certain piece of software (hereinafter referred to as "target software"). In other words, the commit history storage unit 15 stores information indicating the correspondence between the commit message and the changes for each commit. In this embodiment, software is synonymous with a program.

[0023] A commit refers to adding a set of changes and explanations of the changes (commit message) to a repository when changes are made to the source code of the target software due to function changes, bug fixes, etc. A repository refers to a storage location in a software version control system for storing source code, change history, etc. by version (see, for example, https: / / wa3.i-3-i.info / word15664.html). In this embodiment, the commit history storage unit 15 may constitute a part of the repository.

[0024] A commit message is a string that describes (explains) the changes made to the source code in natural language (see, for example, https: / / backlog.com / ja / git-tutorial / intro / 03 / ). Changes are information that indicates the parts of the source code that have been changed. Changes are identified by file or function. For example, if a file that does not contain units that clearly distinguish the components of the source code, such as HTML files login.html and register.html in Figure 4, is changed, the file name of the file where the change occurred is the change. However, if the changes are limited to a specific node or below in the HTML tree structure, information indicating that node may also be considered the change. On the other hand, if a file composed of units (such as classes and functions) that clearly distinguish the components of the source code, such as form.py and views.py in Figure 4, is changed, the name of the lowest-level unit (such as the class name or function name) where the change occurred is the change. A single commit does not necessarily contain only one change. Figure 9 shows an example in which multiple changes are registered for each commit.

[0025] Note that in Figure 9, for units other than files (classes and functions), information indicating higher-level units that include the unit in question is also included. For example, in commit 2, at the file level, register.html, form.py, and views.py have been changed, but for form.py and views.py, the changes are further limited to class or function levels. Specifically, for form.py, it is shown that two changes have been made: def clean(self) and class Meta in class RegisterForm(forms.ModelForm). Also, for views.py, it is shown that def register(request) has been changed. Therefore, there are four changes in commit 2.

[0026] The change information extraction unit 11 extracts information indicating the correspondence between commit messages and change locations from the commit history storage unit 15 as dictionary-type data.

[0027] The commit-specific relevance calculation unit 12 calculates the relevance, which is an index showing the strength of the relevance of each commit message with a prompt. A prompt is a character string that expresses in natural language the content of changes to be made (to be made) to the target software.

[0028] The relevance score calculation unit 13 calculates, for each change, the sum of the relevance scores of all the commit messages corresponding to that change (hereinafter referred to as the "relevance score").

[0029] The output unit 14 outputs the changed parts (information indicating the changed parts) in a format according to the relevance score. For example, the output unit 14 outputs only the changed parts with the highest relevance scores.

[0030] The following describes the processing procedure executed by the software analysis device 10. FIG. 10 is a flowchart illustrating an example of the processing procedure executed by the software analysis device 10.

[0031] In step S101, the change information extraction unit 11 initializes the dictionary D with an empty set (Φ). The dictionary D is dictionary-type data for storing pairs each having a commit message as a key and a set of changes as a value.

[0032] Next, the change information extraction unit 11 executes a loop process L1 for each commit stored in the commit history storage unit 15. The loop process L1 includes step S102. The commit that is the processing target in the loop process L1 will be referred to as the "target commit" hereinafter.

[0033] In step S102, the change information extraction unit 11 adds {commit message: group of changes} to the dictionary D. {commit message: group of changes} indicates a pair of the commit message of the target commit and a set of changes in the target commit.

[0034] When the loop process L1 is executed for all commits, the generation of the dictionary D is completed and the process proceeds to step S103. Note that the dictionary D storing information such as that shown in FIG. 11 is generated from the commit history storage unit 15 in FIG.

[0035] In step S103, the commit-specific relevance calculation unit 12 initializes the dictionary R of {change location: relevance score} with an empty set (Φ) (S103). The dictionary R is dictionary-type data for storing pairs in which one change location is used as a key and the relevance score of the change location is used as a value. {change location: relevance score} indicates a pair of a change location and a relevance score.

[0036] Next, the software analysis device 10 executes a loop process L2 for each {commit message: group of changes} included in dictionary D. The loop process L2 includes step S104 and a loop process L3. The {commit message: group of changes} element of dictionary D that is the processing target in the loop process L2 is referred to as a "target pair d."

[0037] In step S104, the commit-specific relevance calculation unit 12 calculates Sim (commit message, prompt) and assigns the calculation result to r. Sim(x, y) is a function that measures the relevance between natural languages ​​x and y and can be calculated using existing technologies (e.g., cosine similarity of embedded expressions) such as BERT (see, for example, https: / / huggingface.co / bert-base-uncased). Therefore, r indicates the relevance between the commit message and the prompt. The prompt is a character string entered by the user that expresses in natural language a change instruction for the source code of the target software for a large language model (LLM). For example, when making the change described in FIG. 1, an example of a prompt would be, "Please add a password confirmation function to the membership registration screen."

[0038] Next, the relevance score calculation unit 13 executes a loop process L3 for each change part included in the change part group of the target pair d. The loop process L3 includes steps S105 to S107. The change part being processed in the loop process L3 is referred to as a "part."

[0039] In step S105, the relevance score calculation unit 13 determines whether a key matching the part has already been registered in the dictionary R. If a key matching the part has not been registered in the dictionary R (No in S105), the relevance score calculation unit 13 initializes R[part] to 0 (S106). R[part] is a value in R corresponding to the key part.

[0040] If the answer is Yes in step S105, or following step S106, the relevance score calculation unit 13 adds the relevance r to R[part] (S107). That is, the relevance score for part is the sum of the relevance r for each commit in which part is a change location.

[0041] Fig. 12 is a diagram showing an example of the contents of the dictionary R. In Fig. 12, the relevance level r is based on existing technology such as BERT.

[0042] 12 (1) shows the contents of dictionary R at the time when loop processing L2 for commit 1 is completed. In this example, the relevance between the prompt and the commit message of commit 1 ("Create login function") is 0.2, and for each change in commit 1, a pair of the change and 0.2 is registered in dictionary R.

[0043] 12 (2) shows the contents of dictionary R at the end of loop L2 for commit 2. In this example, the relevance r between the prompt and the commit message of commit 2 ("Create a member registration function") is 0.8, and for each change in commit 2, a pair of the change and 0.8 is added to dictionary R.

[0044] Next, the output unit 14 sorts the elements of the dictionary R (pairs of the change portion and the relevance score) in descending order of the relevance score (S108).

[0045] Next, the output unit 14 outputs the keys (changed parts) of the elements having the top N relevance scores in the dictionary R (S109), where N is a threshold value indicating how many relevance scores from the top are to be output, and is set in advance by the user.

[0046] For example, when N=1, in the state (2) of Fig. 12, "register.html", "def clean(self)", "class Meta", and "def register(request)", which have the highest relevance scores, are output. The user can input the source code corresponding to these changes as context into the LLM along with the prompts, causing the LLM to make changes to the source code of the target software in response to the prompts.

[0047] Although the above example shows that only the top N changes are output, it is also possible to output all pairs of changes and their relevance scores in descending order of relevance score. The decision of how many changes should be used as contexts may be left to the user. Furthermore, N may be given as a threshold for the relevance score, rather than a threshold for the order. In this case, the output unit 14 may output changes whose relevance scores are N or greater.

[0048] As described above, this embodiment makes it possible to obtain the part (context) related to the character string (prompt) expressed in natural language from the source code, which is expected to promote the use of LLM in software development, for example.

[0049] This embodiment utilizes the characteristic that source code is generally changed on a function or task basis in a commit. For example, in a web application, commits are made at the granularity of a single function, such as a login function, or on a task basis, such as a bug fix. In other words, functions changed in a single commit are likely to be related in some way in terms of function or task, and it is believed that linking them with prompts can provide an appropriate context.

[0050] In this embodiment, the commit-specific relevance calculation unit 12 is an example of a first calculation unit. The relevance score calculation unit 13 is an example of a second calculation unit. The commit message is an example of a first character string. The prompt is an example of a second character string.

[0051] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to such specific embodiments, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0052] REFERENCE SIGNS LIST 10 Software analysis device 11 Change information extraction unit 12 Commit-specific relevance calculation unit 13 Relevance score calculation unit 14 Output unit 15 Commit history storage unit 100 Drive device 101 Recording medium 102 Auxiliary storage device 103 Memory device 104 Processor 105 Interface device B Bus

Claims

1. A software analysis device comprising: a first calculation unit configured to calculate, for each of one or more first character strings expressing in natural language the content of changes made to the source code of the software, a degree of relevance between the first character string and a second character string expressing in natural language the content of changes to be made to the software; a second calculation unit configured to calculate, for each location in the source code that has been changed in relation to the content of the changes expressed by the first character strings, a sum of the degrees of relevance for all of the first character strings that correspond to that location; and an output unit configured to output information indicating the location in a manner corresponding to the sum.

2. The software analysis device according to claim 1, characterized in that the second calculation unit is configured to calculate, for each of the first strings, the sum of the relevance levels for all of the first strings corresponding to that portion based on information indicating a correspondence between each of the first strings and the changed portions in relation to the content of the change represented by that first string.

3. The software analysis device according to claim 1 or 2, wherein the second character string is a character string given as a prompt to a large-scale language model.

4. A software analysis method characterized in that a computer executes the following steps: a first calculation step for calculating, for each of one or more first character strings expressing in natural language the content of changes made to the source code of the software, a degree of relevance with a second character string expressing in natural language the content of changes to be made to the software; a second calculation step for calculating, for each location in the source code that has been changed in relation to the content of the changes expressed by the first character strings, the sum of the degrees of relevance for all of the first character strings that correspond to that location; and an output step for outputting information indicating the location in a manner corresponding to the sum.

5. A program that causes a computer to execute the following steps: a first calculation step for calculating, for each of one or more first character strings expressing in natural language the content of changes made to the source code of the software, a degree of relevance with a second character string expressing in natural language the content of changes to be made to the software; a second calculation step for calculating, for each location in the source code that has been changed in relation to the content of the changes expressed by the first character strings, the sum of the degrees of relevance for all of the first character strings that correspond to that location; and an output step for outputting information indicating that location in a manner corresponding to the sum.

Images