Software development support system and software development support method

Abstract

To improve the development efficiency of software development by using automated program code generation by AI. [Solution] The software development support system manages a software development project by dividing it into multiple development cycles. In the first development cycle of the multiple development cycles, the software development support system calculates a score for evaluating the first code of the software generated using either artificial intelligence (AI) or a code generation method other than artificial intelligence, based on predetermined evaluation criteria. Based on the calculated score, the software development support system then determines whether to use artificial intelligence, another artificial intelligence, or a code generation method other than artificial intelligence when generating the second code in the second development cycle following the first development cycle of the multiple development cycles.

Description

Technical Field

[0004] ,

[0006] , , , ,

[0005] , , , ,

[0001] The present invention relates to a software development support system and a software development support method.

Background Art

[0002] In recent years, software development using generative AI (Artificial Intelligence) has been progressing. For example, Patent Document 1 discloses a technique related to automatically generating program code that causes a learning model to generate program code that satisfies text-based requirements and specifications input by a user and providing it to the user.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Generally, the learning level of a generative AI learning model varies depending on factors such as the model version of the generative AI and the presence or absence of fine-tuning (additional learning). Therefore, in software development, by generating program code with an inappropriate generative AI, more trials may be repeated and more man-hours may be required until program code that satisfies the requirements and specifications is generated compared to manual generation, and conversely, there is a risk that development efficiency may decrease.

[0005] The present invention has been made in view of the above problems, and an object thereof is to improve the development efficiency of software development using automatic generation of program code by generative AI.

Means for Solving the Problems

[0006] To achieve the above-mentioned objectives, the present invention, in one embodiment, provides a software development support system for assisting in the development of software, wherein the software development support system comprises a processor and memory, the processor manages the software development project by dividing it into a plurality of development cycles, calculates a score for evaluating the first code of the software generated using either a generation AI (Artificial Intelligence) or a code generation method other than the generation AI in the first development cycle of the plurality of development cycles based on predetermined evaluation criteria, and determines, based on the score, whether to use the generation AI, another generation AI, or a code generation method other than the generation AI when generating the second code in the second development cycle following the first development cycle of the plurality of development cycles. [Effects of the Invention]

[0007] According to the present invention, for example, it is possible to improve the development efficiency of software development using automatic generation of program code by generation AI. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram showing the configuration of a software development support system according to the embodiment. [Figure 2] An explanatory diagram of the productivity score for generating AI software development according to the embodiment. [Figure 3] A schematic diagram illustrating the selection of the code generation method according to the embodiment. [Figure 4] A flowchart illustrating the code generation method selection process according to the embodiment. [Figure 5] A flowchart illustrating the AI ​​code quality analysis process according to the embodiment. [Figure 6] A flowchart illustrating the AI ​​input data analysis process according to the embodiment. [Figure 7] A flowchart illustrating the code quality evaluation correction score calculation process according to the embodiment. [Figure 8]A flowchart illustrating the score calculation process for input data evaluation correction according to the embodiment. [Figure 9] A diagram illustrating a modified example 1 relating to the metric values ​​of the generated AI code quality evaluation score according to the embodiment. [Figure 10] A diagram illustrating a modified example 2 relating to the metric values ​​of the AI ​​input data productivity evaluation score generated according to the embodiment. [Figure 11] A diagram illustrating a modified example 3 relating to the code generation method selection process according to the embodiment. [Figure 12A] A diagram illustrating a modified example 4 relating to the code generation method selection process according to the embodiment. [Figure 12B] An explanatory diagram of the generated AI code quality evaluation score for Modification 4 related to the code generation method selection process according to the embodiment. [Figure 13] A diagram illustrating computer hardware. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings. The embodiments are illustrative examples for explaining the present invention, and have been omitted and simplified as appropriate for clarity of explanation. Furthermore, the present invention can also be implemented in various other forms, for example, by combining some or all of one embodiment or modification with some or all of other embodiments or modifications.

[0010] (Configuration of the software development support system S according to the embodiment) Figure 1 shows the configuration of a software development support system S according to an embodiment. The software development support system S is connected to a database 20 and a software development environment 31 via a network or communication line.

[0011] The database 20 has generative AI (Artificial Intelligence) code 21, release code 22, test code 23, PJ management data 24, prompt data 25, and knowledge data 26. The software development environment 31 is the user's computer in which an environment for executing software development is constructed, and has generative AI 31a. The database 20 and the software development environment 31 may be included in the software development support system S or may be connected via a network. The generative AI 31a may be constructed outside the software development environment 31 via a network.

[0012] The generative AI code 21 is program code generated by the generative AI 31a. The release code 22 is program code provided for function confirmation and quality confirmation. The test code 23 is test code for executing tests of each generative AI code 21 generated by the generative AI 31a.

[0013] The PJ management data 24 is development deliverables (code design information including requirement definition and specification definition, test code of the code, etc.). The prompt data 25 is prompt data input to the generative AI 31a to cause the generative AI 31a to generate program code, and is stored in the vector DB. The knowledge data 26 is the development deliverables of past projects.

[0014] The software development environment 31 receives a prompt from the user, acquires program code generated by the generative AI 31a based on the prompt, and stores it in the generative AI code 21.

[0015] The software development support system S has a generative AI code-related output data acquisition unit 11, a generative AI-related input data acquisition unit 12, a generative AI code quality analysis unit 13, and a generative AI input data analysis unit 14. The software development support system S also has a development method selection unit 15, a conventional method development productivity evaluation score calculation unit 16, a generative AI code quality evaluation score correction unit 17, and a generative AI input data productivity evaluation score correction unit 18.

[0016] The generated AI code-related output data acquisition unit 11 acquires the generated AI code 21, the release code 22, and the test code 23. The generated AI-related input data acquisition unit 12 acquires the PJ management data 24, the prompt data 25, and the knowledge data 26.

[0017] The generated AI code quality analysis unit 13 executes the generated AI code quality analysis process. The details of the generated AI code quality analysis process will be described later with reference to FIGS. 4 and 5.

[0018] The generated AI input data analysis unit 14 executes the generated AI input data analysis process. The details of the generated AI input data analysis process will be described later with reference to FIGS. 4 and 6.

[0019] The development method selection unit 15 calculates the generated AI code quality evaluation score S1, the generated AI input data productivity evaluation score S2, and the generated AI software development productivity evaluation score S3 for each process in each step of software development. The generated AI code quality evaluation score (code quality evaluation score) S1 is an index for evaluating the quality of the generated AI code generated by the generated AI 31a. The generated AI input data productivity evaluation score (input data evaluation score) S2 is an index for evaluating the productivity, quality, etc. of the input information such as the prompt input to the generated AI 31a for the generation of the generated AI code.

[0020] The development method selection unit 15 determines whether to generate the program code in the next step of the step for which the score is calculated using either the generated AI 31a or the conventional code development method (such as manual code generation). The development method selection unit 15 determines the method to be used for generating the program code in the next step based on the generated AI code quality evaluation score S1, the generated AI input data productivity evaluation score S2, and the generated AI software development productivity evaluation score S3.

[0021] The Conventional Method Development Productivity Evaluation Score Calculation Unit 16 calculates the Conventional Method Development Productivity Evaluation Score S4 related to the development productivity of the Conventional Code Development Method using well-known metrics analysis methods, etc. The Conventional Code Development Method is a manual code generation method. The metrics used to calculate the Conventional Method Development Productivity Evaluation S4 include the development effort (person-months) of the system / service, the number of task tickets managed, the number of bugs detected, LOC (Lines of Code), the number of system / service requests or requirements (number of topics within requests), etc.

[0022] The generated AI code quality evaluation score correction unit 17 calculates a code quality evaluation correction score C1 to correct the generated AI code quality evaluation score S1.

[0023] The generated AI input data productivity evaluation score correction unit 18 calculates an input data evaluation correction score C2 to correct the generated AI input data productivity evaluation score S2.

[0024] (Productivity evaluation score S3 for AI software development according to the embodiment) Figure 2 is an explanatory diagram of the generative AI software development productivity evaluation score S3 according to the embodiment.

[0025] The productivity of code implementation using a generated AI (hereinafter referred to as the "generated AI method") is estimated based on an indicator that includes a generated AI code quality evaluation score S1, which represents the quality of the generated AI code, and a generated AI input data productivity evaluation score S2), which represents the productivity of the input data to the generated AI. Based on the estimated productivity of code implementation using a generated AI, a decision is made as to whether to use a generated AI in the next project, development cycle, or the next development stage within the development cycle, and the development method is selected. The selection of a development method may involve switching from conventional manual code implementation (hereinafter referred to as the "conventional method") to a generated AI method, or from a generated AI method to a conventional method. It may also involve switching from a generated AI method using one generated AI to a generated AI method using another generated AI.

[0026] As shown in Figure 2, the X-axis is set to "1 / (Generated AI Input Data Productivity Evaluation Score S2)" and the Y-axis is set to "Generated AI Code Quality Evaluation Score S1". The slope α = Generated AI Code Quality Evaluation Score S1 / {1 / (Generated AI Input Data Productivity Evaluation Score S2)} is defined as the Generated AI Software Development Productivity Evaluation Score S3. A larger value of "1 / (Generated AI Input Data Productivity Evaluation Score S2)" indicates lower productivity of generated AI input data such as prompts. A larger value of "Generated AI Code Quality Evaluation Score S1" indicates higher quality of code implementation by the generated AI.

[0027] As shown in Figure 2, α0 is a user-defined slope threshold representing the threshold for determining when to terminate code generation by the generative AI. When the generative AI software development productivity evaluation score S3 (slope α) falls below the user-defined slope threshold α0 (α < α0), the generative AI method is terminated, and the conventional method is executed thereafter. The user-defined slope threshold α0 is a threshold that indicates whether or not there was a delay in the development plan in past projects included in the knowledge data 26, and α < α0 indicates that the generative AI software development productivity evaluation score S3 was low and that past projects were delayed.

[0028] Furthermore, with a threshold α1=1, if α1≦α, the productivity of the generative AI method is greater than that of the conventional method, so the generative AI method is recommended. If α<α1, the conventional method is more productive than the generative AI method, so the conventional method is recommended. However, this is not limited to α1=1.

[0029] If α < α1, and γ ≤ α, then in the next development stage, code generation may be performed using a different generative AI method instead of the generative AI method whose productivity was evaluated. However, γ is a predetermined threshold satisfying α0 < γ < α1. If γ ≤ α, the productivity of the other generative AI method is greater than or equal to that of the conventional method, so the selection of the other generative AI method is recommended. On the other hand, if α < γ, the productivity of the conventional method is greater than or equal to that of the other generative AI method, so the selection of the conventional method is recommended.

[0030] The thresholds α0, α1, α2, and γ mentioned above may be set to different values ​​for each development project, development cycle, module function, or phase gate (requirements definition, basic design, detailed design) as described later.

[0031] (Overview of the selection of code generation methods according to the embodiment) Figure 3 is a schematic diagram illustrating the selection of a code generation method according to the embodiment.

[0032] Consider a system or service development project with N development cycles (or development processes). In this case, at the end of each development cycle and at each software release, a generated AI software development productivity evaluation score S3 is calculated from the generated AI code quality evaluation score S1 and the generated AI input data productivity evaluation score S2. Then, based on the comparison result between the generated AI software development productivity evaluation score S3 and α1, it is determined whether to use the generated AI method or the conventional method in the next development cycle. If N=1, the method to be used in the next development project is determined.

[0033] In the example shown in Figure 3, for software releases 1 through 3, α = Generative AI software development productivity evaluation score S3 ≥ α1, so the generative AI method is selected in the next development cycles 2 through 4. However, for software release 4, α = Generative AI software development productivity evaluation score S3 < α1, so the conventional method is selected in the next development cycle 5.

[0034] (Code generation method selection process according to the embodiment) Figure 4 is a flowchart showing the code generation method selection process according to the embodiment.

[0035] First, in step S11, the software development support system S determines whether the final software release for the target development project has been completed. If the final software release has been completed (step S11YES), the software development support system S terminates the code generation method selection process (step S11YES), and if it has not been completed (step S11NO), it moves on to step S12.

[0036] In step S12, the software development support system S determines whether the development cycle (the first development cycle or the third development cycle) has been completed in the target development project. If the development cycle has been completed (step S12YES), the software development support system S proceeds to step S13; if the development cycle has not been completed (step S12NO), it repeats step S12 and waits for the development cycle to be completed.

[0037] In step S13, the software development support system S determines whether the selected method in the current development cycle is a generative AI method. If the selected method in the current development cycle is a generative AI method (step S13 YES), the software development support system S moves to step S14; otherwise, it moves to step S23.

[0038] In step S14, the generated AI code quality analysis unit 13 executes the generated AI code quality analysis process. Details of the generated AI code quality analysis process will be described later with reference to Figure 5. In step S14, metric values ​​for calculating the generated AI code quality evaluation score S1 are calculated.

[0039] Next, in step S15, the generated AI input data analysis unit 14 performs the generated AI input data analysis process. Details of the generated AI input data analysis process will be described later with reference to Figure 6. In step S15, metric values ​​for calculating the generated AI input data productivity evaluation score S2 are calculated.

[0040] Next, in step S16, the generated AI code quality evaluation score correction unit 17 executes a code quality evaluation correction score calculation process. Details of the code quality evaluation correction score calculation process will be described later with reference to Figure 7. In step S16, a correction score is calculated to correct the generated AI code quality evaluation score S1.

[0041] Next, in step S17, the generated AI input data productivity evaluation score correction unit 18 executes the input data evaluation correction score calculation process. Details of the input data evaluation correction score calculation process will be described later with reference to Figure 8. In step S17, a correction score is calculated to correct the generated AI input data productivity evaluation score S2.

[0042] Next, in step S18, the development method selection unit 15 calculates the generated AI code quality evaluation score S1 based on equation (1). However, in step S18, the right-hand side of equation (1) is the sum over the number of types n of metric values ​​xi calculated in step S14e (Figure 5) described later. In this embodiment, the number of types n is only one, x1 = LOC difference. wi is a weighting coefficient calculated in step S16f (Figure 7) described later, which absorbs differences in metric values ​​within the same project. ai is a weighting coefficient calculated in step S16f (Figure 7) described later, which absorbs differences in each metric value between different projects. ai is always "1" when it is not necessary to consider differences between different projects. S1={Σ(wi×ai×xi)} / n···(1)

[0043] Next, in step S19, the development method selection unit 15 calculates the generated AI input data productivity evaluation score S2 based on equation (1) described above. However, in step S19, “S1” on the left side of equation (1) is read as “S2”. The right side of equation (1) is the sum over the number of types n of the metric value xi calculated in steps S15c and S15d (Figure 6) described later. In this embodiment, the number of types n is two: x1 = number of tokens in the prompt data and x2 = number of tokens in the design specification data. wi is a weighting coefficient that absorbs the difference between the two metric values ​​x1 = number of tokens in the prompt data and x2 = design specification data. ai is a weighting coefficient that absorbs the difference between each metric value between different projects.

[0044] Equation (1) calculates the generated AI code quality evaluation score S1 and the generated AI input data productivity evaluation score S2 as weighted sums of multiple indicators related to multiple predetermined evaluation perspectives.

[0045] Next, in step S20, the development method selection unit 15 calculates the generated AI software development productivity evaluation score S3 from the generated AI code quality evaluation score S1 calculated in step S18 and the generated AI input data productivity evaluation score S2 calculated in step S19.

[0046] Next, in step S21, the development method selection unit 15 determines whether the generated AI software development productivity evaluation score S3 ≥ threshold α1. If the generated AI software development productivity evaluation score S3 ≥ threshold α1, the development method selection unit 15 moves to step S22; otherwise, it moves to step S25.

[0047] In step S22, the development method selection unit 15 selects the generation AI method as the development method for the program code (second code) in the next development cycle (second development cycle) (or next development project). When step S22 is completed, the software development support system S returns to step S11.

[0048] On the other hand, in step S23, the conventional method development productivity evaluation score calculation unit 16 calculates the conventional method development productivity evaluation score S4 based on the release code 22 (first code or third code) released in the development cycle that was determined to be completed in step S12. The release code 22 here is generated by a manual code generation method (conventional method).

[0049] Next, in step S24, the development method selection unit 15 determines whether the conventional method development productivity evaluation score S4 > threshold α2. Threshold α2 is a predetermined threshold for evaluating the conventional method development productivity evaluation score S4, and is calculated in the same way as threshold α1. If S4 > α2 (step S24 YES), the development method selection unit 15 moves to step S25, and if S4 ≤ α2 (step S24 NO), it moves to step S22.

[0050] In step S25, the development method selection unit 15 selects a conventional method as the development method for the program code (second code or fourth code) in the next development cycle (second development cycle or fourth development cycle) (or the next development project). When step S25 is completed, the software development support system S returns to step S11.

[0051] (Variations of the code generation method selection process) In step S21 described above, the selection of the development method is determined based on the generative AI software development productivity evaluation score S3. However, this is not the only option; a binary classification model that has learned the weighting coefficients wi and ai in equation (1) may also be used to determine whether to select the generative AI method or the conventional method. In this binary classification model, the explanatory variable is the metric value xi, and the dependent variable is a two-label system of success / failure.

[0052] (Generated AI code quality analysis process according to the embodiment) Figure 5 is a flowchart showing the generated AI code quality analysis process according to the embodiment. In the generated AI code quality analysis process, metric values ​​are calculated for calculating the generated AI code quality evaluation score S1.

[0053] First, in step S14a, the generated AI code quality analysis unit 13 obtains the release code 22 released in the development cycle that was determined to be completed in step S12. The release code 22 may have been modified as needed in the code generated by the generated AI 31a or manual code generation method during each development cycle.

[0054] Next, in step S14b, the generated AI code quality analysis unit 13 obtains the generated AI code 21 (first code) which was the basis for the release code 22 released in the development cycle that was determined to be completed in step S12.

[0055] Next, in step S14c, the generated AI code quality analysis unit 13 calculates the LOC of the release code 22 obtained in step S14a using an existing static analysis tool. Next, in step S14d, the generated AI code quality analysis unit 13 calculates the LOC of the generated AI code 21 obtained in step S14b using an existing static analysis tool.

[0056] Next, in step S14e, the generated AI code quality analysis unit 13 calculates the LOC difference between the LOC of the release code 22 calculated in step S14c and the LOC of the generated AI code 21 calculated in step S14d. The difference here is the LOC of the release code 22 minus the LOC of the generated AI code 21.

[0057] The difference calculated in step S14e is the metric value used to calculate the generated AI code quality evaluation score S1. In this embodiment, the number of types n of metric values ​​used when calculating the generated AI code quality evaluation score S1 is only one, the "LOC difference" calculated in step S14e.

[0058] (Generated AI input data analysis process according to the embodiment) Figure 6 is a flowchart showing the generation AI input data analysis process according to the embodiment. In the generation AI input data analysis process, metric values ​​are calculated for calculating the generation AI input data productivity evaluation score S2.

[0059] First, in step S15a, the generation AI input data analysis unit 14 obtains prompt data 25 (an example of the first input data) corresponding to the generation AI code 21 that formed the basis of the release code 22 released in the development cycle determined to be completed in step S12. Next, in step S15b, the generation AI input data analysis unit 14 obtains design specification data (an example of the first input data) used to create the vector database (DB) for RAG (Retrieval-Augmented Generation). This design specification data is included in the PJ management data 24 and is vectorized design specification data used to create a vector DB for RAG, which searches for additional information to be added to the input information to be fed into the generation AI 31a for code generation.

[0060] Next, in step S15c, the generated AI input data analysis unit 14 calculates the number of tokens in the prompt data 25 acquired in step S15a. Next, in step S15d, the generated AI input data analysis unit 14 calculates the number of tokens in the design specification data acquired in step S15b.

[0061] The number of tokens in the prompt data calculated in step S15c and the number of tokens in the design specifications calculated in step S15d are the metric values ​​for calculating the generated AI input data productivity evaluation score S2. In this embodiment, the number of types n of metric values ​​used when calculating the generated AI input data productivity evaluation score S2 is the two numbers of tokens calculated in steps S15c and S15d.

[0062] (Code quality evaluation correction score calculation process according to this embodiment) Figure 7 is a flowchart showing the code quality evaluation correction score calculation process according to the embodiment. In the code quality evaluation correction score calculation process, a correction score is calculated to correct the generated AI code quality evaluation score S1.

[0063] First, in step S16a, the generated AI code quality evaluation score correction unit 17 obtains the generated AI code 21 that formed the basis of the release code 22 released in the development cycle determined to be completed in step S12. Next, in step S16b, the generated AI code quality evaluation score correction unit 17 calculates metric values ​​for the generated AI code 21 using a static analysis tool for existing source code. These metric values ​​include LOC and cyclomatic complexity.

[0064] Next, in step S16c, the generated AI code quality evaluation score correction unit 17 calculates the metric values ​​(such as LOC and cyclomatic complexity) of the generated AI code 21 created in the past, and calculates statistical values ​​such as the mean and standard deviation.

[0065] Next, in step S16d, the generated AI code quality evaluation score correction unit 17 calculates the difference between the metric values ​​(statistical values ​​such as mean and standard deviation) of the generated AI code 21 created in the past and the metric values ​​of the generated AI code 21 calculated in step S16b. The purpose of step S16d is to indicate that the higher the proportion of the generated AI code's metric values ​​that fall within the range of mean + (standard deviation + α) of the metric values ​​of past release codes, the less variability and more stable the generated code is considered to be (high stability of the generated AI code). Therefore, in step S16f described later, a correction score is calculated such that the higher the stability of the generated AI code, the higher the generated AI code quality evaluation score S1.

[0066] Next, in step S16e, the generated AI code quality evaluation score correction unit 17 calculates the number of requests or requirements (number of topics within requests) included in the requirements definition that formed the basis of the release code 22 released in the development cycle determined to be completed in step S12. The purpose of step S16e is to calculate a correction score for the generated AI code quality evaluation score S1 such that the generated AI code quality evaluation score S1 is higher in step S16f, which will be described later, assuming that the more requests or requirements there are, the greater the amount and difficulty of development.

[0067] Next, in step S16f, the generated AI code quality evaluation score correction unit 17 calculates a code quality evaluation correction score based on the difference calculated in step S16d and the number of requests or requirements calculated in step S16f.

[0068] In this embodiment, since the number of types of metric values ​​n in the generated AI code quality evaluation score S1 is n=1, there is no need to consider the weighting coefficient wi that absorbs differences in metric values ​​within the same project (wi can always be set to 1). That is, the code quality evaluation correction score calculated in step S16f is only the weighting coefficient ai(i=1) in equation (1) above that absorbs differences in metric values ​​within different projects. Therefore, in step S16f, the generated AI code quality evaluation score correction unit 17 calculates the weighting coefficient ai(i=1) in equation (1) above based on the difference calculated in step S16d and the number of requests or requirements calculated in step S16f.

[0069] (Score calculation process for input data evaluation correction according to the embodiment) Figure 8 is a flowchart showing the input data evaluation correction score calculation process according to the embodiment. In the input data evaluation correction score calculation process, a correction score is calculated to correct the generated AI input data productivity evaluation score S2.

[0070] First, in step S17a, the generated AI input data productivity evaluation score correction unit 18 acquires log data. The acquired log data is the log data when the prompt data 25 corresponding to the generated AI code 21 that formed the basis of the release code 22 released in the development cycle that was determined to be completed in step S12 is executed.

[0071] Next, in step S17b, the generated AI input data productivity evaluation score correction unit 18 identifies the prompt development process and detailed design based on the log data acquired in step S17a. The development processes identified here include, for example, requirements definition, basic design (architecture specifications, etc.), and detailed design (API specifications, etc.). The purpose of step S17b is to calculate a correction score for the generated AI input data productivity evaluation score S2 such that the further upstream the development process is, the larger the generated AI input data productivity evaluation score S2 will be in step S17c, described later.

[0072] Next, in step S17c, the generated AI input data productivity evaluation score correction unit 18 calculates a correction score for the generated AI input data productivity evaluation score S2 based on the development process identified in step S17b.

[0073] In this embodiment, the number of types of metric values ​​n in the generated AI input data productivity evaluation score S2 is n=2. Therefore, based on the development process identified in step S17b, a weighting coefficient wi that absorbs differences in metric values ​​within the same project and a weighting coefficient ai that absorbs differences in metric values ​​between different projects are calculated for each metric value. That is, in step S17c, the generated AI input data productivity evaluation score correction unit 18 calculates the weighting coefficients wi and ai (i=1,2) in equation (1) above based on the development process identified in step S17b.

[0074] (Modified example 1 relating to the metric values ​​of the generated AI code quality evaluation score according to the embodiment) Figure 9 is an explanatory diagram of a modified example 1 relating to the metric values ​​of the generated AI code quality evaluation score S1 according to the embodiment. In the above embodiment, the metric value used in the generated AI code quality evaluation score S1 was the LOC difference between the LOC of the release code 22 and the LOC of the generated AI code 21. However, the generated AI code quality evaluation score S1 is not limited to this.

[0075] Specifically, it may be any one or more combinations of the LOC difference and the following metric values ​​shown in the numbers # in Figure 9. (1) The difference in cyclomatic complexity between release code 22 and generated AI code 21. Similar to the LOC difference, a smaller difference results in a higher generated AI code quality evaluation score S1. Similar to the LOC difference, this is an indicator based on the evaluation criteria of code quality and code reusability of generated AI code 21. (2) Differences in the code I / F (function and method arguments) and / or metric values ​​(cohesion, coupling) between release code 22 and generated AI code 21. The smaller the difference, the higher the generated AI code quality evaluation score S1. Represents the stability of the module definition in generated AI code 21. This is an indicator based on the evaluation criteria for the code reusability of generated AI code 21. (3) The test success rate calculated by running several pre-prepared test codes on the generated AI code 21. The higher the success rate, the larger the generated AI code quality evaluation score S1. This is an index based on the evaluation criteria for code requirement satisfaction (code quality) of the generated AI code 21. (4) The test success rate calculated by running the test code created by Generator AI 31a on Generator AI Code 21. A higher success rate results in a larger Generator AI Code Quality Evaluation S1. This serves as an indicator based on the evaluation criteria for the degree of related deliverable generation (productivity) of Generator AI Code 21.

[0076] (Modified example 2 of the metric value of the AI ​​input data productivity evaluation score S2 according to the embodiment) Figure 10 is an explanatory diagram of a modified example 2 relating to the metric values ​​of the generated AI input data productivity evaluation score S2 according to the embodiment. In the above embodiment, the metric values ​​used in the generated AI input data productivity evaluation score S2 were the number of tokens in the prompt data and design specification data. However, the generated AI input data productivity evaluation score S2 is not limited to this.

[0077] Specifically, this may be a combination of the number of tokens in the prompt data and design specification data, and one or more of the following metric values ​​shown in the numbers # in Figure 10. (1) The difference between the number of prompt queries at the time of software release, based on the prompt execution log, and the number of prompt queries between the two software releases. The smaller the number of queries and the smaller the difference in the number of queries (i.e., the less context), the higher the generated AI input data productivity evaluation score S2. This serves as an indicator based on the evaluation criteria of prompt complexity and reproducibility. (2) The number of prompt trials (increment) required to output the generated AI code 21 at the time of software release. A smaller number of trials (increment) results in a higher generated AI input data productivity evaluation score S2. This serves as an indicator based on the evaluation criteria for the amount of development of the prompt data 25 (vector DB).

[0078] (Modified example 3 relating to the code generation method selection process according to the embodiment) Figure 11 is an explanatory diagram of a modified example 3 relating to the code generation method selection process according to the embodiment. In the above-described embodiment, the code generation method selection process determined whether to adopt the generation AI method or the conventional method, regardless of the module.

[0079] In contrast, in modification 3, the decision to adopt either the generative AI method or the conventional method is made and selected independently for each module. In the example shown in Figure 11, the generative AI method is used for both the development of module (communication) and module (application 1) in development cycles 1 to 4. However, in development cycle 5, the generative AI method continues to be used for module (communication), while module (application 1) switches to and adopts the conventional method.

[0080] (Modification 4 of the code generation method selection process according to the embodiment) Figure 12A is an explanatory diagram of a modified example 4 relating to the code generation method selection process according to the embodiment. In the embodiment described above, a code release is performed once at the end of each development cycle, and based on each generated AI code after the release, it is decided whether to adopt the generation AI method or the conventional method in the next development cycle.

[0081] In contrast, in Modification 4, the method to be adopted in the next development stage is determined based on the generated AI code quality evaluation score in the code implementation at the completion point (phase gate unit) of multiple development stages included in each development cycle of a single system / service development project.

[0082] As shown in Figure 12A, development processes within each development cycle include, for example, requirements definition, basic design, detailed design, code implementation, and testing. Examples of AI code implementation for each development process include: in requirements definition, the requirements definition document is implemented using AI code; in basic design, the architecture design document and system-level design pattern specifications are implemented using AI code; and in detailed design, API specifications and software-level design pattern specifications are implemented using AI code.

[0083] (Quality evaluation score of the generated AI code in modified example 4) Figure 12B is an explanatory diagram of the generated AI code quality evaluation score S5 in Modified Example 4 of the code generation method selection process according to the embodiment. The generated AI code quality evaluation score S5 in Modified Example 4 is used in place of the generated AI code quality evaluation score S1. The generated AI code quality evaluation score S5 is determined by the similarity of metrics when compared with the final release code in the same development cycle within the same or similar past project.

[0084] As shown in Figure 12B, the metrics for the final release code are set to 100. In this case, when implementation code is generated using the generative AI method in the requirements definition phase, the generated AI code quality evaluation score is assumed to be 40. Furthermore, when implementation code is generated using the generative AI method in the basic design phase, the generated AI code quality evaluation score is assumed to be 80. And when implementation code is generated using the generative AI method in the detailed design phase, the generated AI code quality evaluation score is assumed to be 82.

[0085] From the above, the detailed design shows an improvement of 2 in the generated AI code quality evaluation score compared to the basic design, and there is no significant improvement beyond a predetermined value. Therefore, it is judged that the productivity of code generation using the generated AI method is low in the detailed design. Accordingly, in Modification 2, generated AI is used up to the basic design, and development from the detailed design onward is done using the conventional method or a different generated AI.

[0086] (Modified example 5 of the method for calculating threshold α1) In the embodiment described above, the threshold α1 used to determine the generated AI software development productivity evaluation score S3 in step S21 (Figure 5) was set to a fixed value (e.g., 1).

[0087] However, this is not the only method; you may also calculate it using either of the following two threshold setting methods. (1) A method for setting thresholds based on the deviation between the planned development effort (man-hours) and the actual development effort required. (2) A threshold setting method using the median or mean of the generated AI software development productivity evaluation score S3.

[0088] First, (1) we will explain the method for setting thresholds based on the deviation between the software release plan (effort) and the actual development effort required. (i) Measure the development effort for each software release. (ii) Calculate the generation AI software development productivity evaluation score S3 for each release. (i) and (ii) shall be implemented in one or more development projects. Since the score trends may change depending on the type of product or service, it is desirable not to mix multiple types of products or services to avoid bias in the types of products or services. (iii) Extract only the data for software releases that satisfy equation (2), and set the generated AI software development productivity evaluation score S3 for the release with the maximum development effort up to the software release from the extracted data as the threshold α1. Development effort until software release ≤ Planned development effort × (1.0 + AD) ...(2)

[0089] In equation (2), AD is pre-set as an acceptable delay, and a value such as +0.1 is set according to how much delay is acceptable. Negative values ​​can also be set for AD. Furthermore, AD can be set based on the distribution of development effort data for each software release. For example, if there are generally many releases that fall below the planned development effort, the value of AD may be set to a negative value or made smaller. On the other hand, if there are generally many releases that exceed the planned development effort, the value of AD may be made larger.

[0090] Next, we will explain (2) the threshold setting method using the median or mean of the generated AI software development productivity evaluation score S3. First, (i) and (ii) are performed, similar to (1) the threshold setting method based on the deviation between the software release plan (man-hours) and the actual development man-hours required. Next, (iii)' below is performed. (iii)' Determine the threshold α1 based on equation (3-1) or equation (3-2). α = m1 × (1 + CF) ... (3 - 1) α = m² × (1 + CF)···(3-3) m1 is the mean of the Generative AI Software Development Productivity Evaluation S3. m2 is the median of the Generative AI Software Development Productivity Evaluation S3. CF is the correction factor. For frequency distribution data, for example, if there are generally many releases that fall below the planned development effort, the value of CF should be increased. On the other hand, if there are generally many releases that exceed the planned development effort, it may be considered to make the value of CF negative or decrease it.

[0091] (Effects of the embodiment) In the embodiment described above, based on the evaluation score of the code generated in the first development cycle, it is determined whether to use a generation AI or a manual code generation method when generating the second code in the second development cycle following the first development cycle. Therefore, according to the embodiment, the method with higher productivity for code implementation can be selected from the generation AI, other generation AIs, and the manual code generation method, and the code implementation, development cycle, and development project can be advanced efficiently. In other words, the development efficiency of software development using automatic generation of program code by generation AI can be improved.

[0092] Furthermore, in the above-described embodiment, the input data evaluation score and the code quality evaluation score are each calculated as a weighted sum of multiple indicators related to multiple predetermined evaluation perspectives. Therefore, various metrics for evaluating the generated AI code and input data can be reflected in the evaluation score, making it possible to further optimize the selection of the code generation method.

[0093] Furthermore, in the above-described embodiment, by appropriately correcting the evaluation score of the generated AI code, the stability of the generated AI code (whether the same code is generated each time), the amount of development required, and the level of detail and complexity of the input data for code generation, such as prompts and design specification data, can be reflected in the evaluation score. In other words, the selection of the code generation method can be made more appropriate.

[0094] Furthermore, in the above-described embodiment, an input data evaluation score, a code quality evaluation score, and a generated AI software development productivity evaluation score are calculated for each function of the module or for each of the multiple development processes included in each development cycle. Then, a code generation method is selected based on the generated AI software development productivity evaluation score. This allows for a more appropriate selection of the code generation method, reflecting the functional characteristics of the module and the characteristics of the development process.

[0095] (Hardware configuration of Computer 1000) Figure 14 shows an example of the hardware configuration of computer 1000. Computer 1000 implements each part of the software development support system S by executing a predetermined program.

[0096] The computer 1000 comprises a processor 1001 including a CPU, a main memory 1002, an auxiliary memory 1003, a network interface 1004, an input device 1005, and an output device 1006, all interconnected via an internal communication line 1007 such as a bus.

[0097] The processor 1001 controls the operation of the entire computer 1000. The main memory 1002 is composed of, for example, volatile semiconductor memory and is used as the work memory of the processor 1001. The auxiliary storage device 1003 is composed of a large-capacity non-volatile storage device such as a hard disk drive, SSD (Solid State Drive), or flash memory and is used to retain various programs and data for long periods of time.

[0098] The executable program 1003a stored in the auxiliary storage device 1003 is loaded into the main memory device 1002 when the computer 1000 starts up or when needed, and is executed by the processor 1001.

[0099] The executable program 1003a may be recorded on a non-temporary recording medium, read from the non-temporary recording medium by a media reader, and loaded into the main memory 1002. Alternatively, the executable program 1003a may be obtained from an external computer via a network and loaded into the main memory 1002.

[0100] The auxiliary storage device 1003 stores various executable programs 1003a.

[0101] The network interface 1004 is an interface device for connecting computer 1000 to various networks within the system or for communicating with other computers. The network interface 1004 consists of, for example, a NIC (Network Interface Card) such as a wired LAN (Local Area Network) or a wireless LAN.

[0102] The input device 1005 consists of a keyboard, a pointing device such as a mouse, and is used by the user to input various instructions and information into the computer 1000. The output device 1006 consists of a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display, or an audio output device such as a speaker, and is used to present necessary information to the user when needed.

[0103] Although several embodiments have been described above, these are merely illustrative examples for explaining the present invention and are not intended to limit the scope of the present invention to these embodiments only. The present invention can also be implemented in various other forms, such as forms in which some of the components of the above embodiments are omitted, forms in which at least some of the components are replaced, forms in which components are added, or forms that combine some or all of the embodiments. [Explanation of symbols]

[0104] S: Software development support system, 1001: Processor, 1002: Main memory.

Claims

1. A software development support system that assists in the development of software, The aforementioned software development support system has a processor and memory, The aforementioned processor, The aforementioned software development project is managed by dividing it into multiple development cycles. In the first development cycle of the aforementioned multiple development cycles, a score is calculated to evaluate the first code of the software generated using either artificial intelligence (AI) or a code generation method other than the AI, based on predetermined evaluation criteria. Based on the score, it is determined whether to use the generation AI, another generation AI, or a code generation method other than the generation AI when generating the second code in the second development cycle following the first development cycle among the multiple development cycles. A software development support system characterized by the following features.

2. A software development support system according to claim 1, The aforementioned processor, Based on the results of analyzing the first input data input to the generating AI for generating the first code, an input data evaluation score is calculated for evaluating the first input data based on predetermined evaluation criteria. Based on the results of analyzing the first code generated by the generation AI based on the first input data, a code quality evaluation score is calculated to evaluate the quality of the first code based on predetermined evaluation criteria. Based on the input data evaluation score and the code quality evaluation score, a generation AI software development productivity evaluation score is calculated as the score for evaluating the productivity of software development by the generation AI. Based on the aforementioned AI software development productivity evaluation score, it is determined whether to use the aforementioned AI, another AI, or the manual code generation method when generating the second code. A software development support system characterized by the following features.

3. A software development support system according to claim 1, The aforementioned processor, In the third development cycle of the aforementioned multiple development cycles, based on the results of analyzing the third code generated using the manual code generation method, a development productivity evaluation score is calculated as the score for evaluating the third code based on predetermined evaluation criteria. Based on the aforementioned development productivity evaluation score, it is determined whether to use the aforementioned generation AI, another generation AI, or the manual code generation method when generating the fourth code in the fourth development cycle following the third development cycle among the multiple development cycles. A software development support system characterized by the following features.

4. A software development support system according to claim 2, The aforementioned processor, The input data evaluation score and the code quality evaluation score are each calculated as a weighted sum of multiple indicators related to multiple predetermined evaluation perspectives. A software development support system characterized by the following features.

5. A software development support system according to claim 2, The aforementioned processor, An input data evaluation correction score for correcting the aforementioned input data evaluation score is calculated based on the development process identified from the first input data. A code quality evaluation correction score for correcting the aforementioned code quality evaluation score is calculated based on a predetermined metric value of the first code. The input data evaluation score is corrected using the input data evaluation correction score. The code quality evaluation score is corrected using the code quality evaluation correction score. The generated AI software development productivity evaluation score is calculated based on the corrected input data evaluation score and the code quality evaluation score. A software development support system characterized by the following features.

6. A software development support system according to claim 2, The aforementioned processor, The development project of the aforementioned software is managed by dividing it into the aforementioned multiple development cycles for each module function, For each function of the module, the input data evaluation score, the code quality evaluation score, and the generation AI software development productivity evaluation score are calculated, and based on the generation AI software development productivity evaluation score, it is determined whether to use the generation AI, another generation AI, or the manual code generation method when generating the second code. A software development support system characterized by the following features.

7. A software development support system according to claim 2, The aforementioned processor, Each of the aforementioned development cycles is further divided into multiple development processes for management, For each development step, the input data evaluation score, the code quality evaluation score, and the generation AI software development productivity evaluation score are calculated, and based on the generation AI software development productivity evaluation score, it is determined whether to use the generation AI, another generation AI, or the manual code generation method when generating code in the next development step following each development step. A software development support system characterized by the following features.

8. A software development support method performed by a software development support system that assists in software development, The aforementioned software development support system has a processor and memory, The aforementioned processor, The aforementioned software development project is managed by dividing it into multiple development cycles. In the first development cycle of the aforementioned multiple development cycles, a score is calculated to evaluate the first code of the software generated using either artificial intelligence (AI) or a code generation method other than the AI, based on predetermined evaluation criteria. Based on the score, it is determined whether to use the generation AI, another generation AI, or a code generation method other than the generation AI when generating the second code in the second development cycle following the first development cycle among the multiple development cycles. A software development support method characterized by having each of the following processes.

Images