Software performance verification system and software performance verification method

Abstract

A software performance verification system that effectively performs verification of performance of software extracts a part of a code of a program that constitutes the software, generates a feature amount vector based on the part of the code, generates information based on an output obtained by inputting the part of the code of a verification target into a performance verification model as a verification result of the part of the code, the performance verification model is a machine learning model that is learned by learning data including a feature amount vector of a part of a code for learning and information indicating performance of software implemented based on the part of the code, the software performance verification system, for example, generates a vector including an index value obtained from the part of the code as an element as the feature amount vector, and the performance verification model, for example, outputs a probability that there is a problem in performance of processing implemented based on the part of the code.

Description

Technical Field

[0001] This invention relates to a software performance verification system and a software performance verification method. Background Technology

[0002] This application claims priority based on Japanese Patent Application No. 2021-038744, filed on March 10, 2021, the entire contents of which are incorporated herein by reference.

[0003] Patent document 1 describes a software performance prediction system for predicting software performance. The system receives input from multiple terminals, including initial or modified source code accompanying software development. It registers the received source code as source code information in a database. By comparing the software performance resulting from the modified source code with the performance resulting from the old source code included in the registered source code information, it calculates the performance reduction ratio between the old and new source codes, determines whether the performance reduction ratio exceeds a predetermined value, and if so, notifies an external party of the determination result.

[0004] Patent document 2 describes a system analysis device for predicting performance bottlenecks in system development. The system analysis device detects past system design information similar to the development system design information as similar system detection results, retrieves past system functional components similar to the development system functional components contained in the development system design information as similar function retrieval results, and uses development system requirements, past system requirements, development system functional components, past system functional components, and past system measurement results of past systems to obtain system component performance information that establishes a correlation between the development system functional components and past system functional components and performance information. Based on the similar system detection results and system component performance information, it detects system functional components that act as performance bottlenecks.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2012-234448

[0008] Patent Document 2: Japanese Patent Application Publication No. 2020-149681 Summary of the Invention

[0009] The problem that the invention aims to solve

[0010] During software development, it is necessary to meet requirements regarding the functions that should be installed in the software (hereinafter referred to as "functional requirements") and performance requirements that the software should meet (hereinafter referred to as "performance requirements"). While developers can confirm whether the functional requirements are met at various stages of software development based on the source code, it is difficult to judge the performance requirements at the source code level. Furthermore, software performance verification using so-called performance analysis tools (performance analyzers, profilers) cannot be performed in the later stages of development if the source code and data required for code generation (compile, build, etc.) are not readily available. Therefore, if performance requirements are not met during this verification, the amount of rework is significant, greatly impacting production efficiency. In addition, software performance verification requires setting up test scenarios that consider various execution conditions and preparing a large amount of test data for each test scenario, thus involving a significant preparation workload.

[0011] Patent Document 1 compares the performance of software resulting from changes to the source code with the performance of software resulting from the old source code contained in the registered source code information. However, this comparison requires compiling and building the changed source code, and performance cannot be verified until the necessary source code and data are readily available. Furthermore, the technology described in this document does not envision a programming language that does not require compilation, such as an interpreted language. Additionally, while the document calculates the performance reduction rate between the old and new source codes, calculating this reduction rate requires preparing test data separately for both before and after the change.

[0012] Patent Document 2 achieves system design efficiency by comparing the design information of the developed system with past system design information, i.e., by comparing at the level of design information. However, it does not envision verifying performance issues at the specific code level. The technology described in this document is not intended to reduce the burden of performance verification during software development.

[0013] The present invention was made in view of the following background, and its purpose is to provide a software performance verification system and a software performance verification method that can efficiently verify the performance of software.

[0014] Methods used to solve problems

[0015] One technical solution of the present invention for achieving the above-mentioned objective is a software performance verification system configured using an information processing device; comprising: a storage unit for storing code of a program constituting software; a partial code extraction unit for extracting partial code that is part of the aforementioned code; a feature vector generation unit for generating feature vectors based on the aforementioned partial code; and a performance verification processing unit for generating information based on the output obtained by inputting the aforementioned partial code of the verification object into a performance verification model as the verification result of the aforementioned partial code, wherein the performance verification model is a machine learning model learned using learning data including the aforementioned feature vectors of the aforementioned partial code used for learning and performance information representing the performance of the software implemented based on the aforementioned partial code.

[0016] In addition, the problems and solutions disclosed in this application will become clear from the detailed description and accompanying drawings.

[0017] Invention Effects

[0018] According to the present invention, software performance can be verified efficiently. Attached Figure Description

[0019] Figure 1 This is a system flowchart that represents the general structure of the performance verification system.

[0020] Figure 2A This is a diagram illustrating an example of partial code extraction and processing.

[0021] Figure 2B This is a flowchart illustrating an example of partial code extraction and processing.

[0022] Figure 3A This is a diagram illustrating other examples of partial code extraction and processing.

[0023] Figure 3B This is a flowchart illustrating other examples of partial code extraction and processing.

[0024] Figure 4A This is a diagram illustrating an example of feature vector generation.

[0025] Figure 4B This is a flowchart illustrating an example of feature vector generation.

[0026] Figure 5A This is a diagram illustrating other examples of feature vector generation processing.

[0027] Figure 5B This is a flowchart illustrating the process of generating a word dictionary.

[0028] Figure 5C This is a flowchart illustrating the vectorization process of feature quantities.

[0029] Figure 6A This is a diagram illustrating an example of a learning method for performance validation models.

[0030] Figure 6B This is a flowchart illustrating an example of a learning method for a performance validation model.

[0031] Figure 7A This is a diagram illustrating other examples of learning methods for performance validation models.

[0032] Figure 7B This is a flowchart illustrating other examples of learning methods for performance validation models.

[0033] Figure 8A This is a diagram illustrating an application example of a performance verification system.

[0034] Figure 8B This is a flowchart illustrating the processing in an application example of a performance verification system.

[0035] Figure 9A This is a diagram illustrating other application examples of the performance verification system.

[0036] Figure 9B This is a flowchart illustrating the processing in other application examples of the performance verification system.

[0037] Figure 10 This is an example of an information processing device used in structures such as performance verification systems. Detailed Implementation

[0038] The following is a reference to the appendix. Figure 1 The embodiments will be described below. Furthermore, the following description and accompanying drawings are illustrative of the invention, and omissions or simplifications have been appropriately made for clarity. The invention can also be implemented in various other forms. Unless otherwise specified, each component can be a single element or multiple elements. In addition, in the following description, the same reference numerals are sometimes assigned to the same or similar structures, and repeated descriptions are omitted. Furthermore, in the following description, the character "S" preceding the reference numerals indicates a processing step.

[0039] Figure 1This is a system flowchart illustrating the general structure of a software performance verification system (hereinafter referred to as "performance verification system 1"), which is an information processing system described as one embodiment. Performance verification system 1 verifies the performance of software by inputting program code (hereinafter referred to as "code"), such as source code described in a specified programming language generated during software development or maintenance, into a machine learning model (hereinafter referred to as "performance verification model 216"). Indicators representing the aforementioned performance include, for example, throughput, response time, and resource usage.

[0040] For performance verification model 216, for example, learning data (training data) is built by using information representing the performance of existing code (hereinafter referred to as "performance information") as labels (positive solution data) for learning (training).

[0041] The language used to describe the code of the verification object is not necessarily limited, as long as it is a language that a programmer with standard skills can interpret. Furthermore, the method of generating the executable code based on the above code is not necessarily limited. For example, it can be a compiled language that requires compilation and building (also called linking, making, etc.) during the generation of executable code, or it can be an interpreted language that executes sequentially.

[0042] Furthermore, the type of code mentioned above is not necessarily limited. For example, it can be any of the following: web page description language, scripting language, application description language that runs on the server side, system description language of operating system, description language of embedded software, or batch processing description language.

[0043] Furthermore, the code for the verification object can be either newly created during development or updated or added during maintenance. Moreover, the aforementioned code can be described by users such as software developers in a standalone development environment, or generated in a collaborative development environment using a repository environment connected via a communication network, where multiple information processing devices are linked together.

[0044] In addition, the following example illustrates the case where the code being verified is the source code of a compiled language.

[0045] The type of performance validation model 216 is not necessarily limited; for example, it could be envisioned as a model performing binary classification or multi-class classification. Machine learning mechanisms used to implement performance validation model 216 include, for example, DNN (Deep Neural Network), Support Vector Machine (SVM), decision tree, and k-nearest neighbor (k-means) methods. Performance validation model 216 can be represented, for example, by a matrix containing information (parameters) including features and the weights of each feature.

[0046] The existing code used in the generation of learning data includes, for example, code used in other existing information processing systems (hereinafter referred to as "existing similar systems") that are similar in function and structure to the information processing system implemented by the verification object (hereinafter referred to as "verification object system"). Furthermore, the similarity in function and structure between the verification object system and the existing similar systems is determined based on factors such as the application domain of each system, the implementation method of each system, the execution environment of the software constituting each system, the types of users utilizing each system, and the universality of the programming languages ​​used in development.

[0047] Labels can be established to correspond to existing code used in the generation of learning data, for example, using information obtained from execution logs of executable code based on existing code in a production or testing environment, or information obtained from the results of execution tests or simulations performed on the aforementioned executable code. Additionally, performance information set by individuals proficient in existing similar systems can also be used as labels.

[0048] like Figure 1 As shown, the performance verification system 1 includes the functions of a performance verification unit 100 and a model learning unit 200. The performance verification unit 100 performs performance verification on the source code of the verification object using a performance verification model 216. Furthermore, the performance verification unit 100 generates learning data based on existing code and learns the performance verification model 216 using the generated learning data.

[0049] As shown in the figure, the performance verification unit 100 includes a partial code extraction unit 120, a feature vector generation unit 130, and a performance verification processing unit 140.

[0050] Specifically, the partial code extraction unit 120 extracts descriptions (hereinafter referred to as "partial code (verification objects)") corresponding to each method in the verification object method list 112 from the source code group (verification object) 111, and outputs the extracted partial code (verification objects) as partial code group (verification object) 113. The verification object method list 112 is a list of method names of the verification objects, and its content is set by the user, for example.

[0051] The feature vector generation unit 130 transforms each part of the code (verification object) of the partial code group (verification object) 113 into a feature vector, and outputs the transformed feature vector as the feature vector group (verification object) 114.

[0052] The performance verification processing unit 140 inputs each feature vector of the feature vector group (verification object) 114 into the performance verification model 216, and generates information based on the performance information output by the performance verification model 216 regarding the input as a verification result 115. The performance verification processing unit 140 provides the verification result 115 to the user, for example, via a user interface.

[0053] As shown in the figure, the model learning unit 200 includes a partial code extraction unit 220, a feature vector generation unit 230, a learning data generation unit 240, and a learning processing unit 250.

[0054] The partial code extraction unit 220 extracts code (hereinafter referred to as "partial code (learning)") from the source code used in the learning of the performance verification model 216 (hereinafter referred to as "source code group (learning) 211"), and outputs the extracted partial code (learning) as partial code group (learning) 213.

[0055] The feature vector generation unit 230 transforms a portion of the code (for learning purposes) 213 into a feature vector, and outputs the transformed feature vector as the feature vector group (for learning purposes) 214.

[0056] The learning data generation unit 240 generates one or more learning data sets (a set of feature vectors and labels) by assigning corresponding labels of performance labels 212 to each feature vector of the feature vector set (verification object) 114, and outputs each generated learning data set as a learning data set 215.

[0057] The learning processing unit 250 inputs the feature vectors of each learning data in the learning data group 215 into the performance verification model 216. It uses the output of the performance verification model 216 with respect to the input feature vectors and establishes the corresponding label difference with the input feature vectors to adjust the parameters of the performance verification model 216, thereby enabling the performance verification model 216 to learn.

[0058] Next, regarding Figure 1 Each function shown is described in detail.

[0059] Figure 2A yes Figure 1 The illustrated partial code extraction unit 120 or partial code extraction unit 220 (hereinafter collectively referred to as "partial code extraction unit") extracts partial code (verification object) or partial code (learning use) (hereinafter collectively referred to as "partial code") from the source code group (verification object) 111 or the source code group (learning use) 211 (hereinafter collectively referred to as "source code group") (hereinafter referred to as "partial code"). In this example, the partial code extraction unit extracts partial code from the source code group on a method-by-method basis.

[0060] Additionally, the partial code extraction unit 120 extracts, for example, the descriptions of the methods corresponding to each method in the method list 112 of the verification object from the source code group (verification object) 111 as partial code (verification object). Furthermore, the partial code extraction unit 220 extracts, for example, the descriptions of each method included in the source code group (learning) 211 as partial code (learning).

[0061] Figure 2B This is a flowchart illustrating the partial code extraction process S300. The partial code extraction unit first obtains source code from the source code group (S311). Next, the partial code extraction unit extracts partial code from the obtained source code (S312).

[0062] Figure 3A This is a diagram illustrating another example of the partial code extraction process S300 (hereinafter referred to as "partial code extraction process S400"). In this example, the partial code extraction unit extracts code from a source code group (for example, a source code group that constitutes an information processing system, such as an application system), including the description of a method and the description of a method group (hereinafter referred to as "associated method group") that has a calling relationship with that method, as partial code.

[0063] As shown in the figure, the partial code extraction unit first generates a graph representing the calling relationship of each method from the source code group. Based on the generated graph, it extracts the code containing the description of a certain method and the description of the associated method group of that method as partial code.

[0064] Additionally, the partial code extraction unit 120 extracts code from the source code group (verification object) 111, specifically from the methods in the verification object method list 112, including the description of the method and the description of the associated method group, as partial code (verification object). Furthermore, the partial code extraction unit 220 extracts code from, for example, methods contained in the source code group (learning) 211, including the description of the method and the description of the associated method group, as partial code (learning).

[0065] Figure 3B This is a flowchart illustrating the partial code extraction process S400. As shown in the diagram, firstly, the partial code extraction unit obtains source code from the source code group (S411). Next, the partial code extraction unit generates a graph representing the call relationships between methods based on the obtained source code (S412). Then, for each method contained in the source code, the partial code extraction unit extracts code including the description of the method and the description of the method's associated method group as partial code (S413).

[0066] Figure 4A This is an explanation Figure 1 The diagram illustrates an example (hereinafter referred to as "feature vector generation process S500") of the feature vector generation unit 130 or feature vector generation unit 230 (hereinafter collectively referred to as "feature vector generation unit") transforming partial code (verification object) or partial code (learning material) (hereinafter collectively referred to as "partial code") into feature vectors. In this example, the feature vector generation unit generates feature vectors based on metric values ​​representing the features of the partial code. Metric values ​​include, for example, the number of lines in the partial code, the number of occurrences of loop statements, the maximum level of loop statements, the number of occurrences of conditional branch statements, the maximum level of conditional branch statements, the number of variables used, the number of method calls, the number of lines in the method calls, the number of occurrences of loop statements in the method calls, the maximum level of loop statements in the method calls, the number of occurrences of conditional branch statements in the method calls, the maximum level of conditional branch statements in the method calls, and the number of variables in the method calls. The feature vector generation unit generates a vector with these metric values ​​as its feature vector.

[0067] By using the metric values ​​of a portion of the code to generate feature vectors, it is possible to generate feature vectors that capture the metric values ​​of a portion of the code as features, and to perform performance verification based on the differences in metric values ​​of a portion of the code.

[0068] Figure 4BThis is a flowchart illustrating the feature vector generation process S500. The feature vector generation unit first obtains partial codes from the partial code group (S511). Next, the feature vector generation unit calculates the index value of each obtained partial code (S512). Then, for each partial code, the feature vector generation unit generates a vector with its respective index value as a feature vector (S513).

[0069] Figure 5A This is a diagram illustrating another example of the feature vector generation process S500 (hereinafter referred to as "feature vector generation process S600"). In this example, the feature vector generation unit generates feature vectors based on a word dictionary, which consists of information that establishes a correspondence between symbols and words expressed in a general form.

[0070] Specifically, the feature vector generation unit transforms words (method names (function names), variable types, variable names, data types, storage class specifiers, etc.) described in a portion of the code group extracted from the source code group (e.g., the source code group that constitutes an information processing system, such as an application system) into generalized expressions, generating a dictionary that establishes a correspondence between different symbols and each expression. In the example shown in the figure, the feature vector generation unit transforms the storage class specifier "static" described in the portion of the code group into "Static", the data type "void" into "Void", and the method name "loadData" into "mth#A", generating a dictionary that establishes a correspondence between the symbols "1", "2", "3", and "4" and each expression, respectively.

[0071] Next, the feature vector generation unit transforms the partial code of the transformation object into a feature vector by converting the words contained in that partial code into a general form of expression, obtains the symbols corresponding to the transformed expressions from a word dictionary, and generates a vector with each obtained symbol as an element as the feature vector. In this figure, the feature vector is generated by arranging the elements of the symbols "1", "2", and "3" corresponding to the general forms of the words "Static", "Void", and "mth#A" contained in the partial code of the transformation object in the order of their appearance in that partial code.

[0072] By generating feature vectors based on combinations of words described in parts of the code, it is possible to generate feature vectors that capture the construction of parts of the code as features, and to perform performance verification based on differences in the construction of parts of the code.

[0073] Figure 5BThis is a flowchart illustrating the process of generating a word dictionary from partial code groups in the feature vector generation process S600 (hereinafter referred to as "word dictionary generation process S610"). First, the feature vector generation unit preprocesses a portion of the code group (removing unnecessary descriptions such as comment statements) (S611). Next, the feature vector generation unit transforms the words described in the preprocessed partial code into generalized expressions (S612). Then, the feature vector generation unit extracts words expressed in generalized forms from the transformed partial code (S613), and generates a word dictionary by establishing a correspondence between symbols and each extracted word in generalized form (S614).

[0074] Figure 5C This is a flowchart illustrating the process in the feature vector generation process S600 where the feature vector generation unit transforms a portion of the code of the transformation object into a feature vector by referring to a word dictionary (hereinafter referred to as "feature vectorization processing"). First, the feature vector generation unit obtains a portion of the code of the transformation object and preprocesses the obtained portion of code (removing unnecessary descriptions such as comment statements) (S621). Next, the feature vector generation unit transforms the words described in the preprocessed portion of code into a general form of expression (S622). Then, the feature vector generation unit extracts words expressed in the general form from the transformed portion of code (S623), transforms each extracted word into a symbol by referring to a word dictionary, and generates a vector by arranging the transformed symbols in the order of appearance as the feature vector (S624).

[0075] Figure 6A This is an explanation Figure 1 The diagram shows an example of a process (hereinafter referred to as "learning process S710") in which the learning processing unit 250 uses the learning data from the learning data group 215 to train the performance verification model 216. This example illustrates a case where the performance verification model 216 is a binary classification model, and the performance verification model 216 outputs a single metric representing the performance of a portion of the code. By using a binary classification model as the performance verification model 216, it is possible to verify, for example, whether the performance meets the benchmark. Furthermore, in this example, we will illustrate the case where the aforementioned metric represents the probability that there is a problem with the performance of the processing based on the partial code implementation.

[0076] In this example, the learning processing unit 250 inputs the feature vector from the learning data (a group of feature vectors and labels) into the performance verification model 216, and adjusts the parameters of the performance verification model 216 based on the difference between the value of the input and output of the performance verification model 216 for the input ("0.3" in this example) and the label of the learning data ("0.0" in this example).

[0077] Figure 6BThis is a flowchart illustrating the learning process S710. The learning processing unit 250 first obtains learning data from the learning data group 215 and inputs the feature vectors of the obtained learning data into the performance verification model 216 (S711). Next, the learning processing unit 250 calculates the difference between the output of the performance verification model 216 and the label of the learning data (S712). Then, the learning processing unit 250 updates the parameters of the performance verification model 216 based on the calculated difference (S713).

[0078] Figure 7A This is an explanation Figure 1 The diagram shows another example (hereinafter referred to as "learning process S810") of the learning processing unit 250 using the learning data group 215 to train the performance verification model 216. This example is a case where the performance verification model 216 is a multi-class classification model, and the performance verification model 216 outputs multiple metrics (each based on a different viewpoint) representing the performance of a portion of the code. By using a multi-class classification model as the performance verification model 216, it is possible to verify whether the performance meets the benchmark from multiple different viewpoints (e.g., loop count, memory operation count, etc.).

[0079] In this example, the learning processing unit 250 inputs the feature vectors from the learning data (a group of feature vectors and labels) into the performance verification model 216, and adjusts the parameters of the performance verification model 216 based on the difference between the values ​​of the performance verification model 216 for each viewpoint of the input and output (in this example, "number of loops: 0.3", "number of memory operations: 0.3") and the labels of the learning data (in this example, "number of loops occurring: 1", "number of memory operations occurring: 0") .

[0080] Figure 7B This is a flowchart illustrating an example of the learning process S810. The learning processing unit 250 first obtains learning data from the learning data group 215 and inputs the feature vectors of the obtained learning data into the performance verification model 216 (S811). Next, the learning processing unit 250 calculates the difference between the output of the performance verification model 216 and the label of the learning data for each viewpoint (S812). Then, based on the calculated difference, the learning processing unit 250 updates the parameters of the performance verification model 216 (S813).

[0081] As explained above, the performance verification system 1 of this embodiment verifies the performance of the source code (including the performance of the executable code of the source code, including the partial code) by extracting a portion of the code from the source code, transforming the extracted portion of the code into a feature vector, and inputting the transformed feature vector into the performance verification model 216. Thus, the performance verification system 1 verifies performance based on the source code, allowing performance verification even before, for example, generating executable code. Therefore, users, such as developers, can verify the performance of the source code while writing it down, reducing rework compared to verification after generating executable code. Furthermore, it allows for early detection of code that could potentially become a performance bottleneck within the source code. Moreover, since the performance verification system 1 verifies performance based on the source code, there is no need to prepare test scenario settings or test data. Furthermore, the performance verification system 1 is applicable to programming languages ​​that do not require compilation, such as interpreted languages. Thus, the performance verification system 1 of this embodiment significantly reduces the burden of verifying software performance during software development or maintenance, enabling efficient system development and maintenance.

[0082] Next, an applicable example (application example) of performance verification system 1 will be explained.

[0083] Figure 8A This diagram illustrates an application example of the performance verification system 1. In this example, a user developing or maintaining software uses user device 2 (information processing device) to create source code. User device 2 sends a verification request for the user-created source code to performance verification unit 100. Performance verification unit 100 receives the verification request, generates a verification result 115 for the source code, and sends the generated verification result 115 to user device 2.

[0084] By utilizing the performance verification system 1 in this way, users can, for example, use an integrated development environment (IDE) running on user device 2 to easily and quickly verify the performance of the source code while writing it as the object of development or maintenance. Therefore, users can perform development or maintenance work while simultaneously verifying the performance of the source code, enabling them to efficiently produce high-quality software that meets performance requirements.

[0085] Figure 8B This means that in Figure 8AThe flowchart illustrates the processing performed by user device 2 and performance verification unit 100 in the illustrated application example. First, user device 2 sends a verification request attached to the source code (source code group (verification object) 111 and verification object method list 112) to performance verification unit 100 (S911). Based on the received source code, performance verification unit 100 generates a verification result 115 (S912) and sends the generated verification result 115 to user device 2 (S913). If user device 2 receives the verification result 115, it displays its contents to the user (S914).

[0086] Figure 9A This diagram illustrates another application example of the performance verification system 1. This example depicts a scenario where multiple users collaborate on software development or maintenance using a collaborative development environment 60. The collaborative development environment 60 is communicatively connected to multiple user devices 2 operated by different users, and manages a group 62 of source code as objects of development or maintenance in a repository 61.

[0087] The co-development environment 60 is communicatively connected to the performance verification unit 100. If the co-development environment 60 receives source code and a registration request for the source code from the user device 2, it sends the verification request along with the received source code to the performance verification unit 100. The performance verification unit 100 receives the verification request, generates a verification result 115 for the source code, and sends the generated verification result 115 to the co-development environment 60.

[0088] If the performance indicated by the received verification result 115 meets the preset performance requirements, the collaborative development environment 60 registers the updated source code into the repository 61. On the other hand, if the performance indicated by the received verification result 115 does not meet the preset performance requirements, the collaborative development environment 60 does not register the updated source code into the repository 61, and sends a notification to the user device 2 that the source code does not meet the performance requirements.

[0089] By utilizing the performance verification system 1 in this way, managing only the source code group 62 that meets the performance requirements in the repository 61 of the co-development environment 60 can consistently ensure the quality of the software.

[0090] Figure 9B This means that in Figure 9AThe flowchart illustrates the processes performed by the user device 2, the collaborative development environment 60, and the performance verification unit 100 in the illustrated application example. The collaborative development environment 60 monitors the registration request attached to the source code (source code group (verification object) 111 and verification object method list 112) received from the user device 2 (S1011: No). If a registration request is received (S1011: Yes), the system sends the verification request for the source code, along with the received source code, to the performance verification unit 100 (S1012). If the performance verification unit 100 receives the source code and the verification request, it generates a verification result 115 for the received source code and sends the generated verification result 115 to the collaborative development environment 60 (S1013). If the collaborative development environment 60 receives the verification result 115 (S1014), it determines whether the performance represented by the verification result 115 meets the preset performance requirements (S1015). If the performance indicated by verification result 115 does not meet the preset performance requirements (S1015: No), the co-development environment 60 sends a notification to user device 2 that the source code received in S1011 does not meet the performance requirements and cannot be registered in the repository (registration disallowance notification) (S1021), and then the process returns to S1011. On the other hand, if the performance indicated by verification result 115 meets the preset performance requirements (S1015: Yes), the co-development environment 60 registers the source code received in S1011 into the repository and sends a registration completion notification of the source code to user device 2 (S1016). Then the process returns to S1011.

[0091] Figure 10 This is an example of an information processing device (computer) used in the performance verification system 1, user device 2, and co-development environment 60 described above. The illustrated information processing device 10 includes a processor 101, a main storage device 102, an auxiliary storage device 103, an input device 104, an output device 105, and a communication device 106.

[0092] Alternatively, the illustrated information processing device 10 may be wholly or partially implemented using virtual information processing resources provided by virtualization technology and process space separation technology, such as virtual servers provided by a cloud system. Furthermore, all or part of the functions provided by the information processing device 10 may also be implemented through services provided by the cloud system via APIs (Application Program Interfaces). Additionally, a single information processing device 10 may be configured as a shared structure for two or more components, such as a performance verification system 1, a user device 2, and a collaborative development environment 60.

[0093] In this figure, the processor 101 may be composed of, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or an AI (Artificial Intelligence) chip.

[0094] The main storage device 102 is a device for storing programs and data, such as ROM (Read Only Memory), RAM (Random Access Memory), or non-volatile RAM (NVRAM).

[0095] The auxiliary storage device 103 may be, for example, an SSD (Solid State Drive), a hard disk drive, an optical storage device (CD (Compact Disc), DVD (Digital Versatile Disc), etc.), a storage system, an IC card, an SD card, or a recording medium read / write device, or a storage area of ​​a cloud server. Programs and data can be read into the auxiliary storage device 103 via a recording medium read device or a communication device 106. Programs and data stored in the auxiliary storage device 103 can be read into the main storage device 102 at any time.

[0096] The input device 104 is an interface that accepts input from the outside, such as a keyboard, mouse, touch panel, card reader, tablet computer with pen input, voice input device, etc.

[0097] Output device 105 is an interface for outputting various information such as processing progress and processing results. Output device 105 can be, for example, a display device that visualizes the aforementioned information (liquid crystal monitor, LCD, graphics card, etc.), a device that converts the aforementioned information into sound (sound output device (speaker, etc.)), or a device that converts the aforementioned information into characters (printer, etc.). Alternatively, it can be configured such that the information processing device 10 can input or output information to other devices via the communication device 106.

[0098] The input device 104 and the output device 105 constitute a user interface for receiving and prompting information to the user.

[0099] The communication device 106 (communication unit) is a device that enables communication with other devices. The communication device 106 is a wired or wireless communication interface that enables communication with other devices via a communication network (control system network 50, information / control system network 51, information system network 52), such as a NIC (Network Interface Card), a wireless communication module, a USB module, etc.

[0100] In the information processing device 10, for example, an operating system, file system, DBMS (Database Management System) (relational database, NoSQL, etc.), KVS (Key-Value Store), and various other software (software, middleware, and various applications that implement user interfaces based on GUI (Graphical User Interface) through input device 104 and output device 105).

[0101] The functions of the performance verification system 1, user device 2, and co-development environment 60 can be implemented, for example, by the processor 101 reading and executing the program stored in the main storage device 102, or by the hardware (FPGA, ASIC, AI chip, etc.) of the information processing device 10. Furthermore, various data stored in the performance verification system 1, user device 2, and co-development environment 60 are stored in the main storage device 102 or the auxiliary storage device 103 (storage unit).

[0102] Furthermore, all or part of the various functions of the performance verification system 1 can be implemented using, for example, various known data mining methods such as text data mining, various known processing methods (morphological parsing, syntactic parsing, semantic parsing, text comprehension, feature extraction, word dispersion representation, intrinsic representation extraction, text classification, series labeling), and various known machine learning methods (deep learning (DNN (Deep Neural Network), RNN (Recurrent Neural Network), etc.)).

[0103] The present invention has been described above as an embodiment, but the present invention is not limited to the above embodiment, and various modifications can be made without departing from its spirit. For example, the above embodiment has been described in detail for the purpose of easily understanding the present invention, and is not limited to having all the structures described. Furthermore, regarding a part of the structure of the above embodiment, other structures can be added, deleted, or replaced.

[0104] For example, by modifying the extraction method of the above partial code ( Figure 2A , Figure 3A The above-mentioned method for generating feature vectors is a variation of ( ). Figure 4A , Figure 5A ), variations of model learning methods ( Figure 6A , Figure 7A By combining them in various ways, a variety of implementation methods can be conceived.

[0105] Furthermore, while the above embodiments illustrate extracting code portions from the code on a "method" basis, it is also possible to extract code portions from the code on other types of processing blocks, such as "functions" or "classes." Additionally, code portions may be extracted according to a user-defined extraction method.

[0106] Furthermore, some or all of the aforementioned structures, functional units, processing units, and processing mechanisms can be designed into integrated circuits and implemented in hardware. Alternatively, the aforementioned structures and functions can be implemented in software by a processor interpreting and executing programs that perform each function. Information such as programs, tables, and files that implement each function can be stored in memory or recording devices such as hard disks, SSDs (Solid State Drives), IC cards, SD cards, DVDs, etc.

[0107] Furthermore, the configurations of the various functional units, processing units, and databases of the information processing devices described above are merely examples. The configurations of these functional units, processing units, and databases can be modified to achieve the optimal configuration based on the performance, processing efficiency, and communication efficiency of the hardware or software possessed by these devices.

[0108] Furthermore, the structure (schema, etc.) of the database that stores the aforementioned data can be flexibly changed from the perspectives of efficient resource utilization, improved processing efficiency, improved access efficiency, and improved retrieval efficiency.

[0109] Label Explanation

[0110] 1 Performance Verification System; 2 User Device; 60 Collaborative Development Environment; 61 Repository; 62 Source Code Group; 100 Performance Verification Department; 111 Source Code Group (Verification Object); 112 List of Verification Object Methods; 113 Partial Code Group (Verification Object); 114 Feature Vector Group (Verification Object); 115 Verification Results; 120 Partial Code Extraction Department; 130 Feature Vector Generation Department; 140 Performance Verification Processing Department; 200 Model Learning Department; 211 Source Code Group (for Learning); 212 Performance Labels; 213 Partial Code Set (for learning); 214 Feature Vector Set (for learning); 215 Learning Data Set; 216 Performance Validation Model; 220 Partial Code Extraction Unit; 230 Feature Vector Generation Unit; 240 Learning Data Generation Unit; 250 Learning Processing Unit; S300 Partial Code Extraction Processing; S400 Partial Code Extraction Processing; S500 Feature Vector Generation Processing; S610 Word Dictionary Generation Processing; S620 Feature Vectorization Processing; S710 Learning Processing; S810 Learning Processing.

Claims

1. A software performance verification system, characterized in that, It is composed of information processing devices; The aforementioned software performance verification system possesses: Storage section, which stores the code of the program that makes up the software; The partial code extraction section extracts the portion of code that is part of the aforementioned code. The feature vector generation unit generates feature vectors based on the aforementioned code. as well as The performance verification processing unit generates information as the verification result of the code portion being verified by inputting the aforementioned partial code of the verification object into a performance verification model. The performance verification model is a machine learning model that has been trained using learning data including the feature vectors of the aforementioned partial code and performance information. The performance information represents the performance of the software implemented based on the aforementioned partial code. The aforementioned storage unit stores a word dictionary containing information that establishes a correspondence between descriptions of words used in the code describing the software, converted into general forms, and values ​​assigned to those descriptions. The aforementioned feature vector generation unit transforms the words contained in the aforementioned code into generalized expressions, obtains the values ​​corresponding to the aforementioned expressions of the aforementioned words from the aforementioned word dictionary, and generates a vector with the obtained values ​​as elements as the aforementioned feature vector.

2. The software performance verification system as described in claim 1, characterized in that, The code above includes a description of the method. The code extraction section extracts the code from the above code by method.

3. The software performance verification system as described in claim 1, characterized in that, The code above includes a description of the method. The code extraction section above extracts a description that includes the description of a method and the descriptions of other methods that have a calling relationship with that method, and this is used as the code above.

4. The software performance verification system as described in claim 1, characterized in that, The aforementioned feature vector generation unit generates a vector with index values ​​obtained from the aforementioned code as elements, which serves as the aforementioned feature vector.

5. The software performance verification system as described in claim 1, characterized in that, The performance verification model above outputs the probability that there are problems with the performance of the processing based on the above-mentioned code.

6. The software performance verification system as described in claim 1, characterized in that, The performance verification model described above outputs multiple metrics based on different perspectives, which represent the performance of the processing implemented based on the aforementioned code.

7. The software performance verification system as described in claim 2, characterized in that, It has a user interface that allows the user to accept the specified code and the methods described in the code. The code extraction section extracts the description of the method specified above for the code specified by the user, and uses it as the code section.

8. The software performance verification system as described in claim 1, characterized in that, It also has a communications department that communicates with the collaborative development environment, which manages the code developed jointly by multiple users in a repository and determines whether the code can be registered with the repository based on the verification results. In the event that the aforementioned communications department receives a verification request for the aforementioned code from the aforementioned joint development environment, The code extraction section extracts the aforementioned code from the code above. The aforementioned feature vector generation unit generates feature vectors based on the aforementioned code. The aforementioned performance verification processing unit inputs the aforementioned feature vector of the aforementioned code into the aforementioned performance verification model to generate the aforementioned verification result, and sends the aforementioned verification result to the aforementioned joint development environment.

9. A software performance verification method, characterized in that, The information processing device performs the following: The steps of storing the code that makes up the software program; The steps to extract the code portion that is part of the code above; The steps for generating feature vectors based on the above code snippet; as well as The step of generating information as the verification result of a portion of the code by inputting the aforementioned portion of the code of the verification object into the performance verification model, wherein the performance verification model is a machine learning model that has been learned using learning data including the aforementioned feature vector of the aforementioned portion of the code used for learning and performance information, wherein the aforementioned performance information represents the performance of the software implemented based on the aforementioned portion of the code. The aforementioned information processing device stores a dictionary containing information that establishes a correspondence between expressions obtained by transforming the words used in the code describing the software into their general forms and values ​​set for those expressions. In the step of generating feature vectors based on the above-mentioned code, the words contained in the above-mentioned code are transformed into generalized expressions, the values ​​corresponding to the above-mentioned expressions of the above-mentioned words are obtained from the above-mentioned word dictionary, and a vector with the obtained values ​​as elements is generated as the above-mentioned feature vector.

Images