Design specification answer system and design specification answer method

Abstract

To answer a question to a design specification with high accuracy by using logic information of a design specification in an information processing system and physical information of a source code or the like.SOLUTION: A design specification answer system 1 for inputting a question 6 inputted from a user 5 and related to a design specification of the information processing system to a language model 4 and outputting an acquired answer 7 to the user 5, inputs meta information 32 including a correspondence relation between a design specification 33 and a source code 34 related to the information processing system as text information in the language model 4, retrieves the design specification 33 and the source code 34 on the basis of a retrieval range to the design specification 33 and the source code 34 and information of a retrieval keyword obtained by inputting the question 6 in the language model 4, performs re-question processing for re-inquiring the language model 4 for the design specification 33 relating to a retrieval result and the information of the source code 34 as context information, and outputs information related to the design specification answered from the language model 4 to the user 5.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to technology that utilizes generative AI and LLMs (Large Language Models), and in particular to technology that is effective when applied to a design specification answering system and design specification answering method that answer questions about the design specifications of an information processing system. [Background technology]

[0002] In the field of information processing system development and maintenance, users routinely ask questions, make inquiries, and confirm information about system design specifications, such as "Please tell me the judgment conditions for the ZZ check on the YY screen of the XX system." Manual responses to such questions by administrators or staff are highly subjective, and there are limitations to providing accurate and precise responses. In addition, so-called FAQs (Frequently Asked Questions) are sometimes created, but there are limitations in terms of the burden of creating them and the comprehensiveness of the questions they can cover.

[0003] On the other hand, recently, LLMs that have demonstrated amazing conversational and understanding abilities, such as ChatGPT (registered trademark), have become available, and it is possible to flexibly respond to questions from users in a conversational format. However, when using such LLMs, basically only answers based on the contents that have already been learned can be given. For example, when using a public LLM such as ChatGPT, it is difficult to obtain answers based on information that is private and highly confidential, such as information on the design specifications of an information processing system, and is inappropriate for a public LLM to learn.

[0004] In response to this, a method known as prompt engineering is used in which the text of the question (prompt) entered into the LLM is used to provide the LLM with information that was previously unknown to the LLM, without requiring additional learning (so-called fine-tuning), and an answer is obtained based on this.

[0005] As a related technique, for example, Non-Patent Document 1 describes the construction of a Q&A system that reads private data while utilizing public LLMs such as ChatGPT through prompt engineering and indexing of internal documents. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] “RetrievalQA and ChatGPT: QA tools for internal documents,” [online], GMO Internet Group, Inc., [Retrieved August 16, 2023], Internet <URL:https: / / recruit.gmo.jp / engineer / jisedai / blog / qa_tools_with_retrievalqa_and_chatgpt / > Summary of the Invention [Problem to be solved by the invention]

[0007] According to conventional technology such as that described in non-patent document 1, text data containing design specifications, such as design documents and manuals for information processing systems, can be indexed / vectorized, and design specification information similar to the text of a question can be extracted through calculations. This can then be used for prompt engineering when entering a question into the LLM, making it possible to obtain answers to questions about the design specifications from the LLM.

[0008] However, the contents written in design documents, manuals, etc. do not comprehensively and accurately express the entire behavior of the information processing system, but can be said to be expressions in natural language (logical information) of the concept, design philosophy, requirements, instructions, etc. In contrast, the actual behavior of the system is specified by the source code (physical information), and in order to answer questions about accurate design specifications that take into account the actual behavior of the information processing system, it is insufficient to provide an appropriate answer based only on logical information such as design documents, or physical information such as the source code.

[0009] Therefore, an object of the present invention is to provide a design specification reply system and a design specification reply method that reply to questions about design specifications with high accuracy by using logical information such as design documents in an information processing system and physical information such as source code.

[0010] The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings. [Means for solving the problem]

[0011] Among the inventions disclosed in this application, a brief summary of representative inventions is as follows.

[0012] A design specification response system that is a representative embodiment of the present invention is a design specification response system that inputs a question related to the design specifications of an information processing system input by a user into a language model, and outputs the response obtained from the language model to the user. Meta-information including the correspondence between the design document and source code related to the information processing system is input into the language model as context information, the design document and the source code are searched based on information on the search range and search keywords for the design document and the source code obtained by inputting the question into the language model, a re-questioning process is performed to re-question the language model with information on the design document and the source code related to the search result as context information, and the information related to the design specification answered from the language model is output to the user.

[0013] The present invention can also be applied to a method executed in the design specification reply system described above. Effect of the Invention

[0014] The effects obtained by the representative inventions disclosed in this application can be briefly explained as follows.

[0015] That is, according to a representative embodiment of the present invention, it is possible to answer questions about design specifications with high accuracy by using logical information such as design documents in an information processing system and physical information such as source code. [Brief description of the drawings]

[0016] [Figure 1] 1 is a diagram showing an overview of a configuration example of a design specification reply system according to an embodiment of the present invention; [Diagram 2] FIG. 2 is a diagram showing an outline of an example of a question input by a user and a reply given to the user in one embodiment of the present invention. [Diagram 3] 1 is a flowchart outlining an example of a process flow related to a design specification reply in one embodiment of the present invention. [Figure 4] FIG. 2 is a diagram showing an overview of an example of meta information according to an embodiment of the present invention. [Diagram 5] FIG. 13 is a diagram showing an overview of an example of information related to a reply rule in the embodiment of the present invention. [Figure 6] FIG. 13 is a diagram showing an outline of an example of a response from a language model in one embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In all the drawings for explaining the embodiment, the same parts are generally given the same reference numerals, and the repeated explanations will be omitted. On the other hand, a part that has been described with a reference numeral in a certain drawing may be referred to with the same reference numeral in the explanation of another drawing, although it will not be shown again.

[0018] <Summary> In the design specification answering system according to one embodiment of the present invention, not only logical information such as design documents in an information processing system but also physical information such as source code is used as the basis for answers, and the LLM is provided with information on the correspondence and structure between logical information such as design documents and physical information such as source code as a context, which enables the LLM to answer questions about design specifications with higher accuracy. In addition, in this embodiment, the LLM proposes search ranges and search keywords for accumulated design documents and source codes as answers, and the LLM is asked questions again using the design documents and source codes searched for using the keywords. By repeating such exchanges of questions and answers between this system and the LLM, it is possible to arrive at the desired answer.

[0019] <System configuration> 1 is a diagram showing an overview of a configuration example of a design specification reply system according to an embodiment of the present invention. The design specification reply system 1 is composed of, for example, a server device, a virtual server constructed on a cloud computing service, a PC, etc., and realizes various functions for replying to questions related to the design specifications of an information processing system by executing an OS (Operating System), a DBMS (DataBase Management System), middleware such as a Web server program, and software running thereon, which are expanded on a memory from a recording device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive) by a CPU (Central Processing Unit) not shown.

[0020] This design specification reply system 1 has, for example, a question processing unit 2 implemented as software. It also has data such as an index 31 and meta information 32 generated for a design document 33, which is a design document or manual related to a target information processing system, and a source code 34, which is a source code or a setting file.

[0021] The question processing unit 2 has an interface function for using the language model 4 and a function related to prompt engineering at that time, in which the question 6 inputted from the user 5 is received, a design document 33 and a source code 34 thought to be related to the question 6 are obtained, and the obtained answer is inputted to the language model 4 such as LLM as a context by being included in the question 6, and the answer obtained is outputted to the user 5 as an answer 7. For example, ChatGPT can be used as the language model 4, and the question processing unit 2 can be constructed using a framework for implementing In-Context Learning such as LlamaIndex, but is not limited thereto. Details of the processing by the question processing unit 2 will be described later.

[0022] The index 31 is a database that holds data that has been indexed or vectorized in advance for each design document 33 and source code 34. If the design document 33 is a drawing or table that is not text data, the data may be converted to text as necessary before being indexed or vectorized.

[0023] For text conversion, indexing, and vectorization, publicly available publicly known libraries (for example, various text conversion libraries of Python (registered trademark), vectorization libraries such as LlamaIndex and LangChain, etc.) can be used as appropriate. By indexing the design document 33 and the source code 34 and having the index 31, the query processing unit 2 can quickly search the design document 33 and the source code 34 using a full-text search engine such as Elasticsearch (registered trademark).

[0024] The meta information 32 is text data that defines in advance the correspondence and related structure between the design documents 33 (logical information) and the source code 34 (physical information). The meta information 32 includes, for example, category information and format information of each design document 33, information related to the correspondence between the design documents 33, and information related to the correspondence between the design documents 33 and the source code 34. The meta information 32 is created in advance, for example, by a developer of the target information processing system based on the design documents 33 and the source code 34. If possible, the meta information 32 may be automatically generated using a tool or the like.

[0025] <Example questions and answers> 2 is a diagram outlining an example of a question 6 input by a user 5 in one embodiment of the present invention and an example of an answer 7 given to the user 5. The upper diagram shows an example of the question 6, and the lower diagram shows an example of the answer 7.

[0026] In the example in the figure, answer 7 to question 6 regarding the design specifications of the "XX system" ("Decision conditions for ZZ check on YY screen") provides the design specifications ("There is one order condition, and the order amount is smaller than the available order amount"), along with providing design document 33 ("YY screen.aaa") in which the specifications are stated, and source code 34 ("xxAction.java", "xxCheck.java") in which the specifications are implemented, as well as the relevant parts of the code.

[0027] <Processing flow> 3 is a flowchart outlining an example of a process flow for answering design specifications in an embodiment of the present invention. First, meta-information 32 created in advance for a target information processing system and information related to rules for answering questions are input in advance as context to the language model 4 (S01).

[0028] Note that, when there are multiple target information processing systems or when the target information processing system has multiple functions, for example, meta information 32 created in advance for each information processing system or function, and information related to rules for answering questions may be prepared, and when the user 5 selects any one of the information processing systems or functions, the meta information 32 related to the selected information processing system or function and the information related to rules for answering questions may be input in advance as context to the language model 4. This makes it possible to respond to questions directed to multiple information processing systems or an information processing system having multiple functions.

[0029] In addition, the contents of the meta information 32 and / or information relating to the rules for answering questions may be presented to the user 5 before being input into the language model 4, allowing the user 5 to edit the information as necessary. For example, the user may be able to add or replace additional information, files, etc. relating to the meta information 32.

[0030] 4 is a diagram showing an example of meta information 32 in one embodiment of the present invention. In the example in the diagram, it is shown that the instruction to the language model 4 is to give meta information 32 such as an overview of the target information processing system, an overview of each design document 33 and source code 34, their associated structures, and information on their location (index).

[0031] 5 is a diagram showing an overview of an example of information related to answering rules in an embodiment of the present invention. As shown in the figure, similar to the meta information 32 in FIG. 4, the content defines the method and rules for answering questions as instructions to the language model 4. In this embodiment, as will be described later, as answering rules, a search range (index) and search keywords for identifying design documents 33 and source code 34 considered to be related to the corresponding design specifications are output as answers until a desired answer 7 (design specifications corresponding to question 6) is obtained as an output of the language model 4.

[0032] Returning to FIG. 3, thereafter, the question processing unit 2 accepts input of a question 6 from the user 5 (S02), inputs this to the language model 4 (S03), and obtains an answer from the language model 4 (S04). The question 6 from the user 5 may also be input in accordance with a predetermined rule (for example, "include the category name of the target design document 33 as a keyword"). In the example of FIG. 3, the meta information 32 is input to the language model 4 in step S01 before the question 6 from the user 5 is accepted, but it may be input to the language model 4 before the question 6 is input to the language model 4 in step S03 after the question 6 from the user 5 is accepted. At this time, the meta information 32 to be input may be narrowed down according to the content and target range of the question 6.

[0033] The answer acquired from the language model 4 in step S04 includes information on a search range and search keywords for identifying the design document 33 and source code 34 considered to be related to the target design specification, in accordance with the instructions shown in the example of Fig. 5. Fig. 6 is a diagram outlining an example of an answer from the language model 4 in one embodiment of the present invention. As shown in the example in the figure, the search range and search keywords are specified by the language model 4, and an instruction is given to search the design document 33 and source code 34.

[0034] The query processing unit 2 extracts information on the search range and search keywords from the answer (S05), and searches the design document 33 and source code 34 by full-text search using the index 31 based on the extracted information (S06).Then, the information on the design document 33 and source code 34 obtained as the search results is included in the context information and a query is posed again to the language model 4 (S07), and a reply is obtained from the language model 4 (S08).

[0035] Then, it is determined whether the desired answer 7 is obtained in the answer, that is, whether the design document 33 and source code 34 related to the design specification corresponding to the question 6 are identified by the language model 4 and the target design specification is answered based on these (S09). This determination may be performed manually by the user 5, or automatically by the question processing unit 2 according to a predetermined criterion based on the text included in the answer. The predetermined criterion may be included in the information related to the rule for answering the question input to the language model 4 in step S01, for example, and further, the predetermined criterion may be edited by the user 5 as necessary.

[0036] If the target design specification is not answered in the response (No in step S09), Returning to step S05, the more precise search range and search keyword information contained in the answer is extracted, and the design document 33 and source code 34 are searched (S06). The information on the design document 33 and source code 34 obtained as the search results is included in the context information and queried again to the language model 4 (S07). As a result of repeating this process, if an answer related to the design specifications is finally obtained from the language model 4 as shown in the lower diagram of the example in Figure 2 (Yes in step S09), the answer is output to the user 5 as answer 7 (S10), and the process ends.

[0037] As described above, according to the design specification reply system 1 which is an embodiment of the present invention, not only the design document 33 (logical information) but also the source code 34 (physical information) are included as the basis of the reply to the question 6 related to the design specification of the information processing system, and the corresponding relationship and related structure between the design document 33 (logical information) and the source code 34 (physical information) are defined as meta information 32 and given to the language model 4 as context information. As a result, for the question 6 in natural language (logical information), the language model 4 converts logical information into physical information, identifies the part (physical information) of the source code 34 corresponding to the reply 7, and takes into account the consistency with the design document 33 (logical information), making it possible to output the reply 7 with higher accuracy.

[0038] In addition, by defining the method and rules for answering questions for the language model 4 and inputting this as context information, it is possible for the question processing unit 2 to automatically repeat questions and answers between the language model 4 based on meta-information 32 relating to the correspondence and structure between the above-mentioned design document 33 (logical information) and source code 34 (physical information), and arrive at the desired answer (design specifications corresponding to question 6).

[0039] The invention made by the present inventor has been specifically described above based on the embodiment, but it goes without saying that the present invention is not limited to the above embodiment and can be modified in various ways without departing from the gist of the invention. Furthermore, the above embodiment has been described in detail to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to having all of the configurations described. Furthermore, it is possible to add, delete, or replace part of the configuration of the above embodiment with other configurations.

[0040] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the program, table, file, etc. that realizes each function can be stored in a recording device such as a memory, a hard disk, or an SSD, or in a recording medium such as an IC card, an SD card, or a DVD.

[0041] In addition, in each of the above figures, the control lines and information lines are shown as those considered necessary for explanation, and do not necessarily show all the control lines and information lines in the actual implementation. In reality, it can be considered that almost all the components are connected to each other. [Industrial Applicability]

[0042] The present invention can be used in a design specification reply system and a design specification reply method for replying to questions about the design specifications of an information processing system. [Explanation of symbols]

[0043] 1...design specification answer system, 2...question processing unit, 4...language model, 5...user, 6...question, 7...answer, 31...Index, 32...Meta information, 33...Design document, 34...Source code

Claims

1. A design specification response system that inputs a question related to a design specification of an information processing system input by a user into a language model and outputs an answer obtained from the language model to the user, comprising: inputting meta-information including a correspondence between a design document and a source code relating to the information processing system into the language model as context information; a re-interrogation process for re-interrogating the language model using information on the design document and the source code related to the search result as context information; and an output process for outputting information related to the design specification provided by the language model to the user.

2. In the design specification reply system according to claim 1, the re-questioning process is repeated for the language model until information related to the design specification is returned as a response to the re-questioning process.

3. 3. The design specification reply system according to claim 2, A design specification answering system that inputs, as context information, instructions to answer information on a search range and search keywords for the design document and the source code to the language model until information related to the design specification is obtained as an answer to a question.

4. A design specification answering method by a design specification answering system, which inputs a question related to a design specification of an information processing system input by a user into a language model, and outputs an answer obtained from the language model to the user, comprising: The design specification response system, inputting meta-information including a correspondence between a design document and source code related to the information processing system into the language model as context information; a step of searching the design document and the source code based on information on a search range and a search keyword for the design document and the source code obtained by inputting the query into the language model, and re-querying the language model using information on the design document and the source code related to the search result as context information; and outputting information related to the design specifications answered from the language model to the user.

Images