Information processing program, information processing method, and information processing device

Abstract

Improve data recognition accuracy. [Solution] An information processing device 10 inputs a first instruction statement 4a to a dialogue system 12a using a machine learning model 3, instructing the dialogue system 12a to estimate first information based on data 1 including multiple pieces of first information and multiple pieces of second information representing relationships between the multiple pieces of first information. The information processing device 10 inputs a second instruction statement 4b to the dialogue system 12a instructing the dialogue system 12a to estimate the meaning of the first information based on third information related to the meaning of the first information. The information processing device 10 inputs a third instruction statement 4c to the dialogue system 12a instructing the dialogue system 12a to estimate second information based on data 1. The information processing device 10 obtains output information 4g from the dialogue system 12a, which is generated based on the second information estimated in accordance with the third instruction statement 4c.

Description

[Technical Field]

[0001] The present invention relates to an information processing program, an information processing method, and an information processing device. [Background technology]

[0002] One type of model generated by machine learning is the Large Language Model (LLM). An LLM is a machine learning model that understands natural language and other content and generates answers. For example, an LLM is composed of a neural network with many parameters. An LLM can be used as generative AI (artificial intelligence) that generates answers in natural language. Generative AI also includes multimodal dialogue technology that combines not only linguistic information but also image information.

[0003] AI technologies include a table image recognizer that can extract the correct structure from complex tables. A machine learning model using generative models has also been proposed for use within creative visual editors. A system has also been proposed for recognizing the arrangement of multiple objects on a computing device. A system has also been proposed for automatically extracting information from flowchart images. Techniques have also been proposed for training and using task-oriented interactive systems. Furthermore, techniques have been proposed for recognizing individual symbols and understanding the structure of offline hand-drawn diagrams. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2023 / 188362 [Patent Document 2] US Patent Application Publication No. 2024 / 0135611 [Patent Document 3] Special Publication No. 2019-506672 [Patent Document 4] U.S. Patent Application Publication No. 2021 / 0365679 [Patent Document 5] Japanese Patent Publication No. 2024-027070 [Non-patent literature]

[0005] [Non-Patent Document 1] Jiaqi Fang, Zhen Feng and Bo Cai, "DrawnNet: Offline Hand-Drawn Diagram Recognition Based on Keypoint Prediction of Aggregating Geometric Characteristics", Entropy, 2022, 24, 425, MDPI, 19 March 2022 Summary of the Invention [Problem to be solved by the invention]

[0006] Data to be recognized using a machine learning model may contain types of information that are difficult to correctly recognize. For example, in a flow diagram, the process flow or screen transitions are represented by nodes and edges. When a flow diagram is recognized using a conventional machine learning model, the accuracy of recognizing edge connections decreases when the start and end points of the edges are unclear or when edges overlap. If the edge connections are incorrectly recognized, for example, the process flow represented in the flow diagram cannot be correctly recognized.

[0007] In this way, when a machine learning model is used to recognize data, there is a possibility that the model may not recognize all of the information accurately enough, resulting in an incorrect recognition of the entire data. In one aspect, the present invention aims to improve the accuracy of data recognition. [Means for solving the problem]

[0008] In one proposal, an information processing program is provided that causes a computer to perform the following processing. The computer inputs a first instruction statement to a dialogue system using a machine learning model, instructing the system to estimate the first information based on data including a plurality of pieces of first information and a plurality of pieces of second information representing relationships between the plurality of pieces of first information. The computer inputs a second instruction statement to the dialogue system instructing the system to estimate the meaning of the first information based on third information related to the meaning of the first information. The computer inputs a third instruction statement to the dialogue system instructing the system to estimate the second information based on the data. The computer then obtains output information from the dialogue system that is generated based on the second information estimated in accordance with the third instruction statement. [Effects of the Invention]

[0009] According to one aspect, the accuracy of data recognition is improved. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 2 illustrates an example of an information processing method according to a first embodiment. [Figure 2] FIG. 10 illustrates an example of a system configuration according to a second embodiment. [Figure 3] FIG. 2 illustrates an example of hardware of a terminal device. [Figure 4] FIG. 10 is a diagram illustrating a first example of a flow diagram. [Figure 5] FIG. 10 is a diagram illustrating a second example of a flow diagram. [Figure 6] FIG. 2 is a block diagram showing the functions of each device for document processing using LLM. [Figure 7] FIG. 10 is a diagram illustrating an example of recognition of a flow diagram. [Figure 8] FIG. 10 is a diagram showing an example of a legend diagram. [Figure 9] FIG. 10 is a flowchart illustrating an example of a processing procedure for a document task. [Figure 10] FIG. 2 is a diagram illustrating an example of a document to be processed. [Figure 11] FIG. 10 is a flowchart showing an example of a procedure for a flow diagram analysis process. [Figure 12] FIG. 1 shows an example of a flow diagram analysis (1 / 6). [Figure 13] FIG. 2 shows an example of a flow diagram analysis (2 / 6). [Figure 14] FIG. 3 / 6 shows an example of flow diagram analysis. [Figure 15] FIG. 4 / 6 shows an example of flow diagram analysis. [Figure 16] FIG. 5 / 6 shows an example of flow diagram analysis. [Figure 17] FIG. 6 / 6 shows an example of a flow diagram analysis. [Figure 18] FIG. 1 is a diagram showing an example of a document including a handwritten flow diagram. [Figure 19] FIG. 10 is a diagram illustrating an example of image understanding of a legend diagram. [Figure 20] FIG. 10 is a diagram showing an example of a result of inferring the meaning of a node. [Figure 21] FIG. 10 is a diagram illustrating an example of a recognition result of a flow diagram. DETAILED DESCRIPTION OF THE INVENTION

[0011] The present embodiment will be described below with reference to the drawings. Note that each embodiment can be implemented in combination with a plurality of other embodiments within a range that does not contradict each other. [First embodiment] The first embodiment is an information processing method for recognizing data with high accuracy using a machine learning model.

[0012] Fig. 1 is a diagram showing an example of an information processing method according to a first embodiment. Fig. 1 shows an information processing device 10 for implementing the information processing method according to the first embodiment. The information processing device 10 can implement the information processing method according to the first embodiment by, for example, executing an information processing program.

[0013] The information processing device 10 includes a storage unit 11 and a processing unit 12. The storage unit 11 is, for example, a memory or storage device included in the information processing device 10. The processing unit 12 is, for example, a processor included in the information processing device 10.

[0014] The memory unit 11 stores, for example, data 1 and legend information 2. Data 1 includes a plurality of pieces of first information and a plurality of pieces of second information that represent the relationships between the plurality of pieces of first information. Data 1 is, for example, a flow diagram made up of nodes and edges. In the flow diagram, a node is an example of first information, and an edge is an example of second information. Legend information 2 is information that indicates the display mode for each meaning of the first information. For example, if the first information is a node, the meaning of each node shape is indicated in legend information 2.

[0015] The processing unit 12 is capable of executing a dialogue system 12a. The dialogue system 12a is an information processing function capable of conducting dialogue in natural language through thought chain reasoning using a machine learning model 3. The machine learning model 3 is a multimodal model that can accept, for example, an image or a sentence in natural language as input.

[0016] The processing unit 12 uses the dialogue system 12a to execute a predetermined process on the data 1. For example, the processing unit 12 inputs a first instruction statement 4a to the dialogue system 12a, which instructs the dialogue system 12a to estimate first information based on the data 1. As a result, the dialogue system 12a estimates the first information (e.g., a node) included in the data 1.

[0017] Thereafter, the processing unit 12 inputs an arbitrary instruction sentence to the dialogue system 12a, and further inputs a second instruction sentence 4b to the dialogue system 12a, the second instruction sentence 4b instructing the dialogue system 12a to estimate the meaning of the first information based on third information related to the meaning of the first information. As a result, the dialogue system 12a estimates the meaning of the first information.

[0018] Furthermore, the processing unit 12 inputs to the dialogue system 12a a third instruction sentence 4c that instructs the dialogue system 12a to estimate the second information based on the data 1. As a result, the dialogue system 12a estimates the second information included in the data 1.

[0019] The processing unit 12 then acquires, from the dialogue system 12a, output information 4g generated based on the second information estimated in accordance with the third instruction sentence 4c. For example, information indicating the relationship between the first information indicated in the second information is acquired as the output information 4g.

[0020] In this way, the processing unit 12 first causes the dialogue system 12a to estimate the first information and its meaning, and then causes the dialogue system 12a to estimate multiple pieces of second information that indicate the relationships between the multiple pieces of first information. The dialogue system 12a performs thought-chain inference using the machine learning model 3, so the second information is estimated taking into account the estimation results of the first information and its meaning. Because the meaning of the first information has already been recognized when the second information is estimated, it is possible to estimate the second information that indicates the relationships between the first information by taking into account the meaning of the first information. As a result, the estimation accuracy of the second information is improved.

[0021] The improvement in estimation accuracy of the second information also improves the recognition accuracy for the entire data 1. As a result, the processing unit 12 can perform various inference processes for the data 1 with high reliability, for example.

[0022] As the third information, for example, legend information 2 is used. For example, before inputting the second instruction statement 4b to the dialogue system 12a, the processing unit 12 inputs a fourth instruction statement 4d to the dialogue system 12a instructing the dialogue system 12a to recognize legend information 2 indicating the display mode for each meaning of the first information. Then, when inputting the second instruction statement 4b to the dialogue system 12a, the processing unit 12 sets the legend information 2 recognized in accordance with the fourth instruction statement 4d as the third information.

[0023] By using legend information 2 as third information related to the meaning of the first information in this way, it becomes possible to accurately estimate the meaning of the first information when the first information has a display mode according to its meaning. By recognizing the meaning of the first information with high accuracy, for example, the estimation accuracy of second information that connects first information having the same kind of meaning is improved.

[0024] Furthermore, when the first information is represented by a graphic of a different shape for each meaning, the processing unit 12 may treat the classification result of the first information based on the shape as the third information. For example, the processing unit 12 inputs a fifth instruction statement 4e that instructs the dialogue system 12a to classify the first information by shape. As a result, the first information is classified by shape. First information with the same shape will belong to the same group. Then, when inputting the second instruction statement 4b to the dialogue system 12a, the processing unit 12 treats the classification result of the first information classified according to the fifth instruction statement 4e as the third information.

[0025] By treating the classification result as third information related to the meaning of the first information in this way, it becomes possible to accurately estimate a set of first information having the same meaning. By recognizing a set of first information having the same meaning, for example, the estimation accuracy of second information that connects first information having the same kind of meaning is improved.

[0026] Furthermore, the processing unit 12 may classify the first information based on words and sentences included in the nodes that are the first information as the third information. When the data 1 is a flow diagram, the nodes, which are the first information, are represented by figures with different shapes for each meaning. The edges, which are the second information, are represented by lines drawn between two figures. In this case, the processing unit 12 may cause the dialogue system 12a to estimate the edges by effectively utilizing the intersections between the nodes and the edges. For example, the processing unit 12 inputs a sixth directive 4f to the dialogue system 12a, instructing the dialogue system 12a to estimate the intersections between the node figures and the edge lines. As a result, the dialogue system 12a estimates the intersections between the nodes and the edges. When inputting the third directive 4c to the dialogue system 12a, the processing unit 12 sets the third directive 4c to a sentence instructing the dialogue system 12a to perform estimation that places emphasis on the meaning of the first information related to the intersections estimated in accordance with the sixth directive 4f.

[0027] In this way, the processing unit 12 instructs the dialogue system 12a to estimate the intersection and estimate the edge by placing emphasis on the meaning of the first information related to the intersection. This prevents the dialogue system 12a from being guided into excessive inference about non-intersecting points. As a result, the accuracy of edge estimation is improved.

[0028] Second Embodiment The second embodiment is a computer system that executes document processing tasks including flow diagrams (flowcharts) using LLM services provided over a network such as a cloud computing system. A flow diagram is a diagram that shows the flow of processes, screens, etc., using nodes represented by shapes of predetermined shapes and edges represented by lines connecting the nodes. Flow diagrams include various diagrams that combine nodes and edges, such as sequence diagrams and UML (Unified Modeling Language) activity diagrams.

[0029] 2 is a diagram illustrating an example of a system configuration according to the second embodiment. A terminal device 100 is connected to a server 200 via a network 20. The terminal device 100 is a computer used by a user. The server 200 is a computer that provides services using LLM.

[0030] 3 is a diagram showing an example of hardware of a terminal device. The entire terminal device 100 is controlled by a processor 101. A memory 102 and a plurality of peripheral devices are connected to the processor 101 via a bus 109.

[0031] The terminal device 100 may be a multiprocessor system having multiple processors. A collection of multiple processors in a multiprocessor system may be referred to as a processor 101. The processor 101 may also be referred to as a processor circuitry. Each of the multiple processors may execute some or all of the multiple processes executed in the terminal device 100. When there are multiple related processes, a processor that executes one process among the multiple processors may be different from a processor that executes a process other than the related process.

[0032] The processor 101 is, for example, a central processing unit (CPU), a micro processing unit (MPU), or a digital signal processor (DSP). At least some of the functions realized by the processor 101 executing a program may be realized by an electronic circuit such as an application specific integrated circuit (ASIC) or a programmable logic device (PLD).

[0033] The memory 102 is used as a main storage device of the terminal device 100. The memory 102 temporarily stores at least a part of the OS (Operating System) programs and application programs to be executed by the processor 101. The memory 102 also stores various data used in processing by the processor 101. As the memory 102, for example, a volatile semiconductor storage device such as a RAM (Random Access Memory) is used.

[0034] The peripheral devices connected to the bus 109 include a storage device 103, a graphics controller 104, an input interface 105, an optical drive device 106, a device connection interface 107, and a network interface 108.

[0035] The storage device 103 electrically or magnetically writes and reads data to and from a built-in recording medium. The storage device 103 is used as an auxiliary storage device for the terminal device 100. The storage device 103 stores an OS program, application programs, and various data. Note that the storage device 103 may be, for example, an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0036] The graphics controller 104 is an arithmetic device that performs image processing. The graphics controller 104 is, for example, a GPU (Graphics Processing Unit). A monitor 21 is connected to the graphics controller 104. The graphics controller 104 displays an image on the screen of the monitor 21 in accordance with an instruction from the processor 101. The monitor 21 may be a display device using organic EL (Electro Luminescence) or a liquid crystal display device. When a GPU is used as the graphics controller 104, for example, the graphics controller 104 can also perform complex numerical calculations such as matrix calculations.

[0037] The input interface 105 is connected to a keyboard 22 and a mouse 23. The input interface 105 transmits signals sent from the keyboard 22 and the mouse 23 to the processor 101. The mouse 23 is an example of a pointing device, and other pointing devices can also be used. Examples of other pointing devices include a touch panel, a tablet, a touch pad, and a trackball.

[0038] The optical drive device 106 uses a laser beam or the like to read data recorded on an optical disc 24 or write data to the optical disc 24. The optical disc 24 is a portable recording medium on which data is recorded so that it can be read by reflected light. The optical disc 24 includes a DVD (Digital Versatile Disc), a DVD-RAM, a CD-ROM (Compact Disc Read Only Memory), a CD-R (Recordable) / RW (Rewritable), and the like.

[0039] The device connection interface 107 is a communication interface for connecting peripheral devices to the terminal device 100. For example, a memory device 25 or a memory reader / writer 26 can be connected to the device connection interface 107. The memory device 25 is a recording medium equipped with a function for communicating with the device connection interface 107. The memory reader / writer 26 is a device for writing data to the memory card 27 or reading data from the memory card 27. The memory card 27 is a card-type recording medium.

[0040] The network interface 108 is connected to the network 20. The network interface 108 transmits and receives data to and from other computers or communication devices via the network 20. The network interface 108 is a wired communication interface connected by a cable to a wired communication device such as a switch or a router. The network interface 108 may also be a wireless communication interface connected by radio waves to a wireless communication device such as a base station or an access point.

[0041] The terminal device 100 can realize the processing functions of the second embodiment by using the hardware described above. Note that the device shown in the first embodiment can also be realized by using the same hardware as the terminal device 100 shown in FIG.

[0042] The terminal device 100 realizes the processing functions of the second embodiment by executing a program recorded on, for example, a computer-readable recording medium. The program describing the processing to be executed by the terminal device 100 can be recorded on various recording media. For example, the program to be executed by the terminal device 100 can be stored in the storage device 103. The processor 101 loads at least a part of the program in the storage device 103 into the memory 102 and executes the program. The program to be executed by the terminal device 100 can also be recorded on a portable recording medium such as the optical disk 24, the memory device 25, or the memory card 27. The program stored on the portable recording medium becomes executable after being installed on the storage device 103 under the control of, for example, the processor 101. The processor 101 can also directly read and execute the program from the portable recording medium.

[0043] Here, assume that the LLM service provided by server 200 is used to review, for example, a design document for system development. The system development design document may contain a flow chart showing the flow of processing. By using an LLM capable of image recognition, it is possible to check whether the flow of processing shown in the flow chart in the design document complies with the requirements for system development.

[0044] Even with image recognition, LLMs can sometimes have insufficient accuracy in recognizing diagrams consisting of nodes and edges, such as flow diagrams commonly used in the field of software engineering. For example, when the edges have complex shapes (such as dotted lines), when the nodes have a multi-layered structure, or when edges overlap with the boundaries of seam lanes, the accuracy of recognizing the connection destination of the edges decreases.

[0045] Fig. 4 is a diagram showing a first example of a flow diagram. Fig. 4 shows a screen system diagram 30, which shows screen transitions, as an example of a flow diagram. The screen system diagram 30 is made up of process nodes 31a to 31d, screen nodes 32a to 32g, and edges 33a to 33e that connect two of the screen nodes 32a to 32g. The process nodes 31a to 31d represent data processing functions, and the screen nodes 32a to 32g represent input / output screens for data processing. The edges 33a to 33e show transitions between screens.

[0046] When such a screen system diagram 30 is recognized by an LLM, the accuracy of recognition of the connection between the edges 33a to 33e and the screen nodes 32a to 32g decreases because the frames of the figures representing the process nodes 31a to 31d intersect with the edges 33a to 33e.

[0047] Fig. 5 is a diagram showing a second example of a flow diagram. Fig. 5 shows a sequence diagram 40 showing a processing flow as an example of a flow diagram. In the sequence diagram 40, swimlanes 41a to 41c are provided for each execution subject. Each of the swimlanes 41a to 41c shows the processing executed by the corresponding execution subject.

[0048] Sequence diagram 40 includes start nodes 42a and 42b indicating the start of processing and end nodes 43a and 43b indicating the end of processing. Sequence diagram 40 also includes processing nodes 44a and 44b, a branch node 44c, etc. These nodes are connected by edges 46a to 46e.

[0049] Furthermore, data nodes 45a to 45d representing data tables are connected to processing node 44a, which uses the data, by edges 47a to 47d. Similarly, data nodes 45e to 45h are connected to processing node 44b, which uses the data, by edges 47e to 47h.

[0050] In the sequence diagram 40, the lines separating the swimlanes 41a to 41c intersect with the edges 46a, 46c, and 46e. Such intersections can easily lead to misrecognition of the start and end points of the edges 46a, 46c, and 46e when the sequence diagram 40 is input into the LLM.

[0051] If there is an error in the LLM's understanding of the flow diagram, the inference will be made under erroneous conditions, and as a result, the answers generated by the LLM may also be incorrect. Therefore, the terminal device 100 improves the recognition accuracy of flow diagrams by using an appropriate Chain-of-Thought (CoT). CoT is a prompting method that provides information to the LLM in stages to allow it to execute a continuous inference process.

[0052] For example, the terminal device 100 provides the LLM with inference results regarding elements in an image with high recognition accuracy as supplementary information, and then uses that information to gradually infer elements in the image with low recognition accuracy. In the case of a flow diagram, nodes have higher recognition accuracy than edges. Moreover, because the meaning of a node is expressed by its shape, the meaning of the node is also easy to infer. Therefore, the terminal device 100 has the LLM recognize the nodes and their meanings, and then has the LLM recognize edges based on the recognition results. Such appropriate prompting prevents recognition errors in flow diagrams.

[0053] Figure 6 is a block diagram showing the functions of each device for document processing using LLM. Server 200 has LLM 210 and dialogue system 220. LLM 210 is a trained model of an image-compatible multimodal neural network. Input data to LLM 210 can include text data, image data, and audio data. Output data from LLM 210 can also include text data, image data, and audio data.

[0054] The dialogue system 220 uses the LLM 210 to conduct a dialogue in natural language. The dialogue system 220 is, for example, a service called a chatbot. The dialogue system 220 converts instructions (prompts) sent from the terminal device 100 to the LLM 210 into input data for the LLM 210. The dialogue system 220 then inputs the generated input data to the LLM 210 to obtain an output from the LLM 210. The dialogue system 220 transmits the output from the LLM 210 to the terminal device 100.

[0055] The terminal device 100 includes a document processor 110 and a flow diagram analysis control unit 120. The document processor 110 executes a task specified by a user for a document. For example, the document processor 110 executes a task for reviewing a design document. When a processing request is input by a user, the document processor 110 transmits a prompt to the server 200 instructing the server 200 to extract a flow diagram from the input document. At this time, the document processor 110 may instruct the server 200 to extract a legend diagram indicating a legend for the flow diagram along with the flow diagram. When the flow diagram is extracted, the document processor 110 instructs the flow diagram analysis control unit 120 to analyze the flow diagram. After completing the analysis of the flow diagram, the document processor 110 executes task processing for the document using the analysis results of the flow diagram. The document processor 110 appropriately uses the LLM 210 in executing the task processing.

[0056] The flow diagram analysis control unit 120 analyzes the flow diagram using the LLM 210 by using the CoT. For example, the flow diagram analysis control unit 120 instructs the server 200 to use the LLM 210 to perform inference on the nodes of the flow diagram. The flow diagram analysis control unit 120 further instructs the server 200 to use the LLM 210 to perform inference on the semantic information of the nodes. Thereafter, the flow diagram analysis control unit 120 instructs the server 200 to use the LLM 210 to perform inference on the edges connecting the nodes of the flow diagram. The flow diagram analysis control unit 120 transmits the recognition results of the flow diagram to the document processing unit 110.

[0057] The function of each element shown in FIG. 6 can be realized by, for example, having the processor 101 execute a program module corresponding to that element. In this way, the flow diagram analysis control unit 120 first causes the LLM 210 to perform inference on elements with high recognition accuracy, such as node information, and then causes the LLM 210 to perform inference on elements with low recognition accuracy, such as edges. This improves inference accuracy.

[0058] Fig. 7 is a diagram showing an example of recognition of a flow diagram, which shows a process of recognizing the relationship between process nodes 31a and 31b and screen nodes 32a and 32b in the screen system diagram 30 (see Fig. 4).

[0059] When the image of the screen system diagram 30 is input to the LLM 210 by the flow diagram analysis control unit 120, the LLM 210 performs image understanding. Through image understanding, the LLM 210 understands that there are lines indicating, for example, process nodes 31a and 31b and screen nodes 32a and 32b. The results of image understanding are temporarily saved in the server 200.

[0060] Next, based on instructions from the flow diagram analysis control unit 120, the LLM 210 recognizes nodes from the understood image. In the LLM 210, for example, an area surrounded by a closed curve is determined to be a node. When there are intersecting lines, the LLM 210 can also recognize an area surrounded by a closed curve obtained by ignoring one of the lines as a node.

[0061] The LLM 210 can recognize nodes with high accuracy. Therefore, the process nodes 31a and 31b and the screen nodes 32a and 32b are correctly recognized. For example, the process node 31a is recognized as the process node "FD001:Login." The process node 31b is recognized as the process node "FD002:Menu." The screen node 32a is recognized as the screen node "FD001F01:Login." The screen node 32b is recognized as the screen node "FD002F01:Menu."

[0062] Then, based on instructions from the flow diagram analysis control unit 120, the LLM 210 estimates edges from the understood image. For example, since the nodes have already been correctly recognized, lines connecting the already recognized nodes, excluding closed curves surrounding the nodes, are estimated as edges. As a result, edge 33a connecting screen nodes 32a and 32b is correctly estimated.

[0063] The flow diagram analysis control unit 120 may instruct the server 200 to use the LLM 210 to enumerate line intersections based on the image understanding results. In this case, the flow diagram analysis control unit 120 instructs the LLM 210 to infer edge information while taking into consideration the semantic information of the nodes in the surrounding areas. This prevents excessive inference from leading to incorrect meanings for non-intersecting points.

[0064] Furthermore, if there is a legend diagram, the flow diagram analysis control unit 120 can use the information in the legend to infer semantic information. 8 is a diagram showing an example of a legend diagram. For example, a legend for the nodes and edges of the screen system diagram 30 is shown in a legend diagram 50. For example, a process node example 51a shows the shapes of process nodes 31a and 31b and the meanings of the characters displayed within the process nodes 31a and 31b. According to the process node example 51a, the process nodes 31a and 31b are double-lined rectangles, and the process ID and process name are displayed inside.

[0065] Screen node example 51b shows the shapes of screen nodes 32a and 32b and the meanings of the characters shown in those screen nodes 32a and 32b. According to screen node example 51b, screen nodes 32a and 32b are rectangles with rounded corners, and the screen ID and screen name are displayed inside.

[0066] Edge example 52a shows the type of edge that represents a screen transition. According to edge example 52a, the edge that represents a screen transition is a solid arrow. Edge example 52b shows the type of edge that represents a modal call. According to edge example 52b, the edge that represents a modal call is a dashed arrow. Edge example 52c shows the type of edge that represents a modeless call. According to edge example 52c, the edge that represents a modeless call is a dashed arrow.

[0067] By having the LLM 210 recognize a flow diagram based on such a legend 50, the accuracy of the flow diagram recognition is improved. If there is no legend 50, the flow diagram analysis control unit 120 may, for example, have the LLM 210 enumerate nodes of the same shape and infer the meaning of each node shape. In the field of software engineering, the shapes of nodes are generally uniform, and there is a high possibility that meaning is embedded in the shapes. Therefore, by having the LLM 210 infer the meaning of each node of the same shape, the accuracy of the meaning inference is improved.

[0068] Next, a specific procedure for executing a task using the LLM 210 for a document including a flow diagram will be described. 9 is a flow diagram showing an example of a processing procedure for a document task. The processing shown in FIG. 9 will be explained below in order of step number.

[0069] [Step S101] The document processor 110 acquires a document processing request. The processing request specifies the document to be processed. For example, the processing request includes the document to be processed. The processing request may also include the storage location (path) and file name of the document.

[0070] [Step S102] The document processor 110 transmits an instruction to the server 200 to extract a flow chart and a legend chart using the LLM 210. The extraction instruction is, for example, a character string written in a natural language. The extraction instruction includes the document to be processed.

[0071] In the server 200, the dialogue system 220 converts the extraction instructions into input data for the LLM 210. The dialogue system 220 inputs the converted input data to the LLM 210, executes processing in accordance with the LLM 210, and obtains output data. The dialogue system 220 generates answer data based on the output data from the LLM 210. The output data includes, for example, an extracted flow diagram or legend diagram. The dialogue system 220 transmits the answer data to the document processing unit 110.

[0072] [Step S103] The document processor 110 determines whether a flow diagram has been extracted by the extraction process by the LLM 210. If a flow diagram has been extracted, the document processor 110 instructs the flow diagram analysis control unit 120 to analyze the flow diagram, and proceeds to step S104. If a flow diagram has not been extracted, the document processor 110 proceeds to step S105.

[0073] [Step S104] The flow diagram analysis control unit 120 executes a flow diagram analysis process. The flow diagram analysis process will be described in detail later (see FIG. 11). The flow diagram analysis control unit 120 transmits the result of the flow diagram analysis process to the document processing unit 110.

[0074] [Step S105] The document processor 110 executes a document processing task in response to the processing request. At this time, if there is a result of analyzing the flow chart, the document processor 110 executes the task using the result of analyzing the flow chart.

[0075] In this way, if the document to be processed contains a flow diagram, the task will be executed using the analysis results of that flow diagram.If the document to be processed contains a legend for the flow diagram, the accuracy of flow diagram recognition will be improved by having the LLM recognize the legend.

[0076] Fig. 10 is a diagram showing an example of a document to be processed. The document to be processed shown in Fig. 10 is a design document 60 for software system development. The design document 60 shows software specifications and the like. The design document 60 includes a flow diagram 61 showing the software specifications. The design document 60 also includes a legend diagram 62 showing a legend for the flow diagram 61.

[0077] For example, when a processing request for a task to review a design document 60 is input to the document processing unit 110, the document processing unit 110 transmits an instruction to the server 200 to extract a flow diagram 61 and a legend diagram 62 from the design document 60. The server 200 then extracts the flow diagram 61 and the legend diagram 62 using the LLM 210. After the flow diagram 61 has been extracted, the document processing unit 110 instructs the flow diagram analysis control unit 120 to perform flow diagram analysis processing.

[0078] 11 is a flow diagram showing an example of the procedure of the flow diagram analysis process. The process shown in FIG. 11 will be explained below in order of step number. [Step S201] The flow diagram analysis control unit 120 transmits an image of a flow diagram to be understood by the LLM 210 to the server 200. In the server 200, the dialogue system 220 inputs the image of the flow diagram along with a character string instructing the LLM 210 to understand the image. The dialogue system 220 then executes processing in accordance with the LLM 210 and obtains output data. The output data is information such as lines and character strings contained in the image. The dialogue system 220 stores the output data in a memory or the like and transmits response data indicating that the flow diagram has been understood to the document processing unit 110.

[0079] [Step S202] The flow diagram analysis control unit 120 determines whether a legend has been extracted in the processing of step S102 (see FIG. 9). If a legend has been extracted, the flow diagram analysis control unit 120 proceeds to step S203. If a legend has not been extracted, the flow diagram analysis control unit 120 proceeds to step S204.

[0080] [Step S203] The flow diagram analysis control unit 120 transmits an image of the legend to be understood by the LLM 210 to the server 200. In the server 200, the dialogue system 220 inputs the image of the legend along with a character string instructing the LLM 210 to understand the image. The dialogue system 220 then executes processing in accordance with the LLM 210 and obtains output data. The output data is information about example nodes and their explanations, and example edges and their explanations, shown in the legend. The dialogue system 220 saves the output data in a memory or the like and transmits response data indicating that the legend has been understood to the document processing unit 110.

[0081] [Step S204] The flow diagram analysis control unit 120 transmits a command to recognize nodes in the flow diagram to the server 200. In the server 200, the dialogue system 220 inputs the recognition results of the flow diagram, etc., recognized in steps S201 to S203, and a character string instructing node recognition to the LLM 210. The dialogue system 220 then executes processing in accordance with the LLM 210 and obtains output data. The output data is information about nodes estimated to be included in the flow diagram. The dialogue system 220 stores the output data in a memory or the like, and transmits answer data indicating information about the recognized nodes to the document processing unit 110.

[0082] [Step S205] The flow diagram analysis control unit 120 transmits an instruction to classify the nodes by their shape to the server 200. In the server 200, the dialogue system 220 inputs the recognition results obtained in steps S201 to S204 and a character string indicating an instruction to classify the nodes by their shape to the LLM 210. The dialogue system 220 then executes processing in accordance with the LLM 210 and obtains output data. The output data is a list of nodes with the same shape. The dialogue system 220 stores the output data in a memory or the like, and transmits answer data indicating information on the classification of the recognized nodes by their same shape to the document processing unit 110.

[0083] [Step S206] The flow diagram analysis control unit 120 transmits an instruction to estimate the line intersection points to the server 200. In the server 200, the dialogue system 220 inputs the recognition results obtained in steps S201 to S205 and data instructing the LLM 210 to estimate the line intersection points. The dialogue system 220 then executes processing in accordance with the LLM 210 and obtains output data. The output data is information indicating the line intersection points (intersection points). The dialogue system 220 stores the output data in a memory or the like, and transmits response data indicating the line intersection points to the document processing unit 110.

[0084] [Step S207] The flow diagram analysis control unit 120 transmits an instruction to infer the meaning of the node to the server 200. In the server 200, the dialogue system 220 inputs the recognition results obtained in steps S201 to S205, the line intersection points estimated in step S206, and data instructing inference of the meaning of the node to the LLM 210. The dialogue system 220 then executes processing in accordance with the LLM 210 and obtains output data. The output data is information indicating the inference result of the meaning of the node. The dialogue system 220 stores the output data in a memory or the like, and transmits answer data indicating the inference result of the meaning of the node to the document processing unit 110.

[0085] [Step S208] The flow diagram analysis control unit 120 transmits an edge estimation instruction to the server 200. In the server 200, the dialogue system 220 inputs the recognition results obtained in steps S201 to S205, the estimation results obtained in steps S206 to S207, and data instructing the estimation of the meaning of the edge to the LLM 210. The dialogue system 220 then executes processing in accordance with the LLM 210 and obtains output data. The output data is information indicating the estimation result of the meaning of the edge. The dialogue system 220 stores the output data in a memory or the like, and transmits response data indicating the estimation result of the meaning of the edge to the document processing unit 110.

[0086] In this way, the recognition of nodes with high recognition accuracy and the estimation of their meaning are performed first, and then the estimation of edges is performed. Because the nodes are estimated with high accuracy first, the estimation accuracy of subsequent edges also improves. As a result, the flow diagram is recognized with high accuracy.

[0087] Hereinafter, examples of prompts sent from the terminal device 100 to the server 200 for recognizing the contents of the flow chart and answers to those prompts will be described with reference to FIGS.

[0088] 12 is a diagram (1 / 6) showing an example of flow diagram analysis. Image data 71 of a flow diagram is transmitted from the terminal device 100 to the server 200. The server 200 performs image recognition of the flow diagram using the LLM 210. The server 200 transmits response data 72 to the terminal device 100 indicating that the image of the flow diagram has been recognized.

[0089] If a legend diagram is available, image data 73 of the legend diagram is then transmitted from the terminal device 100 to the server 200. The server 200 performs image recognition of the legend diagram using the LLM 210. The server 200 transmits response data 74 to the terminal device 100 indicating that the legend diagram has been recognized.

[0090] FIG. 13 is a diagram (2 / 6) showing an example of flow diagram analysis. A prompt 75 is sent from the terminal device 100 to the server 200 to instruct extraction of node information from the flow diagram. For example, the prompt 75 may read, "Please input an image of the screen system diagram of the software design document. Please list the information about the nodes included in the image." Based on the prompt 75, the server 200 generates node information 76 using the LLM 210. The server 200 then transmits the generated node information 76 to the terminal device 100.

[0091] For example, if the flow diagram is the screen system diagram 30 shown in FIG. 4, the node information 76 is the following character string. "1.FD001: Login 1. FD001F01: Login 2.FD002: Menu 1. FD002F01: Menu 3.FD101: Check store information (head office) 1. FD101F01: Store List 2. FD101F02: Store details 4.FD102: Order Processing 1. FD102F01: Purchase Order Entry 2. FD102F02: Voucher search (code reference) 3. FD102F03: Product Selection (Code Reference) In this way, the nodes in the flow diagram are extracted. At this stage, only the nodes and the character strings within the nodes are extracted, and the meaning of the nodes is not yet interpreted.

[0092] FIG. 14 is a diagram (3 / 6) showing an example of flow diagram analysis. A prompt 77 is transmitted from the terminal device 100 to the server 200, instructing the user to list nodes of the same shape (classify the nodes). For example, the prompt 77 may read, "In the image information and the node information previously listed, please list nodes of the same shape by shape." Based on the prompt 77, the server 200 generates a classification result 78 using the LLM 210. The server 200 then transmits the generated classification result 78 to the terminal device 100.

[0093] For example, if the flow chart is the screen system diagram 30 shown in FIG. 4, the classification result 78 is the following character string. "1. Rectangular node: 1.FD001: Login 2.FD002: Menu 3.FD101: Check store information (head office) 4.FD102: Order Processing 2. Rectangular node with rounded corners: 1.FD001F01: Login 2.FD002F01: Menu 3.FD101F01: Store List 4.FD101F02: Store details 5.FD102F01: Purchase Order Entry 6.FD102F02: Voucher Search (Code Reference) 7.FD102F03: Product selection (reference code) In this way, nodes are classified according to the shape of the figure that represents them. It can be assumed that nodes with the same shape represent similar meanings.

[0094] FIG. 15 is a diagram (4 / 6) showing an example of flow diagram analysis. A prompt 79 instructing the server 200 to list the points where lines intersect is transmitted from the terminal device 100 to the server 200. For example, the prompt 79 may read, "Please list the points where lines intersect, including the contours and edges of the nodes." Based on the prompt 79, the server 200 generates intersection point information 80 using the LLM 210. The server 200 then transmits the generated intersection point information 80 to the terminal device 100. The intersection point information 80 indicates information about the intersection points.

[0095] For example, if the flow chart is the screen system diagram 30 shown in FIG. 4, the intersection point information 80 is the following character string. "1. The edge from FD001F01:Login intersects with FD001:Login and the outer border of FD002:Menu. 2. The edge from FD002F02:Menu intersects with the outer frame of FD002:Menu, FD101:Store Information Check (Head Office), and FD101:Order Processing. By having the server 200 generate such intersection point information 80, the terminal device 100 can cause the server 200 to perform inference focusing on the periphery of the intersection point in the subsequent inference process.

[0096] FIG. 16 is a diagram (5 / 6) showing an example of flow diagram analysis. A prompt 81 instructing the server 200 to infer the meaning of a node is transmitted from the terminal device 100 to the server 200. For example, the prompt 81 may be a sentence such as, "Please infer the meaning of each node based on the legend and the shape information of the nodes." Based on the prompt 81, the server 200 generates a node semantic inference result 82 using the LLM 210. The server 200 then transmits the generated node semantic inference result 82 to the terminal device 100.

[0097] For example, if the flow diagram is the screen system diagram 30 shown in Figure 4, the node semantic inference result 82 is the sentence, "Rectangular nodes may represent functions, and rectangular nodes with rounded corners may represent screens."

[0098] By estimating edges after accurately recognizing the meaning of nodes, edges can be recognized with high accuracy. FIG. 17 is a diagram (6 / 6) showing an example of flow diagram analysis. A prompt 83 instructing edge estimation is transmitted from the terminal device 100 to the server 200. For example, the prompt 83 may read, "Please list the edges based on the image. In particular, please estimate the edges near the intersections of the lines based on the semantic information of the nodes." Based on the prompt 83, the server 200 generates an edge estimation result 84 using the LLM 210. The server 200 then transmits the generated edge estimation result 84 to the terminal device 100.

[0099] For example, if the flow chart is the screen system diagram 30 shown in FIG. 4, the edge estimation result 84 is the following character string. “Rectangular nodes could represent features, rectangular nodes with rounded corners could represent screens. Based on that information, the edge connections are enumerated. Login (FD001F01) → Menu (FD002F01) Menu (FD002F01) → Store Information Check (Head Office) Store List (FD101F01) Store List (FD101F01) → Store Details (FD101F02) Menu (FD002F01) → Purchase Order Entry (FD102F01) in Purchase Order Processing Purchase Order Input (FD102F01) → Voucher Search (Code Reference) (FD102F02) Enter purchase order slip (FD102F01) → Select product (reference code) (FD102F03) In this way, the edges connecting the nodes in the screen system diagram 30 are recognized with high accuracy. By correctly recognizing the flow diagram, tasks for the document to be processed can also be executed with high accuracy. For example, when reviewing a software design document, task execution correctly checks whether the flow diagram is consistent with other descriptions.

[0100] 13 to 17 are set in advance in a storage area managed by, for example, the flow diagram analysis control unit 120 of the terminal device 100. Then, the flow diagram analysis control unit 120 reads out a prompt matching the content of the instruction at the timing of an instruction to the server 200, and transmits the prompt to the server 200.

[0101] By performing such processing, the recognition accuracy can be improved even for handwritten flow charts, for example. FIG. 18 is a diagram showing an example of a document including a handwritten flow diagram. The document 90 includes a handwritten flow diagram 91 and a legend 92. The flow diagram 91 includes, for example, a parallelogram node 91a. The parallelogram node 91a has a different meaning from the rectangular node. However, in the handwritten flow diagram 91, it may be difficult to correctly recognize the parallelogram node 91a. In this case, by effectively using the legend 92, the recognition accuracy of the parallelogram node 91a can be improved.

[0102] For example, the terminal device 100 instructs the server 200 to perform image understanding of the flow diagram 91 using the LLM 210 and image understanding of the legend diagram 92 using the LLM 210 . 19 is a diagram showing an example of image understanding of a legend map. The server 200 performs image understanding of the legend map 92 using the LLM 210, and the LLM 210 outputs a legend map understanding result 92a. The area surrounded by a dashed line in the legend map understanding result 92a indicates that the parallelogram nodes represent data input. When the terminal device 100 instructs the server 200 to infer the meaning of the node, the server 200 infers the meaning of the node using the legend map understanding result 92a.

[0103] 20 is a diagram showing an example of the inference result of the meaning of a node. The shape of the rectangle obtained by image understanding of the flow diagram 91 and the legend diagram understanding result 92a are input to the LLM 210, where the meaning of the node is inferred and, for example, node information 93 is output. The node information 93 indicates each node in the flow diagram 91 and the meaning of that node. By recognizing the edges connecting the nodes based on the node information 93, the entire flow diagram 91 can be recognized.

[0104] 21 is a diagram showing an example of the results of flow diagram recognition. Based on the flow diagram 91 and node information 93, the entire flow diagram 91, which is made up of nodes and edges, is recognized using the LLM 210. For example, the information of the recognized flow diagram 91 is converted into a DSL (Domain-Specific Language) to generate flow diagram data representing the flow diagram 91. A recognized flow diagram 94 is displayed based on the flow diagram data. In the recognized flow diagram 94, the nodes represented by parallelograms in the flow diagram 91 are correctly recognized as data input nodes 94a.

[0105] In this multimodal system, which is designed to handle image recognition results in flow diagrams using an LLM, the terminal device 100 first causes the server 200 to perform inference on node information and other information using the LLM 210, which has high recognition accuracy. The terminal device 100 then causes the server 200 to perform inference on node semantic information using the LLM 210. Based on the inference results, the terminal device 100 causes the server 200 to perform step-by-step edge inference using the LLM 210. This improves the inference accuracy of edges, which originally had difficulty in recognition accuracy, and makes it possible to provide accurate information in response to user questions and requests when using the LLM 210.

[0106] Other Embodiments In the second embodiment, the terminal device 100 and the server 200 work together to recognize the flow diagram, but for example, the functions of the terminal device 100 and the server 200 may be implemented in a single computer.

[0107] Furthermore, even if the diagram is other than a flow diagram (for example, a graph), as long as it is made up of nodes and edges, the content can be recognized with high accuracy by the processing shown in the second embodiment. Furthermore, when two types of objects with different recognition accuracy are displayed in an image, the recognition accuracy of the object with the lower recognition accuracy can be improved by processing similar to that of the second embodiment. For example, the terminal device 100 causes the LLM 210 to recognize the object with the higher recognition accuracy first, and then uses the result to cause the LLM 210 to recognize the object with the lower recognition accuracy.

[0108] Although the embodiments have been described above, the configuration of each part shown in the embodiments can be replaced with other parts having similar functions. Also, any other components or processes may be added. Furthermore, any two or more configurations (features) of the above-described embodiments may be combined. [Explanation of symbols]

[0109] 1. Data 2 Legend information 3. Machine Learning Model 4a 1st directive 4b Second directive 4c 3rd Directive 4d 4th directive 4e 5th Directive 4f 6th directive 4g output information 10. Information processing equipment 11 Storage section 12 Processing section 12a Dialogue Systems

Claims

1. inputting a first instruction statement to a dialogue system using a machine learning model, the first instruction statement instructing the dialogue system to estimate the first information based on data including a plurality of first information pieces and a plurality of second information pieces representing relationships between the plurality of first information pieces; inputting a second instruction sentence to the dialogue system, the second instruction sentence instructing the dialogue system to estimate the meaning of the first information based on third information related to the meaning of the first information; inputting a third instruction sentence to the dialogue system, the third instruction sentence instructing the dialogue system to estimate the second information based on the data; obtaining, from the dialogue system, output information generated based on the second information estimated in response to the third instruction sentence; An information processing program that causes a computer to execute a process.

2. having the computer execute a process of inputting the first instruction sentence to the dialogue system and a process of inputting the second instruction sentence to the dialogue system, and then having the computer execute a process of inputting the third instruction sentence to the dialogue system; The information processing program according to claim 1 .

3. the dialogue system conducts a dialogue in natural language through thought chain reasoning using the machine learning model, The information processing program according to claim 1 .

4. The computer further comprises: executes a process of inputting a fourth instruction sentence to the dialogue system, the fourth instruction sentence instructing the dialogue system to recognize a legend indicating a display mode for each meaning of the first information; In the process of inputting the second instruction sentence to the dialogue system, information of the legend recognized in accordance with the fourth instruction sentence is set as the third information.

4. An information processing program according to claim 1.

5. the first information is represented by a graphic having a different shape for each meaning, causing the computer to further execute a process of inputting a fifth instruction sentence, which instructs the interactive system to classify the first information based on its shape; In the process of inputting the second instruction sentence to the dialogue system, a classification result of the first information classified according to the fifth instruction sentence is set as the third information.

4. An information processing program according to claim 1.

6. the first information is represented by a graphic having a different shape for each meaning, the second information is represented by a line drawn between two figures; causing the computer to further execute a process of inputting a sixth instruction sentence to the dialogue system, the sixth instruction sentence instructing the dialogue system to estimate an intersection point between the figure and the line; and in the process of inputting the third instruction sentence to the dialogue system, the third instruction sentence is a sentence instructing an estimation that places emphasis on the meaning of the first information related to the intersection estimated in accordance with the sixth instruction sentence.

4. An information processing program according to claim 1.

7. inputting a first instruction statement to a dialogue system using a machine learning model, the first instruction statement instructing the dialogue system to estimate the first information based on data including a plurality of first information pieces and a plurality of second information pieces representing relationships between the plurality of first information pieces; inputting a second instruction sentence to the dialogue system, the second instruction sentence instructing the dialogue system to estimate the meaning of the first information based on third information related to the meaning of the first information; inputting a third instruction sentence to the dialogue system, the third instruction sentence instructing the dialogue system to estimate the second information based on the data; obtaining, from the dialogue system, output information generated based on the second information estimated in response to the third instruction sentence; An information processing method in which processing is performed by a computer.

8. a processing unit that inputs a first instruction statement to a dialogue system using a machine learning model, the first instruction statement instructing the dialogue system to estimate a meaning of the first information based on data including a plurality of first information and a plurality of second information representing relationships between the plurality of first information; inputs a second instruction statement to the dialogue system instructing the dialogue system to estimate a meaning of the first information based on third information related to the meaning of the first information; inputs a third instruction statement to the dialogue system instructing the dialogue system to estimate the second information based on the data; and acquires output information from the dialogue system that is generated based on the second information estimated in accordance with the third instruction statement; An information processing device having the above.

Images